Skip to content

fix(e2e): retry the disk install when the test machine's disk isn't ready yet - #1125

Merged
Danathar merged 1 commit into
projectbluefin:mainfrom
Karibhou:fix/1124-dakota-bootc-install
Nov 9, 2026
Merged

Danathar merged 1 commit into
projectbluefin:mainfrom
Karibhou:fix/1124-dakota-bootc-install

Conversation

@Karibhou

@Karibhou Karibhou commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary: The on-demand test run puts the operating-system image onto a blank virtual disk before it can test anything. With current Dakota images, the installer sometimes checks the new disk layout too quickly, before Linux has finished registering it, and gives up. The test run then stopped with a confusing disk error. This change makes the test run try the install again when it fails before writing anything. If every attempt fails, the run now stops with a plain message that names the image and the installer's own error.

What I need from you: Approve it, then start a manual test run against the Dakota image from this branch (I'm not allowed to start workflows).


Root cause

Dakota moved to bootc 1.17.3 on 2026-11-01 (projectbluefin/dakota commit c486a1d1). In bootc 1.17, install to-disk runs sfdisk, sleeps 200 ms, runs udevadm settle, and then reads the root partition's DPS type with lsblk (crates/lib/src/install/baseline.rs at v1.17.3, udev_settle() and the root partition {} has type {} check). The bootc source itself calls that sleep a workaround for a race. On the GitHub-hosted runner in run 42611596358, udev had not processed loop0p3 yet:

11:49:25.97  Calling ioctl() to re-read partition table.
11:49:26.19  error: Installing to disk: Creating rootfs: root partition 3 has type <none>; expected 4f68bce3-…
11:49:26.34  mount: /mnt/root: wrong fs type, bad option, bad superblock on /dev/loop0p3 …
11:49:26.35  ##[error]Process completed with exit code 32.

Nothing was deployed. The step's own diagnostic never ran, because the bare blkid/mount on the empty partition ended the step under bash -e first.

Correction to the issue's table: the 11-01 to 11-06 runs (41259347169, 42124147912, 42124153048, 42126521360, 42129446403, 42221191543) all logged exited 1 after writing deployment …; continuing with direct kernel boot. bootupd is required for ostree-based installs is the late bootloader failure this step already expects and tolerates. Those runs booted and failed later in the suites. Example: 42221191543 ran smoke-a and failed at exit "1" after the results upload. So far, only the 11-09 run failed at install. Dakota's image commits between the 11-06 digest and the 11-09 digest (kanata, shim pin, GameMode) don't touch bootc. That points to an intermittent race, not a new layout requirement.

Change (.github/workflows/e2e.yml, step "Install OCI image and configure disk")

  • Run bootc install to-disk up to 3 times only while no ostree deployment has been written. Before each retry, detach any loop device left on disk.raw and run udevadm settle. A late failure (deployment present, bootloader step failed) still continues to direct kernel boot on the first attempt, as before.
  • Probe ${LOOP}p3 with blkid before mounting it. An empty partition now goes to the retry and the diagnostic path instead of mount: wrong fs type.
  • If every attempt fails, emit ::error title=bootc install wrote no deployment:: with the image ref, the attempt count, the exit code and bootc's first error: line, then print the full log of the last attempt.
  • Updated the step's header comment to describe both failure classes.
  • tests/unit/test_selinux_qualify.py: test_harness_files_are_labeled_before_unmount_with_candidate_setfiles now anchors on the last sudo umount /mnt/root (rindex). The retry loop adds an earlier unmount of a discarded attempt. The test is meant to check the final unmount before boot.
  • Skill write-back: docs/skills/ci-ops/e2e-workflow/references/troubleshooting.md gets a section on the has type <none> signature, its evidence and the retry rule (do not retry once a deployment is written).

Kept inline in e2e.yml rather than in a new scripts/ helper. Helpers come from the test_ref sparse checkout, so a manual run with an older test_ref (like the one #1116 needs) would not have the script.

Verification

  • Ran a local sandbox of the exact run: block, pulled from the YAML with stubs for sudo/podman/losetup/blkid/mount/umount/udevadm:

    bootc outcome per attempt result
    ok 1 attempt, continues
    late (deployment + bootupd is required…) 1 attempt, continues with "after writing deployment"
    early, late retries once, detaches stale /dev/loop0, continues
    early, early, ok 3 attempts, continues
    filesystem without deployment, ok unmounts, retries, continues
    early ×3 exit 1 with ::error title=bootc install wrote no deployment::… dakota@sha256:dc3f… (3 attempt(s), last exit 1): error: Installing to disk: Creating rootfs: root partition 3 has type <none>…
  • ruff check tests/ --select E,F,W --ignore E501: pass. python3 -m pytest tests/unit/ -q: 2660 passed. scripts/validate_docs.py: pass. scripts/generate_skill_index.py --check: in sync.

  • Not run: a real VM run. My session cannot dispatch workflows. Test plan steps 2–3 from the issue are still needed: dispatch manual.yml from this branch with image=ghcr.io/projectbluefin/dakota@sha256:dc3f6aee7a9fbd8373e8cf9d0dfbfd80e6beaf4b19922a7c0daf4725d3086e43, suites=common, then once with a Bluefin image. Because the race is intermittent, one green run shows no regression but cannot prove the retry fires. The retry path is covered by the sandbox above.

Notes

  • File overlap: feat(e2e): boot the e2e VM with SELinux enforcing by default #1119 (held, needs a decision) also edits e2e.yml and test_selinux_qualify.py. Its hunks are e2e.yml L70–80 and L563–571, plus test L337–366. None of them overlap the install-step lines (L392–486) or test L384 changed here.
  • The same install logic is copied in .github/actions/gnome-e2e/action.yml and startup-qualification.yml. Neither is on the manual.yml path, so they are left unchanged here.
  • Upstream: bootc already knows this race exists (the 200 ms sleep). A bootc fix, such as using blkid -p for the type check or waiting for the partition, would remove the need for this retry.

Refs #1124

Not closing #1124 yet: the only "partition 3 has type " failure (run 42611596358) is also the first run on ubuntu-26.04, so it may be consistent there rather than a race. Close once a manual.yml run on the current runner (suites=common) gets past the install.

— hive: backend=omp model=anthropic/claude-opus-5-5


🐝 Hive Agent: contributor | SHA: 924da4b6

bootc 1.17 (in Dakota since 2026-11-01) checks the new root partition's
type through udev about 200 ms after sfdisk. On GitHub-hosted runners
udev has not always seen loopNp3 by then, so bootc aborts with "root
partition 3 has type <none>" before writing anything (run 42611596358).
The install step then died on a bare "mount: wrong fs type".

Retry bootc install to-disk up to three times while no ostree
deployment has been written, probe the root partition for a filesystem
before mounting it, and fail with an annotation naming the image and
bootc's error line when every attempt fails. The late bootloader
failure ("bootupd is required for ostree-based installs") still
continues to direct kernel boot as before.

Closes projectbluefin#1124

Hive-Run: projectbluefin#1124
Hive-Plan: issue-1124 install-step retry
Hive-Spec: projectbluefin#1124#acceptance-criteria
Assisted-by: Claude Opus via omp
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 9, 2026
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@hivecommons-hive

Copy link
Copy Markdown
Contributor
924da4b

⚠️ Sentinel alert — maintainer review required

Hive flagged this PR (author @Karibhou, head 924da4b652d0) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/e2e.yml
  • ci_gate_weakening — CI configuration is deleted, softened or has gate checks removed (Disables or softens CI gates (continue-on-error, || true, removed checks))
    • .github/workflows/e2e.yml

Hive added the sentinel-alert label. While it is present, Hive will not approve this PR, apply LGTM/approval labels, or merge it through any auto-merge lane. Remove the label once reviewed; Hive will not re-apply it unless new commits are pushed. Tune paths and behaviors under sentinel in hive.yaml or the dashboard Security tab.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 9, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge: the PR edits CI workflow / action files: awaiting owner approval — a hive owner / trusted merger's GitHub Approve review on the current head (or a /hive decision approve reply, typed on the PR or from the dashboard) releases the fence and the hive merges it; approval on an earlier head does not count unless the hive's own merge of the base branch is all that moved the head, so re-approve after a push.

Head: verdict approved @924da4b, 7/7 checks green, soak 30m0s elapsed.

What to do: a trusted merger approves this head (924da4b) with a GitHub Approve review (or a /hive decision approve reply, typed here or from the dashboard), which releases the CI workflow-file fence, and the hive merges the PR once the other gates clear; approval on an earlier head carries over only the hive's own merge of the base branch (diff unchanged), and a /hive approve reply does not release the fence.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

Updated 2026-11-09 17:43 UTC.

@Danathar

Danathar commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Sentinel reviewed at 924da4b (the head it scanned). The ci_gate_weakening flag does not hold: the new || true are only on cleanup (losetup -d of stale loops, udevadm settle, the grep for bootc's error line). The step still exit 1s when no deployment is written, there is no continue-on-error, and uses:, permissions: {} and triggers match main. Removing sentinel-alert.

One thing before this closes #1124: the only "partition 3 has type " failure (run 42611596358) is also the first run on ubuntu-26.04. ubuntu-latest was 24.04 on 11-06 (run 42221191543), when the install got past this point. So the failure may be consistent on 26.04 rather than a race, and then all three attempts would fail with only a clearer error. Please change "Closes #1124" to "Refs #1124" until a manual.yml run on the current runner (Dakota digest dc3f6aee…, suites=common) gets past the install. You are right that the "bootupd is required" rows in #1124 were not install failures: those runs continued with direct kernel boot.

@Karibhou

Karibhou commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Addressed review: Closes #1124 → Refs #1124 in the PR body, with the ubuntu-26.04 caveat. No code change; #1124 stays open until a manual.yml run (suites=common) on the current runner gets past the install.

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 924da4b: retry logic for the disk install; no pin changes; the new || true lines are only on cleanup (losetup -d, udevadm settle) and error capture, so no gate is softened. 7/7 checks green.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Danathar approved head 924da4b652d0, which edits CI workflow / action files. That explicit owner approval releases the contributor lane's workflow-path fence: the hive merges this PR itself once the usual gates (CI, soak, holds, review verdict) are clear. A new push re-arms the fence until the new head is approved again; the hive's own merge of the base branch, which leaves the diff unchanged, does not.

@hivecommons-hive hivecommons-hive Bot added the hold Prevent merge. Use `/hold` or `/hold cancel`. label Nov 9, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive will not merge this PR: it changes path(s) listed in auto_merge.human_merge_paths for this repo, so a person must merge it.

  • .github/workflows/e2e.yml

Hive added the hold label. A maintainer should review and merge this PR by hand.

@Danathar
Danathar merged commit 144e7f4 into projectbluefin:main Nov 9, 2026
7 checks passed
@hivecommons-hive hivecommons-hive Bot added stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Nov 9, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Ready for a human merge: verdict approved @924da4b, approved by @Danathar, 7/7 checks green, soak 30m0s elapsed. The PR touches a human-merge path (auto_merge.human_merge_paths); @Danathar approved this head, so the only step left is the merge.

The hive never merges this lane's changes itself: click Merge (or /lgtm + /approve for tide) to land it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off hold Prevent merge. Use `/hold` or `/hold cancel`. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants