Repository navigation
fix(e2e): retry the disk install when the test machine's disk isn't ready yet - #1125
Conversation
bootc 1.17 (in Dakota since 2026-11-01) checks the new root partition's
type through udev about 200 ms after sfdisk. On GitHub-hosted runners
udev has not always seen loopNp3 by then, so bootc aborts with "root
partition 3 has type <none>" before writing anything (run 42611596358).
The install step then died on a bare "mount: wrong fs type".
Retry bootc install to-disk up to three times while no ostree
deployment has been written, probe the root partition for a filesystem
before mounting it, and fail with an annotation naming the image and
bootc's error line when every attempt fails. The late bootloader
failure ("bootupd is required for ostree-based installs") still
continues to direct kernel boot as before.
Closes projectbluefin#1124
Hive-Run: projectbluefin#1124
Hive-Plan: issue-1124 install-step retry
Hive-Spec: projectbluefin#1124#acceptance-criteria
Assisted-by: Claude Opus via omp
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
924da4b
|
|
Needs a decision, not a merge: the PR edits CI workflow / action files: awaiting owner approval — a hive owner / trusted merger's GitHub Approve review on the current head (or a Head: verdict approved @924da4b, 7/7 checks green, soak 30m0s elapsed. What to do: a trusted merger approves this head ( Clear the cause and the hive re-evaluates on its next pass; the stage moves to Updated 2026-11-09 17:43 UTC. |
|
Sentinel reviewed at One thing before this closes #1124: the only "partition 3 has type " failure (run 42611596358) is also the first run on ubuntu-26.04. |
|
Addressed review: |
|
@Danathar approved head |
|
Hive will not merge this PR: it changes path(s) listed in
Hive added the |
|
Ready for a human merge: verdict approved @924da4b, approved by @Danathar, 7/7 checks green, soak 30m0s elapsed. The PR touches a human-merge path (auto_merge.human_merge_paths); @Danathar approved this head, so the only step left is the merge. The hive never merges this lane's changes itself: click Merge (or |
Summary: The on-demand test run puts the operating-system image onto a blank virtual disk before it can test anything. With current Dakota images, the installer sometimes checks the new disk layout too quickly, before Linux has finished registering it, and gives up. The test run then stopped with a confusing disk error. This change makes the test run try the install again when it fails before writing anything. If every attempt fails, the run now stops with a plain message that names the image and the installer's own error.
What I need from you: Approve it, then start a manual test run against the Dakota image from this branch (I'm not allowed to start workflows).
Root cause
Dakota moved to bootc 1.17.3 on 2026-11-01 (
projectbluefin/dakotacommitc486a1d1). In bootc 1.17,install to-diskrunssfdisk, sleeps 200 ms, runsudevadm settle, and then reads the root partition's DPS type withlsblk(crates/lib/src/install/baseline.rsat v1.17.3,udev_settle()and theroot partition {} has type {}check). The bootc source itself calls that sleep a workaround for a race. On the GitHub-hosted runner in run 42611596358, udev had not processedloop0p3yet:Nothing was deployed. The step's own diagnostic never ran, because the bare
blkid/mounton the empty partition ended the step underbash -efirst.Correction to the issue's table: the 11-01 to 11-06 runs (41259347169, 42124147912, 42124153048, 42126521360, 42129446403, 42221191543) all logged
exited 1 after writing deployment …; continuing with direct kernel boot.bootupd is required for ostree-based installsis the late bootloader failure this step already expects and tolerates. Those runs booted and failed later in the suites. Example: 42221191543 ran smoke-a and failed atexit "1"after the results upload. So far, only the 11-09 run failed at install. Dakota's image commits between the 11-06 digest and the 11-09 digest (kanata, shim pin, GameMode) don't touch bootc. That points to an intermittent race, not a new layout requirement.Change (
.github/workflows/e2e.yml, step "Install OCI image and configure disk")bootc install to-diskup to 3 times only while no ostree deployment has been written. Before each retry, detach any loop device left ondisk.rawand runudevadm settle. A late failure (deployment present, bootloader step failed) still continues to direct kernel boot on the first attempt, as before.${LOOP}p3withblkidbefore mounting it. An empty partition now goes to the retry and the diagnostic path instead ofmount: wrong fs type.::error title=bootc install wrote no deployment::with the image ref, the attempt count, the exit code and bootc's firsterror:line, then print the full log of the last attempt.tests/unit/test_selinux_qualify.py:test_harness_files_are_labeled_before_unmount_with_candidate_setfilesnow anchors on the lastsudo umount /mnt/root(rindex). The retry loop adds an earlier unmount of a discarded attempt. The test is meant to check the final unmount before boot.docs/skills/ci-ops/e2e-workflow/references/troubleshooting.mdgets a section on thehas type <none>signature, its evidence and the retry rule (do not retry once a deployment is written).Kept inline in
e2e.ymlrather than in a newscripts/helper. Helpers come from thetest_refsparse checkout, so a manual run with an oldertest_ref(like the one #1116 needs) would not have the script.Verification
Ran a local sandbox of the exact
run:block, pulled from the YAML with stubs forsudo/podman/losetup/blkid/mount/umount/udevadm:bootupd is required…)/dev/loop0, continues::error title=bootc install wrote no deployment::… dakota@sha256:dc3f… (3 attempt(s), last exit 1): error: Installing to disk: Creating rootfs: root partition 3 has type <none>…ruff check tests/ --select E,F,W --ignore E501: pass.python3 -m pytest tests/unit/ -q: 2660 passed.scripts/validate_docs.py: pass.scripts/generate_skill_index.py --check: in sync.Not run: a real VM run. My session cannot dispatch workflows. Test plan steps 2–3 from the issue are still needed: dispatch
manual.ymlfrom this branch withimage=ghcr.io/projectbluefin/dakota@sha256:dc3f6aee7a9fbd8373e8cf9d0dfbfd80e6beaf4b19922a7c0daf4725d3086e43,suites=common, then once with a Bluefin image. Because the race is intermittent, one green run shows no regression but cannot prove the retry fires. The retry path is covered by the sandbox above.Notes
e2e.ymlandtest_selinux_qualify.py. Its hunks are e2e.yml L70–80 and L563–571, plus test L337–366. None of them overlap the install-step lines (L392–486) or test L384 changed here..github/actions/gnome-e2e/action.ymlandstartup-qualification.yml. Neither is on themanual.ymlpath, so they are left unchanged here.blkid -pfor the type check or waiting for the partition, would remove the need for this retry.Refs #1124
Not closing #1124 yet: the only "partition 3 has type " failure (run 42611596358) is also the first run on ubuntu-26.04, so it may be consistent there rather than a race. Close once a manual.yml run on the current runner (suites=common) gets past the install.
— hive: backend=omp model=anthropic/claude-opus-5-5
🐝 Hive Agent:
contributor| SHA:924da4b6