From fcfd233e65f53d31d7ef3fd3a4a1f9e885a87366 Mon Sep 17 00:00:00 2001 From: Danathar Date: Tue, 10 Nov 2026 16:55:02 -0500 Subject: [PATCH] ci(e2e): run the VM jobs on ubuntu-24.04 ubuntu-latest became ubuntu-26.04 on 2026-11-09. On 26.04 bootc's loopback install stops with "root partition 3 has type " before writing anything, so every VM boot fails: the v1 gate has held v1 since 2026-11-09 09:41Z and the startup checks fail on every PR that runs them (#1123). The same image (sha256:c10a8b18) installed on 24.04 on 11-08 (run 42479593163) and fails on 26.04 today (run 42908422162). Run the e2e test job and the startup lane on ubuntu-24.04, as iso-validation.yml already does, until the install works on 26.04. Correct the troubleshooting entry, which called this a timing race. Fixes #1123 Assisted-by: Claude Opus 5.5 via Claude Code Co-Authored-By: Claude Opus 5.5 Signed-off-by: Danathar --- .github/workflows/e2e.yml | 5 ++++- .github/workflows/startup-qualification.yml | 5 ++++- .../ci-ops/e2e-workflow/references/troubleshooting.md | 9 ++++++--- 3 files changed, 14 insertions(+), 5 deletions(-) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 853f9a823..7442acfc8 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -212,7 +212,10 @@ jobs: # Run whenever the matrix resolved, even if compose was skipped (toggle # off) or failed (fallback to the raw image + runtime installs). if: ${{ !cancelled() && needs.matrix.result == 'success' }} - runs-on: ubuntu-latest + # Pinned: on ubuntu-26.04 (ubuntu-latest since 2026-11-09) bootc's + # loopback install stops with "root partition 3 has type "; the + # same image installs on 24.04 (#1123). Lift once it works on 26.04. + runs-on: ubuntu-24.04 timeout-minutes: 120 permissions: contents: read diff --git a/.github/workflows/startup-qualification.yml b/.github/workflows/startup-qualification.yml index fa5b38ef4..f203ceb00 100644 --- a/.github/workflows/startup-qualification.yml +++ b/.github/workflows/startup-qualification.yml @@ -57,7 +57,10 @@ concurrency: jobs: startup: name: startup (${{ matrix.control }}) - runs-on: ubuntu-latest + # Pinned: on ubuntu-26.04 (ubuntu-latest since 2026-11-09) bootc's + # loopback install stops with "root partition 3 has type "; the + # same image installs on 24.04 (#1123). Lift once it works on 26.04. + runs-on: ubuntu-24.04 timeout-minutes: 75 strategy: fail-fast: false diff --git a/docs/skills/ci-ops/e2e-workflow/references/troubleshooting.md b/docs/skills/ci-ops/e2e-workflow/references/troubleshooting.md index 9f91e4c4c..8edd50adb 100644 --- a/docs/skills/ci-ops/e2e-workflow/references/troubleshooting.md +++ b/docs/skills/ci-ops/e2e-workflow/references/troubleshooting.md @@ -76,11 +76,14 @@ still belong only in a fresh disposable guest without publishing credentials. ### Install OCI image fails: "root partition 3 has type " -**Root cause:** bootc 1.17+ (Dakota since 2026-11-01) checks the new root partition's DPS type through `lsblk`/udev about 200 ms after `sfdisk` re-reads the partition table. On GitHub-hosted runners udev has not always processed `loopNp3` by then, so `lsblk` reports no type and bootc aborts before writing anything. It is a race, not a property of the image: runs of the same Dakota image before and after got past it. +**Root cause:** the runner's Ubuntu release, not the image and not a timing race (#1123). bootc 1.17+ (Dakota since 2026-11-01) reads the new root partition's DPS type through `lsblk` after `sfdisk`. On `ubuntu-26.04` runners, which `ubuntu-latest` became on 2026-11-09, it gets no type every time and bootc aborts before writing anything. On `ubuntu-24.04` the same image installs. -**Evidence:** run 42611596358 — `error: Installing to disk: Creating rootfs: root partition 3 has type ; expected 4f68bce3-e8cd-4db1-96e7-fbcaf984b709`, about 0.2 s after `Calling ioctl() to re-read partition table`. +**Evidence:** +- Image `sha256:c10a8b18…` installed on `ubuntu-24.04` (gate run 42479593163, 2026-11-08) and failed on `ubuntu-26.04` (run 42908422162, 2026-11-10). +- Every `v1` gate boot since the switch failed all 3 install attempts identically (for example run 42807521428). +- Mounting the host's `/run/udev` into the install container did not help (run 42905999048). -**Fix:** the "Install OCI image and configure disk" step in `e2e.yml` retries `bootc install to-disk` (3 attempts) whenever no ostree deployment was written, and fails with a `bootc install wrote no deployment` annotation naming the image and bootc's `error:` line if every attempt fails. Do not retry when the deployment *was* written: `bootupd is required for ostree-based installs` (bootc 1.17+ with `--bootloader systemd`) is the expected late bootloader failure, and direct kernel boot continues past it. +**Fix:** the VM jobs (`e2e.yml` test job, `startup-qualification.yml`) run on `ubuntu-24.04` instead of `ubuntu-latest`, as `iso-validation.yml` already does. Lift the pin once the install works on 26.04. Separately, the "Install OCI image and configure disk" step in `e2e.yml` retries `bootc install to-disk` (3 attempts) whenever no ostree deployment was written, and fails with a `bootc install wrote no deployment` annotation naming the image and bootc's `error:` line if every attempt fails. Do not retry when the deployment *was* written: `bootupd is required for ostree-based installs` (bootc 1.17+ with `--bootloader systemd`) is the expected late bootloader failure, and direct kernel boot continues past it. ### SSH never became ready