Repository navigation
ci(iso): reclaim runner disk before builds - #69
Conversation
|
Diagnosis update: none of the failed ISO runs show ENOSPC (xorriso reports ~81 GB free at write time) — the actual breakage is a missing Generated by Claude Code |
hanthor
left a comment
There was a problem hiding this comment.
Summary
The change itself is harmless headroom and merges cleanly. Two things need correcting before it lands: the version comment on the pinned action is wrong, and the "Fixes" claim is disproven by this PR's own CI.
What I ran
- Baseline on unmodified
main(d34ce2b):just checkexit 0,uvx pre-commit run --all-filesall hooks pass. Your listed validation reproduces — but note neither gate touches workflow logic, so a green local run says nothing about whether the build works. - Merge test: merges cleanly,
git diff --name-only --diff-filter=Uis empty. Also merges cleanly stacked with #70, which touches the same file (both add toreusable-build-iso-anaconda.yml, at different hunks).
Blocking: there is no v10 tag upstream
uses: ublue-os/remove-unwanted-software@695eb75bc387dbcd9685a8e72d23439d8686cba6 # v10I resolved this against the real upstream with git ls-remote https://github.com/ublue-os/remove-unwanted-software:
- Tags published:
v1…v9. There is nov10. The newest isv9=cc0becac701cf642c8f0a6613bbdaf5dc36b259e. 695eb75b…isHEAD/refs/heads/master— the tip of the default branch.git tag --contains 695eb75b…returns nothing.- It is the merge commit of upstream PR #11, titled "Merge pull request #11 from ublue-os/v10", dated 2025-10-10. So the
v10in the comment is the branch name that was merged, not a release.
To be fair on what is not wrong: the SHA exists, the pin is a SHA so it is immutable, and extra-squeeze is a genuine input at that commit (action.yml line 7, description: 'Extra 6GB but ~2 min slower'). Functionally this works.
The problem is the comment. Renovate and every human reader key off the trailing # vN to decide what a "safe bump" is; labelling an untagged master tip as a released v10 will produce a wrong bump later. Either write # master@2025-10-10 (untagged), or pin cc0becac701cf642c8f0a6613bbdaf5dc36b259e # v9 if v9 already carries extra-squeeze. Describing it as the "current org-standard v10 pin" in the body should be corrected too.
Blocking: Fixes projectbluefin/common#796 is not supported by this PR's CI
Every build leg on this PR failed (run 30961565027):
Build ISOs (amd64, main, stable)—92209030401failureBuild ISOs (amd64, nvidia-open, stable)—92209030390failureBuild LTS ISOs / Build ISOs (amd64, main, lts)—92209042344failureBuild LTS ISOs / Build ISOs (amd64, gdx, lts)—92209042351failure- both arm64 LTS legs —
92209042338,92209042327failure
So the disk cleanup did not make the build pass. #70 identifies the actual root cause and I confirmed it independently against main: the Build ISO step never writes iso-dest to $GITHUB_OUTPUT, while Rename ISO still reads steps.build.outputs.iso-dest, so OUTPUT_PATH is empty and the job dies on mv: cannot stat ''. #70's CI has all amd64 legs green on the same base. That is decisive: this is not an ENOSPC problem.
Please drop or downgrade Fixes projectbluefin/common#796 — if ENOSPC is a separate intermittent issue, this can be honest headroom for it, but it cannot be claimed as the fix while the builds fail for an unrelated reason. This PR currently carries an lgtm label with entirely red CI.
Non-blocking
- Sequencing vs #70. This is strictly additive headroom; #70 is the actual unblock. Land #70's
iso-destline first, then re-run this so you get a real signal on whether the cleanup helps. - arm64. The step is added unconditionally, so it also runs on
ubuntu-24.04-arm. Worth confirmingremove-unwanted-softwarebehaves on arm64 runners (the paths it removes, e.g./usr/share/miniconda,/opt/hostedtoolcache, differ per image). Aif: matrix.platform == 'amd64'guard may be warranted. - Silent E2E skip. Not caused by you, but visible on this PR and worth flagging:
Stable main ISO E2E(92211359675) andStable NVIDIA ISO E2E(92211359243) both showconclusion: skipped, becausepull-request.ymlgates them withneeds: build-iso-stable. When the build fails, the install tests do not run and do not report red — they just vanish. This is the same pattern that hid missing install coverage indakota-iso#137. A green-looking summary here would be badly misleading.
Disk-destructive behaviour
None. This PR only frees runner disk space before the build; it does not touch partitioning, wiping, bootloader/NVRAM, or install-time disk selection, and no install-path test coverage changes.
Generated by Claude Code
|
Checked whether this had been overtaken by other work, and it has not. It is still needed. main's only merge since this opened is #70 (hanthor, 2026-09-17), which fixed a missing hanthor's changes-requested review from 2026-09-12 is still unaddressed, with no new commits and no reply. The two blocking points both still stand:
@bketelsen it needs a rebase onto current main to pick up #70, plus those two fixes, and then it should be reviewable again. |
Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Correct the untagged action pin label and document the cleanup scope and CI validation limits. Assisted-by: GPT-6 via Codex
170e7d5 to
3553fd5
Compare
|
PTAL |
|
Thanks, both points I raised are fixed at 3553fd5.
I haven't approved yet because no CI has run on this head. All three runs (Skill Drift, Reusable Build, ISO Build PR Check) are waiting for a maintainer to approve them, since this comes from a fork. Once the build legs have run I'll take another look. |
hanthor
left a comment
There was a problem hiding this comment.
Both points from my earlier review are addressed: the action at reusable-build-iso-anaconda.yml:~129 is SHA-pinned with an accurate provenance comment, extra-squeeze exists on that commit, and the incorrect Fixes reference is gone. Not merging yet: the fork workflows still need maintainer approval to run, and the build legs should be green before this lands.
Danathar
left a comment
There was a problem hiding this comment.
The build legs ran today (run 36079133044). The cleanup step worked on every leg: amd64 stable went from 86G to 125G free, arm64 from 108G to 127G. All amd64 legs and arm64 gdx passed. The three red jobs don't come from this change. arm64 main lts fails at podman pull ghcr.io/ublue-os/bluefin:lts with no image found in image index for architecture arm64. That tag lists only amd64, the same error hit another branch on 8 Aug, and main soft-fails arm64 since #70. Both Stable E2E jobs time out because the kickstart never reaches anaconda (Kickstart file /run/install/ks.cfg is missing. in the serial log). They have failed the same way on every run since 7 Aug. Approving.
|
Ready to merge under R2 / org policy (CI change, independent human reviews): approved by @hanthor and @Danathar on head 3553fd5. The three red checks are pre-existing and unrelated to this diff. Build LTS arm64 main fails at |
* fix(e2e): deliver the kickstart through an initramfs overlay The stable E2E jobs have failed on every run since 2026-08-07 with "Kickstart file /run/install/ks.cfg is missing." The harness boots the ISO's extracted vmlinuz/initramfs with inst.ks=http://..., but that boot option only sets --ks=/run/install/ks.cfg inside anaconda (store_const in pyanaconda/argument_parsing.py); fetching the URL into that path is the job of anaconda's dracut module, which the titanoboa initramfs does not carry. Nothing ever wrote the file, anaconda aborted, and the 1800s budget timed out. Append a gzip cpio overlay to the initramfs carrying the rendered kickstart at /etc/e2e/ks.cfg plus an initrd unit (wanted by initrd.target, ordered before initrd-cleanup.service) that copies it to /run/install/ks.cfg, which survives switch-root - the same hand-off anaconda-lib.sh:parse_kickstart() performs. Point inst.ks= at the embedded file and drop the HTTP server nothing fetched from. Raise the default install budget to 2700s: anaconda's optional Subscription module fails to activate in this image and dbus waits its full 600s service_start_timeout before installation starts, which the old 1800s budget did not account for. Refs #69 * fix(e2e): take the kickstart image ref from the ISO's container store Run 36276284373 proved kickstart delivery (e2e-kickstart.service finished at 19s, anaconda ran the kickstart to the payload step) and then failed in `ostree container image deploy --transport=containers-storage` with "reference ... ghcr.io/ublue-os/bluefin:stable does not resolve to an image ID". The store name is not guessable from IMAGE_REF/IMAGE_TAG: titanoboa `podman pull`s whatever the build passed, so the nvidia-open ISO carries ghcr.io/ublue-os/bluefin-nvidia-open:stable while the workflow asked the kickstart for bluefin:stable. Read the stored names out of the ISO (unsquashfs on /LiveOS/squashfs.img at its LBA offset -> var/lib/containers/storage/overlay-images/images.json) and render the single --url from that; IMAGE_REF:IMAGE_TAG only disambiguates when an ISO carries several images, and a store with no match fails loudly. The kickstart now also prints `e2e-pre:` lines (podman images, skopeo inspect of the chosen URL) to the serial log so the installer's own view of the store is in the evidence, and the workflow enables /dev/kvm for the runner so accel=kvm:tcg stops falling back to a ~45-minute TCG install. * test(e2e): log the installer's container store right before the payload step c/image's storage transport reports any store error while resolving a name as 'does not resolve to an image ID', so record podman images / skopeo inspect / the storage mounts from a %pre-install block as well, which runs immediately before 'ostree container image deploy'. --------- Co-authored-by: Danathar <Danathar@users.noreply.github.com>
|
@castrojo friendly ping, since this has been idle since 09-29: it is still approved by hanthor and me at 3553fd5, carries |
Reclaim runner disk space before each shared Titanoboa ISO build by running
ublue-os/remove-unwanted-softwarewithextra-squeezebefore dependency installation and checkout.The action remains pinned to immutable commit
695eb75bc387dbcd9685a8e72d23439d8686cba6, now correctly identified as an untagged master commit from 2025-10-10. Upstream's latest release tag isv9, which does not exposeextra-squeeze.This change provides additional disk headroom while Titanoboa expands OSTree layers. It is not a demonstrated fix for projectbluefin/common#796: the previous builds failed for unrelated reasons, including the missing
iso-destoutput corrected in #70. The branch is rebased onto current main, including #70.Cleanup remains enabled on both architectures. In the previous PR run, the cleanup step succeeded in both arm64 jobs (main, gdx). The pinned action tolerates absent directories and disables swap before removing the swapfile.
Validation:
pre-commit run --all-filesjust checkactionlint -shellcheck='' .github/workflows/reusable-build-iso-anaconda.yml .github/workflows/pull-request.ymlgit diff --checkThese local checks cover syntax and formatting, not a complete ISO build or installation. Fresh CI must be assessed per job: main currently tolerates arm64 build failures, and Stable E2E jobs are skipped when their prerequisite build fails. The workflow skill documents the action pin, cleanup ordering, and these validation limits.