Skip to content

ci(iso): reclaim runner disk before builds - #69

Merged
castrojo merged 2 commits into
projectbluefin:mainfrom
bketelsen:fix/iso-enospc
Oct 9, 2026
Merged

castrojo merged 2 commits into
projectbluefin:mainfrom
bketelsen:fix/iso-enospc

Conversation

@bketelsen

@bketelsen bketelsen commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Reclaim runner disk space before each shared Titanoboa ISO build by running ublue-os/remove-unwanted-software with extra-squeeze before dependency installation and checkout.

The action remains pinned to immutable commit 695eb75bc387dbcd9685a8e72d23439d8686cba6, now correctly identified as an untagged master commit from 2025-10-10. Upstream's latest release tag is v9, which does not expose extra-squeeze.

This change provides additional disk headroom while Titanoboa expands OSTree layers. It is not a demonstrated fix for projectbluefin/common#796: the previous builds failed for unrelated reasons, including the missing iso-dest output corrected in #70. The branch is rebased onto current main, including #70.

Cleanup remains enabled on both architectures. In the previous PR run, the cleanup step succeeded in both arm64 jobs (main, gdx). The pinned action tolerates absent directories and disables swap before removing the swapfile.

Validation:

  • pre-commit run --all-files
  • just check
  • actionlint -shellcheck='' .github/workflows/reusable-build-iso-anaconda.yml .github/workflows/pull-request.yml
  • git diff --check

These local checks cover syntax and formatting, not a complete ISO build or installation. Fresh CI must be assessed per job: main currently tolerates arm64 build failures, and Stable E2E jobs are skipped when their prerequisite build fails. The workflow skill documents the action pin, cleanup ordering, and these validation limits.

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 5, 2026
@hanthor

hanthor commented Aug 7, 2026

Copy link
Copy Markdown
Member

Diagnosis update: none of the failed ISO runs show ENOSPC (xorriso reports ~81 GB free at write time) — the actual breakage is a missing iso-dest step output plus missing arm64 manifest entries upstream, fixed in #70. This PR's cleanup step is harmless headroom, so the plan (maintainer-approved) is to merge it once #70 lands and this branch is updated so its checks can pass.


Generated by Claude Code

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @castrojo for Hive auto-merge on green CI.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @castrojo for Hive auto-merge on green CI.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @castrojo for Hive auto-merge on green CI.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The change itself is harmless headroom and merges cleanly. Two things need correcting before it lands: the version comment on the pinned action is wrong, and the "Fixes" claim is disproven by this PR's own CI.

What I ran

  • Baseline on unmodified main (d34ce2b): just check exit 0, uvx pre-commit run --all-files all hooks pass. Your listed validation reproduces — but note neither gate touches workflow logic, so a green local run says nothing about whether the build works.
  • Merge test: merges cleanly, git diff --name-only --diff-filter=U is empty. Also merges cleanly stacked with #70, which touches the same file (both add to reusable-build-iso-anaconda.yml, at different hunks).

Blocking: there is no v10 tag upstream

uses: ublue-os/remove-unwanted-software@695eb75bc387dbcd9685a8e72d23439d8686cba6 # v10

I resolved this against the real upstream with git ls-remote https://github.com/ublue-os/remove-unwanted-software:

  • Tags published: v1 … v9. There is no v10. The newest is v9 = cc0becac701cf642c8f0a6613bbdaf5dc36b259e.
  • 695eb75b… is HEAD / refs/heads/master — the tip of the default branch. git tag --contains 695eb75b… returns nothing.
  • It is the merge commit of upstream PR #11, titled "Merge pull request #11 from ublue-os/v10", dated 2025-10-10. So the v10 in the comment is the branch name that was merged, not a release.

To be fair on what is not wrong: the SHA exists, the pin is a SHA so it is immutable, and extra-squeeze is a genuine input at that commit (action.yml line 7, description: 'Extra 6GB but ~2 min slower'). Functionally this works.

The problem is the comment. Renovate and every human reader key off the trailing # vN to decide what a "safe bump" is; labelling an untagged master tip as a released v10 will produce a wrong bump later. Either write # master@2025-10-10 (untagged), or pin cc0becac701cf642c8f0a6613bbdaf5dc36b259e # v9 if v9 already carries extra-squeeze. Describing it as the "current org-standard v10 pin" in the body should be corrected too.

Blocking: Fixes projectbluefin/common#796 is not supported by this PR's CI

Every build leg on this PR failed (run 30961565027):

  • Build ISOs (amd64, main, stable) — 92209030401 failure
  • Build ISOs (amd64, nvidia-open, stable) — 92209030390 failure
  • Build LTS ISOs / Build ISOs (amd64, main, lts) — 92209042344 failure
  • Build LTS ISOs / Build ISOs (amd64, gdx, lts) — 92209042351 failure
  • both arm64 LTS legs — 92209042338, 92209042327 failure

So the disk cleanup did not make the build pass. #70 identifies the actual root cause and I confirmed it independently against main: the Build ISO step never writes iso-dest to $GITHUB_OUTPUT, while Rename ISO still reads steps.build.outputs.iso-dest, so OUTPUT_PATH is empty and the job dies on mv: cannot stat ''. #70's CI has all amd64 legs green on the same base. That is decisive: this is not an ENOSPC problem.

Please drop or downgrade Fixes projectbluefin/common#796 — if ENOSPC is a separate intermittent issue, this can be honest headroom for it, but it cannot be claimed as the fix while the builds fail for an unrelated reason. This PR currently carries an lgtm label with entirely red CI.

Non-blocking

  1. Sequencing vs #70. This is strictly additive headroom; #70 is the actual unblock. Land #70's iso-dest line first, then re-run this so you get a real signal on whether the cleanup helps.
  2. arm64. The step is added unconditionally, so it also runs on ubuntu-24.04-arm. Worth confirming remove-unwanted-software behaves on arm64 runners (the paths it removes, e.g. /usr/share/miniconda, /opt/hostedtoolcache, differ per image). A if: matrix.platform == 'amd64' guard may be warranted.
  3. Silent E2E skip. Not caused by you, but visible on this PR and worth flagging: Stable main ISO E2E (92211359675) and Stable NVIDIA ISO E2E (92211359243) both show conclusion: skipped, because pull-request.yml gates them with needs: build-iso-stable. When the build fails, the install tests do not run and do not report red — they just vanish. This is the same pattern that hid missing install coverage in dakota-iso#137. A green-looking summary here would be badly misleading.

Disk-destructive behaviour

None. This PR only frees runner disk space before the build; it does not touch partitioning, wiping, bootloader/NVRAM, or install-time disk selection, and no install-path test coverage changes.


Generated by Claude Code

@Danathar

Copy link
Copy Markdown
Contributor

Checked whether this had been overtaken by other work, and it has not. It is still needed.

main's only merge since this opened is #70 (hanthor, 2026-09-17), which fixed a missing iso-dest GITHUB_OUTPUT. That is a different bug, not disk space. I could not find any commit on main or any merged PR in this repo that adds a runner-disk-cleanup step, and main today still builds on plain ubuntu-24.04 and -arm with no cleanup action at all. So what this PR adds does not exist anywhere else yet.

hanthor's changes-requested review from 2026-09-12 is still unaddressed, with no new commits and no reply. The two blocking points both still stand:

  • The # v10 comment on the pinned SHA is wrong. That SHA is master's tip, not a release, and v9 is the newest real tag.
  • Fixes projectbluefin/common#796 is disproven by this PR's own CI, which is still all red. Build run 30961565027 is unchanged since the review.

@bketelsen it needs a rebase onto current main to pick up #70, plus those two fixes, and then it should be reviewable again.

bketelsen and others added 2 commits September 24, 2026 20:44
Assisted-by: GPT-5.6 Sol via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Correct the untagged action pin label and document the cleanup scope and CI validation limits.

Assisted-by: GPT-6 via Codex
@bketelsen

Copy link
Copy Markdown
Contributor Author

PTAL

@Danathar

Copy link
Copy Markdown
Contributor

Thanks, both points I raised are fixed at 3553fd5.

  • The pin comment now says master@2025-10-10 (untagged). I re-checked upstream: the newest tag is still v9 (cc0beca), there is no v10, and v9's action.yml has no extra-squeeze input, so staying on 695eb75 is the right call.
  • The Fixes projectbluefin/common#796 claim is gone, and the body now says this is disk headroom, not a proven fix.
  • The branch is 0 commits behind main (35c4ea0), so it includes fix(ci): restore ISO rename output and soft-fail arm64 legs #70.

I haven't approved yet because no CI has run on this head. All three runs (Skill Drift, Reusable Build, ISO Build PR Check) are waiting for a maintainer to approve them, since this comes from a fork. Once the build legs have run I'll take another look.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points from my earlier review are addressed: the action at reusable-build-iso-anaconda.yml:~129 is SHA-pinned with an accurate provenance comment, extra-squeeze exists on that commit, and the incorrect Fixes reference is gone. Not merging yet: the fork workflows still need maintainer approval to run, and the build legs should be green before this lands.

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build legs ran today (run 36079133044). The cleanup step worked on every leg: amd64 stable went from 86G to 125G free, arm64 from 108G to 127G. All amd64 legs and arm64 gdx passed. The three red jobs don't come from this change. arm64 main lts fails at podman pull ghcr.io/ublue-os/bluefin:lts with no image found in image index for architecture arm64. That tag lists only amd64, the same error hit another branch on 8 Aug, and main soft-fails arm64 since #70. Both Stable E2E jobs time out because the kickstart never reaches anaconda (Kickstart file /run/install/ks.cfg is missing. in the serial log). They have failed the same way on every run since 7 Aug. Approving.

@castrojo

Copy link
Copy Markdown
Contributor

Ready to merge under R2 / org policy (CI change, independent human reviews): approved by @hanthor and @Danathar on head 3553fd5. The three red checks are pre-existing and unrelated to this diff. Build LTS arm64 main fails at podman pull ghcr.io/ublue-os/bluefin:lts with "no image found in image index for architecture arm64". Stable main and NVIDIA E2E time out at "Installer did not report completion" because the kickstart is never delivered (tracked in #74). The same three jobs failed on run 31227505330 (2026-08-07, unrelated branch). The pin 695eb75 is verified as ublue-os/remove-unwanted-software master HEAD (2025-10-10, merge of the v10 branch; tags stop at v9, which has no extra-squeeze), and its action.yml exposes extra-squeeze. The disk cleanup step succeeded on every leg of run 36079133044. The merge attempt was rejected because the merging token lacks workflow scope (this PR edits .github/workflows/reusable-build-iso-anaconda.yml), so it needs a merge by someone whose token has workflow scope.

castrojo pushed a commit that referenced this pull request Sep 29, 2026
* fix(e2e): deliver the kickstart through an initramfs overlay

The stable E2E jobs have failed on every run since 2026-08-07 with
"Kickstart file /run/install/ks.cfg is missing." The harness boots the
ISO's extracted vmlinuz/initramfs with inst.ks=http://..., but that boot
option only sets --ks=/run/install/ks.cfg inside anaconda (store_const
in pyanaconda/argument_parsing.py); fetching the URL into that path is
the job of anaconda's dracut module, which the titanoboa initramfs does
not carry. Nothing ever wrote the file, anaconda aborted, and the 1800s
budget timed out.

Append a gzip cpio overlay to the initramfs carrying the rendered
kickstart at /etc/e2e/ks.cfg plus an initrd unit (wanted by
initrd.target, ordered before initrd-cleanup.service) that copies it to
/run/install/ks.cfg, which survives switch-root - the same hand-off
anaconda-lib.sh:parse_kickstart() performs. Point inst.ks= at the
embedded file and drop the HTTP server nothing fetched from.

Raise the default install budget to 2700s: anaconda's optional
Subscription module fails to activate in this image and dbus waits its
full 600s service_start_timeout before installation starts, which the
old 1800s budget did not account for.

Refs #69

* fix(e2e): take the kickstart image ref from the ISO's container store

Run 36276284373 proved kickstart delivery (e2e-kickstart.service finished
at 19s, anaconda ran the kickstart to the payload step) and then failed in
`ostree container image deploy --transport=containers-storage` with
"reference ... ghcr.io/ublue-os/bluefin:stable does not resolve to an
image ID".

The store name is not guessable from IMAGE_REF/IMAGE_TAG: titanoboa
`podman pull`s whatever the build passed, so the nvidia-open ISO carries
ghcr.io/ublue-os/bluefin-nvidia-open:stable while the workflow asked the
kickstart for bluefin:stable. Read the stored names out of the ISO
(unsquashfs on /LiveOS/squashfs.img at its LBA offset ->
var/lib/containers/storage/overlay-images/images.json) and render the
single --url from that; IMAGE_REF:IMAGE_TAG only disambiguates when an ISO
carries several images, and a store with no match fails loudly.

The kickstart now also prints `e2e-pre:` lines (podman images, skopeo
inspect of the chosen URL) to the serial log so the installer's own view
of the store is in the evidence, and the workflow enables /dev/kvm for the
runner so accel=kvm:tcg stops falling back to a ~45-minute TCG install.

* test(e2e): log the installer's container store right before the payload step

c/image's storage transport reports any store error while resolving a name
as 'does not resolve to an image ID', so record podman images / skopeo
inspect / the storage mounts from a %pre-install block as well, which runs
immediately before 'ostree container image deploy'.

---------

Co-authored-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

@castrojo friendly ping, since this has been idle since 09-29: it is still approved by hanthor and me at 3553fd5, carries lgtm, and iso has no required checks, so it only needs someone with write access to merge it. If you would rather it be rebased onto current main first (main has moved since #74), say so and @bketelsen can update it.

@castrojo
castrojo merged commit f5988f2 into projectbluefin:main Oct 9, 2026
17 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants