Skip to content

fix: stop shipping the 4 GB package repository in every image (#128) - #156

Merged
hanthor merged 3 commits into
projectbluefin:mainfrom
kylerankin:fix/iso-budget
Sep 19, 2026
Merged

hanthor merged 3 commits into
projectbluefin:mainfrom
kylerankin:fix/iso-budget

Conversation

@kylerankin

@kylerankin kylerankin commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

#128: live ISO grew 7.7G -> 8.6G between the last passing runs

Root cause

The live ISO grew from 7.7G (2026-09-06, last passing run) to 8.6G
(2026-09-18) because the Containerfile copied the digest-pinned packages
stage's RPM repository into a layer that nothing ever removed:

COPY --from=packages /repository /etc/utah-packages

That ~4 GB is two thirds of every published image and of every live ISO,
whose squashfs holds the whole mounted image. It is not a package bump -- it is
the same ~4 GB repository, now committed to the image instead of just mounted
for the install transaction.

Fix

Bind the repository into the two RUN steps that install from it with a
BuildKit mount, so it costs no layer and leaves nothing on disk:

RUN --mount=type=bind,from=packages,source=/repository,target=/etc/utah-packages,ro \

Reproducibility is unchanged: the packages stage is still digest-pinned and
is the only source the package transaction sees -- that is what the old
comment meant by "does not depend on a mutable Pages mirror." The flavor step
now flips utah-packages.repo to enabled=0 last, so later dnf calls on the
image (the live ISO build's included) do not fail on a file:// baseurl that
no longer exists.

Guard added

iso/scripts/build-iso.sh now fails closed above ISO_MAX_GB (override per
run with UTAH_ISO_MAX_GB) once the ISO is written, comparing byte counts so
the GB ceiling is exact. This is the explicit guard the issue asks for: drift
stops landing silently. The ceiling is 8 GB -- deliberately between the
last passing run (7.7G) and the grown size (8.6G), so the exact #128
regression fails the job instead of landing silently; a ceiling at or above
8.6G would be toothless against it. There is headroom for #105 (linux-firmware

  • parity packages); raise N only after that PR is evaluated.

just check now asserts the mount and refuses any COPY --from=packages.

Scope

Focused on #128 only. Deliberately not included here (separate issues):

Verification

just check passes offline (script syntax, skill front-matter/index,
download integrity, the package-repo mount assertions, and a new unit test
that runs the budget guard with du stubbed). The image build itself could
not be run in this environment (no podman); the change matches the proven
RUN --mount fix the maintainer team already landed, and the build's first
gate is just check, which is green. The mount requires BuildKit mode, which
the upstream bootc-build runner provides.

— hive: backend=pi model=lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K

…tbluefin#128)

The live ISO grew from 7.7G (2026-09-06, last passing run) to 8.6G
(2026-09-18) between builds. Root cause: the Containerfile copied the
digest-pinned `packages` stage's RPM repository into a layer with
`COPY --from=packages /repository /etc/utah-packages`, and nothing ever
removed it. That ~4 GB is two thirds of every published image and of every
live ISO, whose squashfs holds the whole mounted image.

Bind the repository into the two RUN steps that install from it with a
BuildKit `RUN --mount=type=bind,from=packages`, so it costs no layer and
leaves nothing on disk. Reproducibility still comes from the digest-pinned
`packages` stage, the only source the package transaction sees. The flavor
step flips `utah-packages.repo` to enabled=0 last, so later dnf calls on the
image (the live ISO build's included) do not fail on a file:// baseurl that
no longer exists.

Add a size budget to iso/scripts/build-iso.sh that fails closed above
ISO_MAX_GB (override UTAH_ISO_MAX_GB) once the ISO is written, so drift
stops landing silently. Raise N only after projectbluefin#105 (linux-firmware + parity
packages) is evaluated.

just check asserts the mount and refuses a `COPY --from=packages`.

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (read at head 8db7766 — the SHA I was dispatched with, 5131d69, has been superseded).

High — the ISO budget guard makes every ISO build fail, not just oversized ones. The guard block (iso/scripts/build-iso.sh:176-181) sits inside the <<'ASSEMBLY' quoted heredoc that is executed by a separate process, podman unshare bash -s (line 60). ISO_MAX_GB is assigned in the outer script (line 23) but never exported, and the quoted heredoc suppresses outer expansion — so inside the assembly shell, which runs set -euo pipefail (line 61), line 176 iso_max_bytes=$(( ISO_MAX_GB * 1024 * 1024 * 1024 )) dies with ISO_MAX_GB: unbound variable. Verified: ISO_MAX_GB=8 bash -s <<'EOF' ... set -u; $((ISO_MAX_GB*1024)) ... EOF → exit 1, unbound variable. Every run fails after the ISO is written, and the UTAH_ISO_MAX_GB override never reaches the guard either. Fix: pass ISO_MAX_GB as an eighth positional arg alongside the existing seven (line 60/62-68), or export it.

Medium — the new test masks exactly this bug. tests/test_iso_ci.py:100-119 extracts only the guard block and then re-defines ISO_MAX_GB=${ISO_MAX_GB:-8} in its own harness before running it, so the test passes while the real script cannot. A test that ran the guard in a context matching the ASSEMBLY heredoc (unset var + set -u) would have caught it; at minimum the harness should not inject the variable the script under test is responsible for providing.

Minor: sed -i 's/^enabled=1$/enabled=0/' (Containerfile:204) flips every enabled=1 line in utah-packages.repo; fine if the file has a single section, but a section-anchored match would be safer if one is ever added.

The Containerfile mount conversion itself (Containerfile:120-121, 189-190) and the just check assertions (Justfile:67-70) look correct for the stated goal.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 8db7766)

High — the new budget guard makes every ISO build fail, even under budget. The guard added at the bottom of iso/scripts/build-iso.sh (lines 173–181) runs inside the <<'ASSEMBLY' quoted heredoc executed by podman unshare bash -s (line 60). That inner bash runs set -euo pipefail (line 61) and receives only the 7 positional args (lines 62–68); ISO_MAX_GB is a plain, unexported assignment in the outer script (line 23), so it is not in the inner shell's environment. The first guard line, iso_max_bytes=$(( ISO_MAX_GB * 1024 * 1024 * 1024 )), therefore dies with ISO_MAX_GB: unbound variable on every run — after xorriso has already written the ISO — so the build exits 1 regardless of size, and the UTAH_ISO_MAX_GB override can never work either. Reproduced:

$ bash -c 'set -u; ISO_MAX_GB=8; bash -s <<'\''H'\''
set -euo pipefail
echo $(( ISO_MAX_GB * 1024 ))
H'
bash: line 2: ISO_MAX_GB: unbound variable

Fix: pass ISO_MAX_GB as an 8th positional arg alongside the others on line 60 (or export ISO_MAX_GB before the heredoc).

Medium — the new test cannot catch this. tests/test_iso_ci.py::test_iso_budget_guard_fails_closed_above_ceiling extracts the guard block and re-runs it with ISO_MAX_GB injected by the test harness itself ("ISO_MAX_GB=${ISO_MAX_GB:-8}; ..."), so it validates the arithmetic but masks exactly the variable-scoping failure above. After fixing the plumbing, consider asserting the variable is actually delivered into the heredoc (e.g., grep for it in the podman unshare bash -s -- arg list).

The Containerfile change itself looks correct: both installing RUN steps get the mount (Containerfile:120, 189), no intermediate RUN touches the repo while enabled=1 with the baseurl absent, and the sed disable (Containerfile:205) matches the literal enabled=1 in packages/utah-packages.repo.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

The budget guard (projectbluefin#128) lived inside the <<'ASSEMBLY' quoted heredoc, which
runs under 'podman unshare bash -s' in a separate set -u shell that only
received the seven positional args. ISO_MAX_GB was a plain, unexported
assignment in the outer script, so the guard died with 'ISO_MAX_GB: unbound
variable' on every build -- after the ISO was already written -- and the
UTAH_ISO_MAX_GB override never reached it either.

Pass ISO_MAX_GB as an eighth positional arg alongside the others. Update the
budget guard test, which injected ISO_MAX_GB itself and thus masked exactly
this scoping failure, to stop injecting it and to assert the variable is
actually delivered into the heredoc arg list.

Fixes review feedback (projectbluefin#156).
@kylerankin

Copy link
Copy Markdown
Contributor Author

Thanks for the review -- this was a real bug and both findings pointed at the same root cause.

High (plumbing): ISO_MAX_GB was set in the outer script but never exported, and the <<'ASSEMBLY' quoted heredoc runs under podman unshare bash -s in its own set -u shell with only the seven positional args, so the guard died with ISO_MAX_GB: unbound variable on every build after the ISO was already written (and the UTAH_ISO_MAX_GB override never reached it). Fixed by passing ISO_MAX_GB as an eighth positional arg alongside the others and reading it as \$8 inside the assembly.

Medium (test): the budget guard test injected ISO_MAX_GB=\${ISO_MAX_GB:-8} itself, which masked exactly this scoping failure -- it passed while the real script could not run. Removed the injection (the script under test now has to supply the variable the way it does in reality), provide it via the test env instead, and added assertions that the variable is actually delivered into the podman unshare bash -s -- arg list. Verified the updated test fails if the arg passing is broken again.

The minor sed/section-anchoring note is noted but left as-is: utah-packages.repo has a single section by construction, and anchoring it would couple the disable to a section header that would itself need regenerating. Happy to add the anchor if a multi-section variant is ever planned.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed for correctness (head d43e4e1). Verified locally: tests/test_iso_ci.py passes (11/11), bash -n iso/scripts/build-iso.sh OK, the Justfile grep assertions hold, and the traced dnf call sites are consistent with the fix — installs happen inside the two mounted RUNs (Containerfile:120, :189) while the repo is still enabled, and the only post-disable dnf call (iso/live/Containerfile:21) is covered by the enabled=0 flip at Containerfile:206.

Two low-severity notes, neither blocking:

  1. Silent-no-op risk in the sed flip (Containerfile:206): sed -i 's/^enabled=1$/enabled=0/' matches today's packages/utah-packages.repo:7, but if that file's formatting ever changes (e.g. enabled = 1), the sed no-ops silently and the live-ISO dnf call fails later with a confusing missing-file://-baseurl error. A grep -q '^enabled=0$' … assertion after the sed would fail fast at the right place.

  2. Test relies on bash's unset-PATH fallback (tests/test_iso_ci.py:120-125): the guard test runs bash -eu -c with an env containing only ISO_MAX_GB/DU_BYTES/DU_HUMAN — no PATH — yet the extracted guard pipes through cut (iso/scripts/build-iso.sh:178). It works because bash falls back to a compiled-in default PATH, but that's implementation-defined; passing PATH through in the test env would make it robust.

Limit of this review: I could not run the actual image build (no podman here); the RUN --mount=type=bind,from=… behavior is verified by inspection only and will be exercised by the PR's own CI build.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…a PATH

- Containerfile: assert enabled=0 actually took after the sed so a
  formatting change can't silently leave the repo enabled and break a
  later dnf call on a missing file:// baseurl.
- test_iso_ci: pass PATH into the budget-guard harness so it no longer
  relies on bash's unset-PATH fallback for the cut pipe.
@kylerankin

Copy link
Copy Markdown
Contributor Author

Addressed the two low-severity notes from the last review:

  1. Silent-no-op risk (Containerfile) — added && grep -q '^enabled=0$' /etc/yum.repos.d/utah-packages.repo right after the sed, so a future formatting change to the repo file fails fast at the flip instead of surfacing later as a confusing missing-file://-baseurl error.

  2. Test PATH fallback (test_iso_ci.py) — the budget-guard harness now passes PATH through in its env, so the cut pipe no longer depends on bash's unset-PATH compiled-in default.

Both existing budget-guard assertions still pass.

🐝 Hive Agent: contributor | SHA: 37e03e2

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants