Repository navigation
fix(image): stop shipping the 4 GB package repository in every image, and check Hummingbird signatures - #133
Conversation
…ignatures
Mapping the published image's layers to its config history showed what the
"3 GB GNOME layer" was:
layer 34 4064 MiB COPY --from=packages /repository /etc/utah-packages
layer 41 979 MiB the actual package transaction
total 5.94 GiB 45 layers, no chunkah annotations of Utah's own
The Containerfile copied the package factory's entire RPM repository into
the image so dnf could install from file:///etc/utah-packages, and nothing
ever removed it: two thirds of every pull, 4 GB under /etc on every
installed system, and the largest single contributor to the 7.9 GB ISO,
whose squashfs holds the image (#130, #128).
The repository is now bind mounted into the two RUN steps that install
from it -- the main transaction and the flavor step that builds the OGC
kernel and NVIDIA module -- and never committed to a layer. The pinned
`packages` stage is still the only source the transaction can see, which
is what the old COPY comment meant by reproducible. The flavor step, the
last one that installs anything, flips utah-packages.repo to enabled=0 so
later dnf calls on the image (the live ISO build's included) do not fail
on a baseurl that is not there. just check now refuses a
COPY --from=packages.
Hummingbird's RPMs are signed with Red Hat's release key 2 (RSA/SHA256,
key ID 199e2f91fd431d51, verified against a downloaded package), but
packages/hummingbird.repo carried gpgcheck=0 copied from upstream's
lockfile-tooling repo file, so the build logged
Warning: skipped OpenPGP checks for 702 packages
The key is committed (fingerprint 567E 347A D004 4ADE 55BA 8A5F 199E 2F91
FD43 1D51), installed under /etc/pki/rpm-gpg in both Containerfiles, and
the repo file runs with gpgcheck=1. The factory repository stays at
gpgcheck=0: its trust root is the pinned image's OCI provenance, as its
file says. Changing hummingbird.repo rotates the kernel-cache tag, so the
first gaming build after this compiles the kernel once.
The live ISO's dracut run failed on every build,
dracut-install: ERROR: installing '/root' to '/root'
dracut[E]: FAILED: ... dracut-install -D ... -f /root /root
because /root is a symlink to /var/roothome and the target does not exist
at build time, and dracut still exited 0 so the build went on with a
module silently dropped (#132). The target now exists for the duration
of the run, the run is under pipefail, and a dracut[E] FAILED line fails
the step.
configure-services.sh no longer claims Utah is rechunked; it is not,
because reusable-build skips chunkah on the testing stream (#131), which
needs a change in projectbluefin/actions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…an option
CI's contract job failed on the previous commit with
grep: unrecognized option '--mount=type=bind,...'
because the pattern starts with two dashes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
There was a problem hiding this comment.
Correctness review (head 04ab0d7)
-
The new
just checkguard againstCOPY --from=packagesis a no-op (Justfile:82). Thecheckrecipe is a#!/usr/bin/env bash/set -euo pipefailscript (Justfile:70-71), and errexit explicitly exempts pipelines prefixed with!— a failing! grep -q ...mid-script neither stops the script nor affects its exit status. Verified empirically:bash -c 'set -euo pipefail; ! grep -q RUN Containerfile; echo reached'printsreachedand exits 0. So the PR body's claim that "just checknow refuses aCOPY --from=packages" is not implemented; only the unit test (tests/test_package_resolution.py:169) actually enforces it. Fix:if grep -q 'COPY --from=packages' Containerfile; then exit 1; fi. -
docs/skills/containerfile.md:48-49 describes a design the diff does not implement. It says
packages/utah-packages.repo"is committed withenabled=0… and they enable the repo explicitly." The file is committed withenabled=1(packages/utah-packages.repo:12), no step enables it, and the flavor step disables it afterwards (Containerfile:201). The doc contradicts both the code and the new test, which assertsenabled=1(tests/test_package_resolution.py:180).
Minor, verified non-issues: configure-services.sh:128-129 runs dnf remove in a step without the mount while the repo is still enabled=1 with a missing baseurl — dnf5 remove does not load available repos, so this should not fail, but it is the one dnf call between the two mounted steps and worth watching in CI. The set -euxo pipefail dracut guard relies on /bin/sh being bash, which holds on this base.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Green on all four flavors, and the effect is measured rather than predicted. Comparing the image-export artifact the
4.57 GB smaller, a 68% reduction, from removing one The bind mount is on both install steps in the real build: and the transaction still resolves 719 packages from Signature checking is working too. The warning went from to so 259 Hummingbird packages are now verified against Red Hat's release key, and the only packages still unchecked are the factory's own, whose trust root is the pinned image's OCI provenance by design. The first gaming build after this did pay the kernel-cache rebuild, as expected, and passed. @castrojo ready for review when you have a moment. The ISO effect follows on its own: its squashfs holds this image, so #128 should land near 3.7 GB. Generated by Claude Code |
… repo file The review on #133 is right on both counts. `! grep -q 'COPY --from=packages' Containerfile` enforced nothing. errexit exempts any pipeline beginning with `!`, so a failing one neither stops the script nor changes its exit status: $ bash -c 'set -euo pipefail; ! grep -q RUN Containerfile; echo reached' reached $ echo $? 0 So the PR body claimed a gate the recipe did not have; only the unit test enforced it. Both `!` guards in `check` are now explicit `if ... exit 1` blocks with a message that says what to do. The workflow-literal guard on the following line had been working by accident: it was the last command in the recipe, so its status was the recipe's status. That is a property of where it sits in the file, not of what it does, and any line appended after it silently switches it off. Worth knowing for #144, which appends exactly such a line. The containerfile skill also described a design this repository does not have. It said utah-packages.repo is committed with enabled=0 and that the two mounted steps enable it explicitly. It is committed with enabled=1 -- the repo file's own comment says why -- and the flavor step flips it to 0 afterwards so later dnf calls on the finished image do not fail on a baseurl that is no longer mounted. The doc now says that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
main has moved a long way under this branch: the bind mount, the enabled=0
flip and its verification, and the repository-identity work all landed while
this was open. Resolved in main's favour everywhere the two describe the same
change, keeping only what is still this branch's own: the Red Hat release key
that lets hummingbird.repo run with gpgcheck=1, and the explicit-if guards.
The merge also tripped the gate that landed this morning:
FAIL: test_every_copied_build_input_changes_the_key
(source='packages/RPM-GPG-KEY-redhat-release-2')
AssertionError: Containerfile.kernel copies
packages/RPM-GPG-KEY-redhat-release-2 into the cache image, but editing it
does not change the cache tag, so CI reuses a cache image built from the
old packages/RPM-GPG-KEY-redhat-release-2.
Which is exactly right, and exactly this branch's bug: Containerfile.kernel
copies the key into the builder, so a rotated key is a different build root,
and scripts/kernel-cache-tag.sh did not hash it. Without that test the cache
tag would have stayed put across a key rotation and the three cached flavors
would have kept unpacking a kernel built against the old one. The key is now
in the hashed set.
tests/test_kernel_cache_tag.py needed its HASHED_INPUTS list extended too,
since it restates the inputs rather than deriving them; that is worth noting
as a wart, because forgetting it turns one clear failure into five confusing
ones.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
|
Both findings were right and are fixed in 1. The So the PR body claimed a gate the recipe did not have. Both Worth recording the part that goes beyond this PR: the other 2. The containerfile skill described a design this repository does not have. It said The merge also caught a bug in this branch, via the gate that landed this morning (#166):
One wart it exposed: Suite is green on the merged result: Generated by Claude Code |
|
#156 merged this morning and carries the 4 GB-repository removal from this PR ( |
There was a problem hiding this comment.
Reviewed (correctness, at 14bdb57) — no blockers.
Verified: the committed packages/RPM-GPG-KEY-redhat-release-2 is byte-identical to Red Hat's published fd431d51 key, and the new tests' assertions (bind mount on both install steps, enabled=0 flip at Containerfile:209, dracut guard) all match the head Containerfiles.
One note: the PR body's first "What changed" bullet (COPY → bind mount, enabled=0 flip) is already on main at the merge base (a2cfbc2, Containerfile:120,189,205) — this diff adds only the enforcement (Justfile gate, tests, docs), not the conversion itself. Worth knowing when reading the changelog. Minor: the new comment in scripts/kernel-cache-tag.sh:29 references tests/test_kernel_cache_key.py; the file is tests/test_kernel_cache_tag.py.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
There was a problem hiding this comment.
Approving at head 14bdb57. Following up on my comment above: with main merged in, the 4 GB repository removal from #156 is no longer part of this diff, so what is left here is the Hummingbird signature check, the dracut gate for the live ISO, the Justfile guards that now actually fail, and the kernel-cache tag hashing the key. That is the rebased-down version I suggested, and this one commits the key file, which #157 does not, so I am fine with this landing.
What I checked. The committed packages/RPM-GPG-KEY-redhat-release-2 is byte-identical to Red Hat's published key: I fetched https://www.redhat.com/security/fd431d51.txt and cmp reports no difference (sha256 d4b25436...c742b on both), and gpg --show-keys gives fingerprint 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 for the committed file, the download, and the copy in /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release on my Fedora host. The key is copied into both Containerfile and Containerfile.kernel, hummingbird.repo points gpgkey at that path, and the live ISO builds on top of the image so it inherits it. python3 tests/run_suite.py at this head: 199 tests, OK (1 skipped). I ran the two rewritten Justfile guards by hand under set -euo pipefail and they exit 1 when the pattern matches. All 14 checks are green on this head. One caveat: dispatch-iso skipped and no ISO workflow has run on this branch, so the dracut change in iso/live/Containerfile is verified by reading and by the unit test only, not by a real ISO build.
One non-blocking nit: the new comment in scripts/kernel-cache-tag.sh line 29 names tests/test_kernel_cache_key.py; the file is tests/test_kernel_cache_tag.py.
packages/nvidia-container.repo shipped enabled=1 and was copied into every image via the Containerfile's yum.repos.d COPY, so a plain utah or utah-gaming install exposed NVIDIA's third-party runtime repository even though scripts/install-nvidia.sh -- the only thing that installs from it -- runs solely on the nvidia flavors. That contradicts the policy that Utah's runtime repositories are Hummingbird plus the pinned utah-packages factory. Ship the repo enabled=0 everywhere (the COPY stays; it is harmless disabled) and turn it on for the single install transaction on the NVIDIA flavors, --enablerepo=nvidia-container-toolkit, the same narrow window #133 gives utah-packages.repo. Non-NVIDIA images now list only Hummingbird in `dnf repolist`, while the NVIDIA flavors still install nvidia-container-toolkit with repo_gpgcheck=1 and the pinned fingerprint intact. Assisted-by: lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K Co-authored-by: Claude https://claude.ai/ghost
linux-firmware joins [parity] so iwlwifi and friends load their blobs; NetworkManager-wifi stays out pending utah-packages#136 (and the stale #133 reference now points at the real issue). No enabled repository packages a terminal, so ptyxis is recorded under [unavailable] tracked by utah-packages#224, while the live session puts Ghostty on the dash and sets its Flatpak export as the default terminal handler. Assisted-by: Muse Code powered by Meta Muse Spark via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
linux-firmware joins [parity] so iwlwifi and friends load their blobs; NetworkManager-wifi stays out pending utah-packages#136 (and the stale #133 reference now points at the real issue). No enabled repository packages a terminal, so ptyxis is recorded under [unavailable] tracked by utah-packages#224, while the live session puts Ghostty on the dash and sets its Flatpak export as the default terminal handler. Assisted-by: Muse Code powered by Meta Muse Spark via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Why
Mapping the published image's layers (
ghcr.io/projectbluefin/utah@sha256:0ffe7751…) to its config history showed what the "3 GB GNOME layer" was:COPY --from=packages /repository /etc/utah-packagesThe Containerfile copied the package factory's entire RPM repository into the image so dnf could install from
file:///etc/utah-packages, and nothing removed it: two thirds of every pull, 4 GB under/etcon every installed system, and the largest contributor to the 7.9 GB ISO (#130, #128).What changed
RUN --mount=type=bind,from=packages,source=/repository,target=/etc/utah-packages,roon the two steps that install from it (main transaction, flavor step). The pinnedpackagesstage is still the only source the transaction sees. The flavor step, the last one that installs anything, flipsutah-packages.repotoenabled=0so later dnf calls on the image (the live ISO build's included) do not fail on a missing baseurl.just checknow refuses aCOPY --from=packages. Closes image: the 4 GB factory RPM repository is copied into /etc/utah-packages and never removed #130.199e2f91fd431d51, verified against a downloaded package withrpmkeys -K).packages/hummingbird.repocarriedgpgcheck=0copied from upstream's lockfile-tooling file, so the build loggedskipped OpenPGP checks for 702 packages. The key is committed aspackages/RPM-GPG-KEY-redhat-release-2(fingerprint567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51), installed in both Containerfiles, and the repo runs withgpgcheck=1. The factory repo stays atgpgcheck=0by its documented design. Changinghummingbird.reporotates the kernel-cache tag, so the first gaming build after this compiles the kernel once (~45 min)./rootis a symlink to/var/roothomewith no target at build time; dracut loggedFAILED … -f /root /rooton every build and exited 0 (ci: the image build logs errors nobody reads — dracut fails in the live initramfs, trivy cannot scan Hummingbird, signatures are never checked #132 item 1). The target now exists during the run, the run is underpipefail, and adracut[E]: FAILEDline fails the step.configure-services.shno longer claims Utah is rechunked; it is not (ci: Utah images are never rechunked, because reusable-build skips chunkah on the testing stream #131, needs aprojectbluefin/actionschange I do not have push access to).Not addressed here: trivy's "OS family none" (#132 item 2) lives in the reusable workflow's scan step.
Verification
gpgcheck=1, and the dracut guard.mainimage should drop to roughly 1.9 GiB. Expected ISO after this: about 3.7 GB.🤖 Generated with Claude Code
https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Generated by Claude Code