Skip to content

fix(image): stop shipping the 4 GB package repository in every image, and check Hummingbird signatures - #133

Merged
hanthor merged 4 commits into
mainfrom
claude/image-size
Sep 20, 2026
Merged

hanthor merged 4 commits into
mainfrom
claude/image-size

Conversation

@hanthor

@hanthor hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Why

Mapping the published image's layers (ghcr.io/projectbluefin/utah@sha256:0ffe7751…) to its config history showed what the "3 GB GNOME layer" was:

layer compressed produced by
34 4064 MiB COPY --from=packages /repository /etc/utah-packages
41 979 MiB the actual package transaction
total 5.94 GiB 45 layers, none of them Utah's own chunkah output

The Containerfile copied the package factory's entire RPM repository into the image so dnf could install from file:///etc/utah-packages, and nothing removed it: two thirds of every pull, 4 GB under /etc on every installed system, and the largest contributor to the 7.9 GB ISO (#130, #128).

What changed

  • Bind mount instead of COPY. RUN --mount=type=bind,from=packages,source=/repository,target=/etc/utah-packages,ro on the two steps that install from it (main transaction, flavor step). The pinned packages stage is still the only source the transaction sees. The flavor step, the last one that installs anything, flips utah-packages.repo to enabled=0 so later dnf calls on the image (the live ISO build's included) do not fail on a missing baseurl. just check now refuses a COPY --from=packages. Closes image: the 4 GB factory RPM repository is copied into /etc/utah-packages and never removed #130.
  • Hummingbird signatures are checked. Their RPMs are signed with Red Hat release key 2 (199e2f91fd431d51, verified against a downloaded package with rpmkeys -K). packages/hummingbird.repo carried gpgcheck=0 copied from upstream's lockfile-tooling file, so the build logged skipped OpenPGP checks for 702 packages. The key is committed as packages/RPM-GPG-KEY-redhat-release-2 (fingerprint 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51), installed in both Containerfiles, and the repo runs with gpgcheck=1. The factory repo stays at gpgcheck=0 by its documented design. Changing hummingbird.repo rotates the kernel-cache tag, so the first gaming build after this compiles the kernel once (~45 min).
  • dracut failures fail the live ISO build. /root is a symlink to /var/roothome with no target at build time; dracut logged FAILED … -f /root /root on every build and exited 0 (ci: the image build logs errors nobody reads — dracut fails in the live initramfs, trivy cannot scan Hummingbird, signatures are never checked #132 item 1). The target now exists during the run, the run is under pipefail, and a dracut[E]: FAILED line fails the step.
  • configure-services.sh no longer claims Utah is rechunked; it is not (ci: Utah images are never rechunked, because reusable-build skips chunkah on the testing stream #131, needs a projectbluefin/actions change I do not have push access to).

Not addressed here: trivy's "OS family none" (#132 item 2) lives in the reusable workflow's scan step.

Verification

  • 85 unit tests pass, including new ones pinning the mount on both install steps, the enabled=0 flip, the key and gpgcheck=1, and the dracut guard.
  • The Justfile check greps pass locally.
  • The real proof is CI: the image build must resolve and install with the mount and with signature checks on, and the main image should drop to roughly 1.9 GiB. Expected ISO after this: about 3.7 GB.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M


Generated by Claude Code

…ignatures

Mapping the published image's layers to its config history showed what the
"3 GB GNOME layer" was:

    layer 34   4064 MiB   COPY --from=packages /repository /etc/utah-packages
    layer 41    979 MiB   the actual package transaction
    total     5.94 GiB    45 layers, no chunkah annotations of Utah's own

The Containerfile copied the package factory's entire RPM repository into
the image so dnf could install from file:///etc/utah-packages, and nothing
ever removed it: two thirds of every pull, 4 GB under /etc on every
installed system, and the largest single contributor to the 7.9 GB ISO,
whose squashfs holds the image (#130, #128).

The repository is now bind mounted into the two RUN steps that install
from it -- the main transaction and the flavor step that builds the OGC
kernel and NVIDIA module -- and never committed to a layer. The pinned
`packages` stage is still the only source the transaction can see, which
is what the old COPY comment meant by reproducible. The flavor step, the
last one that installs anything, flips utah-packages.repo to enabled=0 so
later dnf calls on the image (the live ISO build's included) do not fail
on a baseurl that is not there. just check now refuses a
COPY --from=packages.

Hummingbird's RPMs are signed with Red Hat's release key 2 (RSA/SHA256,
key ID 199e2f91fd431d51, verified against a downloaded package), but
packages/hummingbird.repo carried gpgcheck=0 copied from upstream's
lockfile-tooling repo file, so the build logged

    Warning: skipped OpenPGP checks for 702 packages

The key is committed (fingerprint 567E 347A D004 4ADE 55BA 8A5F 199E 2F91
FD43 1D51), installed under /etc/pki/rpm-gpg in both Containerfiles, and
the repo file runs with gpgcheck=1. The factory repository stays at
gpgcheck=0: its trust root is the pinned image's OCI provenance, as its
file says. Changing hummingbird.repo rotates the kernel-cache tag, so the
first gaming build after this compiles the kernel once.

The live ISO's dracut run failed on every build,

    dracut-install: ERROR: installing '/root' to '/root'
    dracut[E]: FAILED: ... dracut-install -D ... -f /root /root

because /root is a symlink to /var/roothome and the target does not exist
at build time, and dracut still exited 0 so the build went on with a
module silently dropped (#132). The target now exists for the duration
of the run, the run is under pipefail, and a dracut[E] FAILED line fails
the step.

configure-services.sh no longer claims Utah is rechunked; it is not,
because reusable-build skips chunkah on the testing stream (#131), which
needs a change in projectbluefin/actions.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…an option

CI's contract job failed on the previous commit with
    grep: unrecognized option '--mount=type=bind,...'
because the pattern starts with two dashes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 04ab0d7)

  1. The new just check guard against COPY --from=packages is a no-op (Justfile:82). The check recipe is a #!/usr/bin/env bash / set -euo pipefail script (Justfile:70-71), and errexit explicitly exempts pipelines prefixed with ! — a failing ! grep -q ... mid-script neither stops the script nor affects its exit status. Verified empirically: bash -c 'set -euo pipefail; ! grep -q RUN Containerfile; echo reached' prints reached and exits 0. So the PR body's claim that "just check now refuses a COPY --from=packages" is not implemented; only the unit test (tests/test_package_resolution.py:169) actually enforces it. Fix: if grep -q 'COPY --from=packages' Containerfile; then exit 1; fi.

  2. docs/skills/containerfile.md:48-49 describes a design the diff does not implement. It says packages/utah-packages.repo "is committed with enabled=0 … and they enable the repo explicitly." The file is committed with enabled=1 (packages/utah-packages.repo:12), no step enables it, and the flavor step disables it afterwards (Containerfile:201). The doc contradicts both the code and the new test, which asserts enabled=1 (tests/test_package_resolution.py:180).

Minor, verified non-issues: configure-services.sh:128-129 runs dnf remove in a step without the mount while the repo is still enabled=1 with a missing baseurl — dnf5 remove does not load available repos, so this should not fail, but it is the one dnf call between the two mounted steps and worth watching in CI. The set -euxo pipefail dracut guard relies on /bin/sh being bash, which holds on this base.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Green on all four flavors, and the effect is measured rather than predicted. Comparing the image-export artifact the build_main / image (main, utah, testing, x86_64) job uploads, same job on both sides:

build commit artifact
before, job 105519468973 8a0f93f 6,705,711,010 bytes (6.24 GiB)
after, job 105699109326 04ab0d7 2,137,190,003 bytes (1.99 GiB)

4.57 GB smaller, a 68% reduction, from removing one COPY.

The bind mount is on both install steps in the real build:

[4/4] STEP 11/32: RUN --mount=type=bind,from=packages,source=/repository,target=/etc/utah-packages,ro     /usr/local/libexec/utah-install-packages ...
[4/4] STEP 24/32: RUN --mount=type=bind,from=packages,source=/repository,target=/etc/utah-packages,ro     case "${IMAGE_FLAVOR}" in       gaming|nvidia-gaming) ...

and the transaction still resolves 719 packages from --enablerepo=utah-packages --enablerepo=public-hummingbird-x86_64-rpms.

Signature checking is working too. The warning went from

Warning: skipped OpenPGP checks for 702 packages from repositories: public-hummingbird-x86_64-rpms, utah-packages

to

Warning: skipped OpenPGP checks for 443 packages from repository: utah-packages

so 259 Hummingbird packages are now verified against Red Hat's release key, and the only packages still unchecked are the factory's own, whose trust root is the pinned image's OCI provenance by design.

The first gaming build after this did pay the kernel-cache rebuild, as expected, and passed.

@castrojo ready for review when you have a moment. The ISO effect follows on its own: its squashfs holds this image, so #128 should land near 3.7 GB.


Generated by Claude Code

hanthor and others added 2 commits September 19, 2026 11:16
… repo file

The review on #133 is right on both counts.

`! grep -q 'COPY --from=packages' Containerfile` enforced nothing. errexit
exempts any pipeline beginning with `!`, so a failing one neither stops the
script nor changes its exit status:

    $ bash -c 'set -euo pipefail; ! grep -q RUN Containerfile; echo reached'
    reached
    $ echo $?
    0

So the PR body claimed a gate the recipe did not have; only the unit test
enforced it. Both `!` guards in `check` are now explicit `if ... exit 1`
blocks with a message that says what to do.

The workflow-literal guard on the following line had been working by accident:
it was the last command in the recipe, so its status was the recipe's status.
That is a property of where it sits in the file, not of what it does, and any
line appended after it silently switches it off. Worth knowing for #144, which
appends exactly such a line.

The containerfile skill also described a design this repository does not have.
It said utah-packages.repo is committed with enabled=0 and that the two mounted
steps enable it explicitly. It is committed with enabled=1 -- the repo file's
own comment says why -- and the flavor step flips it to 0 afterwards so later
dnf calls on the finished image do not fail on a baseurl that is no longer
mounted. The doc now says that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
main has moved a long way under this branch: the bind mount, the enabled=0
flip and its verification, and the repository-identity work all landed while
this was open. Resolved in main's favour everywhere the two describe the same
change, keeping only what is still this branch's own: the Red Hat release key
that lets hummingbird.repo run with gpgcheck=1, and the explicit-if guards.

The merge also tripped the gate that landed this morning:

    FAIL: test_every_copied_build_input_changes_the_key
      (source='packages/RPM-GPG-KEY-redhat-release-2')
    AssertionError: Containerfile.kernel copies
    packages/RPM-GPG-KEY-redhat-release-2 into the cache image, but editing it
    does not change the cache tag, so CI reuses a cache image built from the
    old packages/RPM-GPG-KEY-redhat-release-2.

Which is exactly right, and exactly this branch's bug: Containerfile.kernel
copies the key into the builder, so a rotated key is a different build root,
and scripts/kernel-cache-tag.sh did not hash it. Without that test the cache
tag would have stayed put across a key rotation and the three cached flavors
would have kept unpacking a kernel built against the old one. The key is now
in the hashed set.

tests/test_kernel_cache_tag.py needed its HASHED_INPUTS list extended too,
since it restates the inputs rather than deriving them; that is worth noting
as a wart, because forgetting it turns one clear failure into five confusing
ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Both findings were right and are fixed in 14bdb57, along with a merge of main.

1. The ! grep guard enforced nothing. Confirmed exactly as you described — errexit exempts a pipeline beginning with !:

$ bash -c 'set -euo pipefail; ! grep -q RUN Containerfile; echo reached'
reached
$ echo $?
0

So the PR body claimed a gate the recipe did not have. Both ! guards in check are now explicit if … exit 1 blocks with a message saying what to do.

Worth recording the part that goes beyond this PR: the other ! guard, the one forbidding flavor literals in workflows, has been working only by accident. It is the last command in the recipe, so its status becomes the recipe's status. That is a property of where it sits in the file, not of what it does — and #144 appends a new line after it, which would silently switch off the workflow-literal check while appearing to strengthen it. I have flagged that on #144.

2. The containerfile skill described a design this repository does not have. It said packages/utah-packages.repo is committed with enabled=0 and that the two mounted steps enable it. It is committed with enabled=1 — the repo file's own comment says why — and the flavor step flips it to 0 afterwards. The doc now says that.

The merge also caught a bug in this branch, via the gate that landed this morning (#166):

FAIL: test_every_copied_build_input_changes_the_key
  (source='packages/RPM-GPG-KEY-redhat-release-2')
AssertionError: Containerfile.kernel copies packages/RPM-GPG-KEY-redhat-release-2
into the cache image, but editing it does not change the cache tag, so CI reuses
a cache image built from the old packages/RPM-GPG-KEY-redhat-release-2.

Containerfile.kernel:28 copies the Red Hat release key into the builder, so a rotated key is a different build root, and scripts/kernel-cache-tag.sh was not hashing it. Left alone, the cache tag would have stayed put across a key rotation and the three cached flavors would have kept unpacking a kernel built against the old one — silently. The key is in the hashed set now. This is the first time that gate has caught something real, and it caught it in my own branch.

One wart it exposed: tests/test_kernel_cache_tag.py restates the hashed inputs in a HASHED_INPUTS tuple rather than deriving them, so adding an input means editing that list too — and forgetting turns one clear failure into five confusing ones about a sandbox that cannot reproduce the tag. Not worth changing here, but worth knowing.

Suite is green on the merged result: Ran 199 tests … OK (skipped=1).


Generated by Claude Code

@Danathar

Copy link
Copy Markdown
Contributor

#156 merged this morning and carries the 4 GB-repository removal from this PR (Containerfile, Justfile, docs/skills/containerfile.md overlap). What's left here that #156 doesn't cover is the Hummingbird GPG signature check (packages/RPM-GPG-KEY-redhat-release-2, packages/hummingbird.repo, iso/live/Containerfile, tests/test_package_resolution.py), and #157 touches the same three of those files for the same purpose. Suggest closing this in favour of #157, or rebasing this down to only the signature-check part if #157's version isn't what you want.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (correctness, at 14bdb57) — no blockers.

Verified: the committed packages/RPM-GPG-KEY-redhat-release-2 is byte-identical to Red Hat's published fd431d51 key, and the new tests' assertions (bind mount on both install steps, enabled=0 flip at Containerfile:209, dracut guard) all match the head Containerfiles.

One note: the PR body's first "What changed" bullet (COPY → bind mount, enabled=0 flip) is already on main at the merge base (a2cfbc2, Containerfile:120,189,205) — this diff adds only the enforcement (Justfile gate, tests, docs), not the conversion itself. Worth knowing when reading the changelog. Minor: the new comment in scripts/kernel-cache-tag.sh:29 references tests/test_kernel_cache_key.py; the file is tests/test_kernel_cache_tag.py.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar Danathar left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at head 14bdb57. Following up on my comment above: with main merged in, the 4 GB repository removal from #156 is no longer part of this diff, so what is left here is the Hummingbird signature check, the dracut gate for the live ISO, the Justfile guards that now actually fail, and the kernel-cache tag hashing the key. That is the rebased-down version I suggested, and this one commits the key file, which #157 does not, so I am fine with this landing.

What I checked. The committed packages/RPM-GPG-KEY-redhat-release-2 is byte-identical to Red Hat's published key: I fetched https://www.redhat.com/security/fd431d51.txt and cmp reports no difference (sha256 d4b25436...c742b on both), and gpg --show-keys gives fingerprint 567E 347A D004 4ADE 55BA 8A5F 199E 2F91 FD43 1D51 for the committed file, the download, and the copy in /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release on my Fedora host. The key is copied into both Containerfile and Containerfile.kernel, hummingbird.repo points gpgkey at that path, and the live ISO builds on top of the image so it inherits it. python3 tests/run_suite.py at this head: 199 tests, OK (1 skipped). I ran the two rewritten Justfile guards by hand under set -euo pipefail and they exit 1 when the pattern matches. All 14 checks are green on this head. One caveat: dispatch-iso skipped and no ISO workflow has run on this branch, so the dracut change in iso/live/Containerfile is verified by reading and by the unit test only, not by a real ISO build.

One non-blocking nit: the new comment in scripts/kernel-cache-tag.sh line 29 names tests/test_kernel_cache_key.py; the file is tests/test_kernel_cache_tag.py.

@hanthor
hanthor merged commit d4bcd38 into main Sep 20, 2026
14 checks passed
@hanthor
hanthor deleted the claude/image-size branch September 20, 2026 15:21
castrojo pushed a commit that referenced this pull request Sep 21, 2026
packages/nvidia-container.repo shipped enabled=1 and was copied into every
image via the Containerfile's yum.repos.d COPY, so a plain utah or
utah-gaming install exposed NVIDIA's third-party runtime repository even
though scripts/install-nvidia.sh -- the only thing that installs from it --
runs solely on the nvidia flavors. That contradicts the policy that Utah's
runtime repositories are Hummingbird plus the pinned utah-packages factory.

Ship the repo enabled=0 everywhere (the COPY stays; it is harmless disabled)
and turn it on for the single install transaction on the NVIDIA flavors,
--enablerepo=nvidia-container-toolkit, the same narrow window #133 gives
utah-packages.repo. Non-NVIDIA images now list only Hummingbird in
`dnf repolist`, while the NVIDIA flavors still install nvidia-container-toolkit
with repo_gpgcheck=1 and the pinned fingerprint intact.

Assisted-by: lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K
Co-authored-by: Claude https://claude.ai/ghost
hanthor added a commit that referenced this pull request Sep 22, 2026
linux-firmware joins [parity] so iwlwifi and friends load their blobs;
NetworkManager-wifi stays out pending utah-packages#136 (and the stale
#133 reference now points at the real issue). No enabled repository
packages a terminal, so ptyxis is recorded under [unavailable] tracked
by utah-packages#224, while the live session puts Ghostty on the dash
and sets its Flatpak export as the default terminal handler.

Assisted-by: Muse Code powered by Meta Muse Spark via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hanthor added a commit that referenced this pull request Sep 22, 2026
linux-firmware joins [parity] so iwlwifi and friends load their blobs;
NetworkManager-wifi stays out pending utah-packages#136 (and the stale
#133 reference now points at the real issue). No enabled repository
packages a terminal, so ptyxis is recorded under [unavailable] tracked
by utah-packages#224, while the live session puts Ghostty on the dash
and sets its Flatpak export as the default terminal handler.

Assisted-by: Muse Code powered by Meta Muse Spark via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

image: the 4 GB factory RPM repository is copied into /etc/utah-packages and never removed

3 participants