Skip to content

fix(kernel-cache): hash Containerfile.kernel whole so a recipe edit rotates the tag - #171

Merged
hanthor merged 1 commit into
mainfrom
scanner/fix-kernel-cache-tag-containerfile
Sep 19, 2026
Merged

hanthor merged 1 commit into
mainfrom
scanner/fix-kernel-cache-tag-containerfile

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Cluster: scripts/kernel-cache-tag.sh, docs/skills/kernel-cache.md § What the tag hashes, the kernel-cache bullet in AGENTS.md, and new tests/test_kernel_cache_tag.py. Touches no Containerfile.kernel content, no install script, and no workflow, so it is disjoint from #133, #154, #158 and #162.

Fix

scripts/kernel-cache-tag.sh declared its own contract — "It must change whenever anything the cache image contains would change" — but hashed only one line of Containerfile.kernel:

grep -m1 '^ARG BASE_IMAGE=' Containerfile.kernel

The body of that file is the cache recipe. The single RUN decides what fills /cache-out; the final stage decides what is copied out with COPY --from=builder. Edit either and the tag did not move:

$ ./scripts/kernel-cache-tag.sh
791f9b07ddd07e54
$ sed -i 's|UTAH_KERNEL_CACHE_OUT_DIR=/cache-out|UTAH_KERNEL_CACHE_OUT_DIR=/cache-out-CHANGED|' Containerfile.kernel
$ ./scripts/kernel-cache-tag.sh
791f9b07ddd07e54

CI builds and pushes the cache image only when its tag is not already published, so after such an edit the old tag is found, the rebuild is skipped, and gaming, nvidia and nvidia-gaming take the previous recipe's image as their BASE_IMAGE. The install scripts find /utah-cache and unpack instead of compiling, so stale artifacts ship with nothing red to show for it.

This is the exact opposite of the bias the script documents — it hashes whole scripts, comments included, so it "can only ever rebuild something that did not need rebuilding, never reuse something stale".

What changed

  • scripts/kernel-cache-tag.sh: grep -m1 '^ARG BASE_IMAGE=' → cat Containerfile.kernel. This strictly widens the key; the ARG BASE_IMAGE= line is still hashed, so Renovate base-image bumps still rotate the tag. Header comment updated to say why the whole file is hashed.
  • tests/test_kernel_cache_tag.py (new): copies the hashed inputs into a scratch tree and asserts each one moves the tag; asserts a recipe edit that provably leaves ARG BASE_IMAGE= untouched still moves it; asserts the base pin alone still moves it, so the widening did not drop what the key already covered. Two of these fail against the previous implementation.
  • docs/skills/kernel-cache.md and AGENTS.md: corrected — both described the narrow behaviour as if it were the contract.

Verification

$ python3 tests/run_suite.py
Ran 126 tests in 6.9s
OK (skipped=1)

(121 before, +5 new.) Against the pre-fix script the new suite is FAILED (failures=2), so it is a gate and not decoration. python3 scripts/generate_skill_index.py --check is clean.

Expected one-time cost

Widening the key rotates the tag once (791f9b07ddd07e54 → 5e80352e212ee6c5), so the first build of each cached flavor after this merges pays one ~45-minute cache rebuild. That is the correct price for a key that was previously able to reuse a stale image.

Closes #170


Filed by scanner agent (ACMM L5 — hold-gated mode). Hold-gated: human review required.

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

…otates the tag

scripts/kernel-cache-tag.sh hashed only the ARG BASE_IMAGE= line of
Containerfile.kernel, but the body of that file is the recipe: the single
RUN decides what lands in /cache-out and the final stage decides what is
copied out of the builder. Editing either left the tag unchanged, and CI
builds the cache image only when its tag is not already published, so the
gaming, nvidia and nvidia-gaming flavors would take a cache image built
from the previous recipe and unpack stale artifacts without any failure.

Hash the whole file instead. This strictly widens the key -- the base
image pin is still covered, so Renovate bumps still rotate the tag -- and
restores the bias the script documents, which is to over-rebuild rather
than ever reuse something stale.

tests/test_kernel_cache_tag.py holds the contract: every hashed input
moves the tag, and a recipe edit that leaves ARG BASE_IMAGE= alone moves
it too. Both new assertions fail against the previous implementation.

The key rotates once as a result, so the next build of each cached flavor
pays one ~45-minute cache rebuild.

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added the hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 19, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "scanner" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will automatically remove the hold label once current policy no longer requires a level hold for "scanner". If this is an outreach PR, a human must review it and remove the label.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving and merging. The repository owner has cleared me to act on hold items, so I am clearing the hold on this one rather than leaving it parked.

I reproduced the defect against main instead of taking the description's word for it — same probe, both implementations:

main   before=a5439a843919c4cc after=a5439a843919c4cc same=YES
fixed  before=5e80352e212ee6c5 after=508588d79be0da52 same=no

where the probe rewrites /cache-out throughout Containerfile.kernel and touches no ARG BASE_IMAGE= line. So on main the recipe that decides what fills the cache could be rewritten wholesale without moving the tag, CI would find the tag published, skip the rebuild, and hand gaming, nvidia and nvidia-gaming an image built by the previous recipe. Nothing would go red — the install scripts would find /utah-cache and unpack it. That is the bad direction of the bias the script's own header claims: reuse something stale rather than rebuild something that did not need it.

The widening is strictly a superset — the ARG line is still inside cat Containerfile.kernel — and test_base_image_pin_still_moves_the_tag guards precisely against the widening having dropped what the narrow key covered, which is the mistake worth guarding here.

Two things I checked beyond the diff:

  • kernel_cache on this run took 24 minutes rather than the usual ~2, because rotating the key actually forced the rebuild. The one-time cost the description predicts is visible in the run, not just asserted.
  • This and #166 both touch docs/skills/kernel-cache.md, so I merged them together locally before merging either: they auto-merge, and all 11 tests from both files pass on the combined tree. They are complementary rather than overlapping — this one covers the recipe body, #166 covers the content of files a future COPY might add.

One small correction to fold in whenever AGENTS.md is next touched: the kernel-cache bullet lists Containerfile.kernel, the two install scripts and packages/hummingbird.repo, but the script also hashes packages/fedora-44.repo. The prose in docs/skills/kernel-cache.md has it right; the bullet is one repository short.


Generated by Claude Code

@hanthor
hanthor merged commit e0b3b4a into main Sep 19, 2026
14 checks passed
@hanthor
hanthor deleted the scanner/fix-kernel-cache-tag-containerfile branch September 19, 2026 08:17
castrojo pushed a commit that referenced this pull request Sep 21, 2026
…ctually build (#182)

* Bump the NVIDIA driver to the release that compiles against kernel 7.2

The base image pin has been frozen since Hummingbird moved to kernel 7.2.
Every attempt to advance it -- #122 is the current one -- fails kernel_cache
22 minutes in, building the NVIDIA open module:

    nvidia/os-interface.c: In function 'os_get_current_process_name':
    nvidia/os-interface.c:764:5: error: implicit declaration of function
        'strncpy' [-Wimplicit-function-declaration]
      764 |     strncpy(buf, current->comm, len - 1);
    make[4]: *** [.../7.2.5-200.fc44.x86_64/...: nvidia/os-interface.o] Error 1

A frozen base pin means no kernel security updates reach the flavored images,
so this is worth fixing properly rather than pinning around.

The cause is the kernel finishing its strscpy migration and removing strncpy,
and the fix is upstream's, not ours. Read from the vendor source at each tag:

    595.84      strncpy(buf, current->comm, len - 1);
    595.91.07   strscpy
    595.99.02   if (strscpy(buf, current->comm, len) < 0) {

So 595.99.02 carries the migration, and it is also what
https://download.nvidia.com/XFree86/Linux-x86_64/latest.txt designates today
-- which is the rule the pin comment already states. This is that policy
working, not a hand-picked version: read latest.txt, take the SHA-256 NVIDIA
publishes beside the installer.

The base bump rides along deliberately. On the current base the old driver
compiles fine, so a driver-only change proves nothing -- the two can only be
verified together, and kernel_cache on this PR is that verification. Both
ARG BASE_IMAGE= lines move together, as `just check` requires.

The cache tag rotates 1ce05c4b16fafe9c -> f1e31a1dee27ffaa, so the three
cached flavors compile the kernel once. That is the intended cost: the tag
moving is what proves the new base and driver were actually built, rather
than a stale cache being reused (#166, #171).

Closes #173.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Make the shim mirroring say what it could not find, and pair the kernel-devel hash with its kernel

The base bump to a kernel-7.2 Hummingbird regressed build_main. It failed
deterministically, three attempts out of three, and printed nothing: the last
line of the log was "Utah desktop contract passed", then exit 1. The step was
four predicates chained with && at the tail of a long RUN --

  shim_version="$(rpm -q --qf '%{VERSION}-%{RELEASE}' shim-x64)" && \
  test -d /usr/lib/bootupd/updates/EFI/fedora && \
  install -d ... && cp -a ...

-- so nothing distinguished "shim-x64 is not in this base" from "shim-x64 is
installed but staged nothing", and those have different answers. The chain now
lives in scripts/mirror-shim.sh, which names the failing predicate, lists what
the bootupd tree actually contains when the payload is absent, and says that a
base which dropped shim-x64 needs it added to the package contract rather than
the step skipped. This does not yet fix the build; it makes the next run say
which of the two happened, which the previous three runs could not.

The second change is the review finding on the same bump. KERNEL_DEVEL_SHA256
guards an unsigned koji download and is documented as a constant precisely
because "the kernel cannot change without BASE_IMAGE changing" -- which is the
reason it must move when BASE_IMAGE does, and it did not. It stayed pinned to
7.1.8-100.fc43 while the base moved to 7.2.5-200.fc44. Nothing failed, because
the dnf path satisfied 7.2 and the fallback never ran; it would have failed the
first time the enabled repositories dropped the kernel, which is the single
situation the fallback exists for.

The hash is now the real one for kernel-devel-7.2.5-200.fc44.x86_64, taken from
koji, and it is recorded beside KERNEL_DEVEL_NEVR. A mismatch between the
recorded kernel and the booted one now fails immediately with both versions
named, instead of after fetching 60 MB to compare against a hash that cannot
match.

Tests drive the real code: mirror-shim.sh runs against a stub rpm and a fake
bootupd tree for all three outcomes, asserting the two failures do not share a
message; the NEVR guard is extracted and executed for both the matching and
mismatched kernel. 204 tests, OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Handle the base image that already mirrors the shim payload itself

I did not need to wait for CI for this. Listing both base images layer by layer
from the registry answers it outright, and the answer is that the new base has
already done the work:

  old base sha256:c5539f9e...
    usr/lib/bootupd/updates/EFI/fedora/{shimx64,shim,mmx64,grubx64}.efi
    no usr/lib/efi/shim at all

  new base sha256:db1007fd...
    usr/lib/efi/shim/16.1-5/EFI/fedora/{shimx64,shim,mmx64}.efi, BOOTX64.CSV
    usr/lib/bootupd/updates/ holds only BIOS.json and EFI.json

So the destination Utah was copying into is exactly what the new base now
ships, and the source it was copying from no longer exists. The step was not
broken by a missing package or a broken transaction; it was asserting a layout
the base has moved on from, and failing on a job that was already done.

mirror-shim.sh now looks for the payload before doing anything else and exits 0
when it is already in place. It searches for the EFI/fedora/shimx64.efi file
rather than assuming a directory name, because the base names that directory
for the shim version with no dist tag and rpm need not report the same string.
Checking the filesystem first also means the base that ships the payload is the
base whose shim-x64 rpm query we no longer depend on.

The old layout still works: the copy path is unchanged and tested. The two real
failures -- no package and no payload anywhere -- still fail, still say which,
and still do not share a message.

Tests drive the real script against synthetic filesystems for both layouts,
including the case where rpm reports 16.1-5.fc44 while the base directory is
16.1-5. 206 tests, OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Make the kernel NEVR overridable, so the hash override is not dead code

Review finding, and correct. I added a guard that exits when the booted kernel
differs from KERNEL_DEVEL_NEVR, to turn a 60 MB download ending in a checksum
mismatch into an immediate failure naming both versions. But
UTAH_KERNEL_DEVEL_SHA256 exists for exactly one situation -- validating a kernel
this file has not been re-recorded for -- and the guard runs before the hash
check, so it blocked precisely the case the override is for. Correct override or
not, that path exited 1. I turned an escape hatch into dead code while adding a
guard meant to make failures clearer.

KERNEL_DEVEL_NEVR now takes UTAH_KERNEL_DEVEL_NEVR the same way the hash takes
its override, and the guard message names both variables, so the way past it is
to set the pair deliberately rather than to edit the script. Overriding one
without the other still fails, which is the intended behaviour: the hash belongs
to a specific NEVR and the two only mean anything together.

Three tests: both constants accept an override, the guard message names them,
and the real guard extracted from the script reaches the download when both are
set to a kernel that is not the recorded one. The first fails against the
previous constant line.

209 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] kernel cache tag ignores Containerfile.kernel's build recipe — a stale cache image is reused silently

1 participant