Skip to content

test: gate the kernel cache key against Containerfile.kernel's build inputs - #166

Merged
hanthor merged 1 commit into
mainfrom
arch/kernel-cache-key-inputs
Sep 19, 2026
Merged

hanthor merged 1 commit into
mainfrom
arch/kernel-cache-key-inputs

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Refactor

Makes the kernel cache key a derived consequence of what
Containerfile.kernel is built from, instead of a second hand-maintained list.

Structural problem. scripts/kernel-cache-tag.sh hashes a list it names
itself: the ARG BASE_IMAGE= line of Containerfile.kernel,
scripts/install-ogc-kernel.sh, scripts/install-nvidia.sh,
packages/hummingbird.repo and packages/fedora-44.repo.
Containerfile.kernel decides independently what the cache image actually
contains — its ARG BASE_IMAGE= pin plus every non---from COPY source. The
two lists agree today; nothing made them agree tomorrow.

Add a COPY to Containerfile.kernel without a matching hash line and the key
stops moving when that input moves. build.yml builds the cache image only when
its tag is not already published, so an unchanged key means no rebuild, and
gaming, nvidia and nvidia-gaming keep unpacking a kernel and an NVIDIA
module built from the superseded input. Nothing fails; the wrong bits ship.

What this adds. tests/test_kernel_cache_key.py derives the input set from
Containerfile.kernel rather than restating it, then asserts the contract
behaviourally against a writable copy of the tree:

  • every derived input, when mutated, changes scripts/kernel-cache-tag.sh's output;
  • the ARG BASE_IMAGE= pin, when repinned, changes it;
  • an unrelated file (scripts/flavors.py) does not change it — the "and not
    otherwise" half of the invariant the script's header states, so the key can't be
    "fixed" by hashing the tree and rebuilding the ~45-minute cache on every edit;
  • the key is deterministic and matches ^[0-9a-f]{16}$;
  • the sandbox reproduces the real repo's key, so the mutation tests cannot pass
    vacuously against an incomplete tree; and the COPY derivation is asserted
    non-empty, with each derived path required to exist.

No production code, no workflow, no behavior change: one new test file plus the
skill-file update AGENTS.md requires.

Verification

$ python3 -m unittest tests.test_kernel_cache_key
Ran 6 tests in 0.283s
OK

$ python3 tests/run_suite.py
Ran 122 tests in 4.459s
OK (skipped=1)

$ bash scripts/check-skill-frontmatter.sh && python3 scripts/generate_skill_index.py --check
docs/skills/index.json and index.md are up to date (8 skills)

The gate is non-vacuous — appending COPY packages/utah.toml /tmp/probe.toml
to Containerfile.kernel (not committed) fails it with the message it exists
to produce:

AssertionError: Containerfile.kernel copies packages/utah.toml into the cache
image, but editing it does not change the cache tag, so CI reuses a cache image
built from the old packages/utah.toml. Add it to the hashed set in
scripts/kernel-cache-tag.sh.

Scope and overlap

Claims exactly tests/test_kernel_cache_key.py (new) and the "What the tag
hashes" section of docs/skills/kernel-cache.md. Checked against every open
PR on this repo: #122 and #133 edit Containerfile.kernel; #154, #158 and #160
edit scripts/install-ogc-kernel.sh / scripts/install-nvidia.sh and
docs/skills/kernel-cache.md. Those change the content of hashed inputs —
which is the case this gate protects — and none of them adds a gate over the
cache-key input set. No overlap with AGENTS.md (#142).

Closes #165


Filed by architect agent (ACMM L5 — hold-gated mode). Hold-gated: human review required.

— hive: agent=architect backend=copilot model=claude-opus-5 copilot=1.0.78

…inputs

scripts/kernel-cache-tag.sh hashes a hand-written list of inputs.
Containerfile.kernel decides, separately, what the cache image is built
from. Nothing tied the two together, so a COPY added there without a
matching hash line yields a key that does not move when that input does
-- and CI builds the cache image only when its tag is not already
published, so gaming, nvidia and nvidia-gaming would keep unpacking a
kernel and an NVIDIA module built from the superseded input, silently.

tests/test_kernel_cache_key.py derives the input set from
Containerfile.kernel (the ARG BASE_IMAGE= line and every non---from COPY
source) instead of restating it, then mutates a copy of the tree and
re-runs the script: every derived input must change the key, an unrelated
file must not, and the key must be deterministic. A sandbox-versus-repo
equality check keeps the mutation tests from passing against an
incomplete tree.

Closes #165

Assisted-by: Claude Opus 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added the hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 19, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "architect" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will automatically remove the hold label once current policy no longer requires a level hold for "architect". If this is an outreach PR, a human must review it and remove the label.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on the substance. Not merging — the hold label says human review is required, and clearing it is not mine to do.

I verified the claim rather than taking it from the description. Checked out the head and ran the suite, then made it fail on purpose:

$ python3 -m unittest tests.test_kernel_cache_key
Ran 6 tests in 0.139s
OK

$ printf 'COPY packages/utah.toml /tmp/probe.toml\n' >> Containerfile.kernel
$ python3 -m unittest tests.test_kernel_cache_key
AssertionError: '791f9b07ddd07e54' == '791f9b07ddd07e54' : Containerfile.kernel copies
packages/utah.toml into the cache image, but editing it does not change the cache tag,
so CI reuses a cache image built from the old packages/utah.toml. Add it to the hashed
set in scripts/kernel-cache-tag.sh.

So the gate is real and the failure message names the fix. Two things make it worth having rather than decorative:

  • It is behavioural. scripts/kernel-cache-tag.sh does cd "$(dirname "$0")/..", so running the copy in the sandbox hashes the sandbox — the mutation tests genuinely re-derive the key rather than asserting on the script's text. test_sandbox_reproduces_the_real_key closes the hole where an incomplete sandbox would make the mutations pass vacuously.
  • It asserts the "and not otherwise" half too. A key that moved on every edit would pass a naive version of this test while rebuilding the 45-minute cache image for nothing, which is the cost the cache exists to avoid.

The failure mode it closes is the quiet one, which is why it is worth a test: an unhashed input means the tag does not move, build.yml skips the rebuild because the tag is already published, and gaming, nvidia and nvidia-gaming keep unpacking a kernel built from the superseded input. Nothing goes red. This is pattern 4 in docs/skills/repeated-mistakes.md — a config change that alters a build has to invalidate the published copy — turned into a gate instead of a paragraph.

One limit worth stating so nobody reads more into the green than is there: the derivation covers ARG BASE_IMAGE= and non---from COPY sources. An input that enters the cache image some other way — a RUN curl, or a second ARG pin — is still outside the gate. That is a strict improvement over nothing, not a complete fence.

All fourteen checks are green and the head is based on current main, so it is ready for whoever clears the hold.


Generated by Claude Code

@hanthor
hanthor merged commit bc7412a into main Sep 19, 2026
14 checks passed
@hanthor
hanthor deleted the arch/kernel-cache-key-inputs branch September 19, 2026 08:17
castrojo pushed a commit that referenced this pull request Sep 21, 2026
…ctually build (#182)

* Bump the NVIDIA driver to the release that compiles against kernel 7.2

The base image pin has been frozen since Hummingbird moved to kernel 7.2.
Every attempt to advance it -- #122 is the current one -- fails kernel_cache
22 minutes in, building the NVIDIA open module:

    nvidia/os-interface.c: In function 'os_get_current_process_name':
    nvidia/os-interface.c:764:5: error: implicit declaration of function
        'strncpy' [-Wimplicit-function-declaration]
      764 |     strncpy(buf, current->comm, len - 1);
    make[4]: *** [.../7.2.5-200.fc44.x86_64/...: nvidia/os-interface.o] Error 1

A frozen base pin means no kernel security updates reach the flavored images,
so this is worth fixing properly rather than pinning around.

The cause is the kernel finishing its strscpy migration and removing strncpy,
and the fix is upstream's, not ours. Read from the vendor source at each tag:

    595.84      strncpy(buf, current->comm, len - 1);
    595.91.07   strscpy
    595.99.02   if (strscpy(buf, current->comm, len) < 0) {

So 595.99.02 carries the migration, and it is also what
https://download.nvidia.com/XFree86/Linux-x86_64/latest.txt designates today
-- which is the rule the pin comment already states. This is that policy
working, not a hand-picked version: read latest.txt, take the SHA-256 NVIDIA
publishes beside the installer.

The base bump rides along deliberately. On the current base the old driver
compiles fine, so a driver-only change proves nothing -- the two can only be
verified together, and kernel_cache on this PR is that verification. Both
ARG BASE_IMAGE= lines move together, as `just check` requires.

The cache tag rotates 1ce05c4b16fafe9c -> f1e31a1dee27ffaa, so the three
cached flavors compile the kernel once. That is the intended cost: the tag
moving is what proves the new base and driver were actually built, rather
than a stale cache being reused (#166, #171).

Closes #173.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Make the shim mirroring say what it could not find, and pair the kernel-devel hash with its kernel

The base bump to a kernel-7.2 Hummingbird regressed build_main. It failed
deterministically, three attempts out of three, and printed nothing: the last
line of the log was "Utah desktop contract passed", then exit 1. The step was
four predicates chained with && at the tail of a long RUN --

  shim_version="$(rpm -q --qf '%{VERSION}-%{RELEASE}' shim-x64)" && \
  test -d /usr/lib/bootupd/updates/EFI/fedora && \
  install -d ... && cp -a ...

-- so nothing distinguished "shim-x64 is not in this base" from "shim-x64 is
installed but staged nothing", and those have different answers. The chain now
lives in scripts/mirror-shim.sh, which names the failing predicate, lists what
the bootupd tree actually contains when the payload is absent, and says that a
base which dropped shim-x64 needs it added to the package contract rather than
the step skipped. This does not yet fix the build; it makes the next run say
which of the two happened, which the previous three runs could not.

The second change is the review finding on the same bump. KERNEL_DEVEL_SHA256
guards an unsigned koji download and is documented as a constant precisely
because "the kernel cannot change without BASE_IMAGE changing" -- which is the
reason it must move when BASE_IMAGE does, and it did not. It stayed pinned to
7.1.8-100.fc43 while the base moved to 7.2.5-200.fc44. Nothing failed, because
the dnf path satisfied 7.2 and the fallback never ran; it would have failed the
first time the enabled repositories dropped the kernel, which is the single
situation the fallback exists for.

The hash is now the real one for kernel-devel-7.2.5-200.fc44.x86_64, taken from
koji, and it is recorded beside KERNEL_DEVEL_NEVR. A mismatch between the
recorded kernel and the booted one now fails immediately with both versions
named, instead of after fetching 60 MB to compare against a hash that cannot
match.

Tests drive the real code: mirror-shim.sh runs against a stub rpm and a fake
bootupd tree for all three outcomes, asserting the two failures do not share a
message; the NEVR guard is extracted and executed for both the matching and
mismatched kernel. 204 tests, OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Handle the base image that already mirrors the shim payload itself

I did not need to wait for CI for this. Listing both base images layer by layer
from the registry answers it outright, and the answer is that the new base has
already done the work:

  old base sha256:c5539f9e...
    usr/lib/bootupd/updates/EFI/fedora/{shimx64,shim,mmx64,grubx64}.efi
    no usr/lib/efi/shim at all

  new base sha256:db1007fd...
    usr/lib/efi/shim/16.1-5/EFI/fedora/{shimx64,shim,mmx64}.efi, BOOTX64.CSV
    usr/lib/bootupd/updates/ holds only BIOS.json and EFI.json

So the destination Utah was copying into is exactly what the new base now
ships, and the source it was copying from no longer exists. The step was not
broken by a missing package or a broken transaction; it was asserting a layout
the base has moved on from, and failing on a job that was already done.

mirror-shim.sh now looks for the payload before doing anything else and exits 0
when it is already in place. It searches for the EFI/fedora/shimx64.efi file
rather than assuming a directory name, because the base names that directory
for the shim version with no dist tag and rpm need not report the same string.
Checking the filesystem first also means the base that ships the payload is the
base whose shim-x64 rpm query we no longer depend on.

The old layout still works: the copy path is unchanged and tested. The two real
failures -- no package and no payload anywhere -- still fail, still say which,
and still do not share a message.

Tests drive the real script against synthetic filesystems for both layouts,
including the case where rpm reports 16.1-5.fc44 while the base directory is
16.1-5. 206 tests, OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Make the kernel NEVR overridable, so the hash override is not dead code

Review finding, and correct. I added a guard that exits when the booted kernel
differs from KERNEL_DEVEL_NEVR, to turn a 60 MB download ending in a checksum
mismatch into an immediate failure naming both versions. But
UTAH_KERNEL_DEVEL_SHA256 exists for exactly one situation -- validating a kernel
this file has not been re-recorded for -- and the guard runs before the hash
check, so it blocked precisely the case the override is for. Correct override or
not, that path exited 1. I turned an escape hatch into dead code while adding a
guard meant to make failures clearer.

KERNEL_DEVEL_NEVR now takes UTAH_KERNEL_DEVEL_NEVR the same way the hash takes
its override, and the guard message names both variables, so the way past it is
to set the pair deliberately rather than to edit the script. Overriding one
without the other still fails, which is the intended behaviour: the hash belongs
to a specific NEVR and the two only mean anything together.

Three tests: both constants accept an override, the guard message names them,
and the real guard extracted from the script reaches the download when both are
set to a kernel that is not the recorded one. The first fails against the
previous constant line.

209 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[architect] kernel cache key restates Containerfile.kernel's build inputs — a COPY added without a hash line silently reuses a stale cache

1 participant