Repository navigation
ci(buildroot): factory-owned build-root mirror, and a cache key that survives a publish - #252
Conversation
|
To fix before this leaves draft:
|
quay.io garbage-collects fedora:44 digests within hours, so prepare pulls the moving tag (repeated-mistakes section 7). That kept runs alive but let the root change under the factory several times a day, and the package cache key includes the root digest, so late stages never hit: webkitgtk built cold in runs 36112238153 and 36117470974 because fedora:44 moved from 2cdfedd to 94e175d between them. Own the bytes instead: - refresh-buildroot.yml (weekly, and on dispatch) copies fedora:44 to ghcr.io/projectbluefin/utah-buildroot under a dated tag, proves the new digest pulls from a clean cache, and opens a PR moving BUILDROOT_IMAGE. - tools/buildroot_pin.py reads/moves the pin and refuses anything but a mirror tag@digest. - prepare pulls a mirror pin by its exact digest and fails if it is absent; the legacy quay.io value keeps the pull-tag-and-warn path until the first refresh PR lands. buildroot_digest stays in the cache key and now moves only on deliberate pin bumps. - check_build_container_pinning.py exempts exactly the refresh workflow's source tag, the one place a moving tag is the point. - repeated-mistakes section 7 records why the mirror exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
The key hashed the published factory image digest alongside the resolved root. Every publish changes that digest, so every publish invalidated every cache entry: run 36159982139, the first after 9e17ca2c published, rebuilt SDL3 and the rest cold on the same build root (94e175d9) that had built them that morning. The digest is redundant. resolved_root is the post-builddep rpm -qa, which already names the exact NEVR of every package the root took from the factory, so a factory change that reaches a build changes its key anyway. Remove it from cache_key(), the CLI and build-stage.yml, bump SCHEMA to 2, and test that a factory-only change leaves the key alone while a changed factory package in the root still moves it. package-build-cache.md explains why the field is gone and not to add it back. Landed together with the build-root mirror so the key changes once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
1e8e15b to
fbeeb0f
Compare
|
Added a second commit, so the key changes once and costs one cold rebuild, not two: fix(cache): drop the factory image digest from the package cache key. The key used to hash the published factory digest. Every publish invalidated every entry. Run 36159982139 rebuilt SDL3 and the rest cold on the same build root (94e175d9) where they had built that morning, only because 9cb66729 had become 9e17ca2c.
I also rebased onto main and kept Renovate's current quay pin (94e175d9) as the legacy value. Tests: 509 pass, and Plan once run 36159982139 publishes: merge this, dispatch Refresh the build-root mirror, merge the pin PR it opens, then dispatch one run. That run will be cold, and every run after it should reuse the cache. |
main moved the build root to the factory mirror (projectbluefin#252, projectbluefin#255): prepare now pulls exactly what config/buildroot-image pins, no workflow carries BUILDROOT_IMAGE, and Renovate no longer tracks quay.io/fedora/fedora:44. The drift gate here compared the lock against workflow pins, so after the merge it failed with "no workflow pins quay.io/fedora/fedora", and, as hanthor noted on projectbluefin#252, it would have failed every weekly refresh PR, because refresh-buildroot.yml only runs buildroot_pin.py set. - config/buildroot-lock.json names the mirror pin (moved with the tool). - tools/validate.py compares every locked root with config/buildroot-image, tag and digest; a missing pin is drift. - tools/buildroot_pin.py set moves the lock beside the pin. It parses the lock before writing either file, and leaves a locked packages list alone so snapshot --strict reports the divergence. - renovate.json goes back to main: the lock is no longer a Renovate target. - The validate output quoted in docs/architecture.md is the real line.
…ance (#186) * feat(supply-chain): lock factory buildroots and require recipe provenance A rebuild is reviewable only if it can say what it consumed. Three inputs decide the output -- the recipe, the source archive, and the build root -- and each was described by something that could move underneath a run without the run noticing. **The build root.** config/buildroot-lock.json declares it by digest, and the preflight job now records what the root actually contained: every package NEVRA, the digest prepare resolved, and the digest the lock expected. rpm -qa runs inside the root and the snapshot is assembled on the runner, so the root needs no interpreter -- installing one so it could inventory itself would change the set being inventoried. The rebuild does not pass --strict, deliberately. quay.io/fedora/fedora:44 is republished several times a day and each previous digest is collected, so failing on a moved tag recreates the outage repeated-mistakes.md section 7 records: thirty-seven jobs dead mid-run on a pin that was correct when the run started. The mismatch is recorded in the snapshot and the manifest instead, so a moved tag is visible rather than asserted away. --strict is there for an operator asking whether a root is exactly what was locked. The digest is now written twice -- in the lock and in BUILDROOT_IMAGE -- and two copies of one digest drift. Renovate's build-root-image manager covers the lock file as well, so both move in one pull request, and tools/validate.py fails the gate when any workflow pins a locked buildroot to a digest the lock does not name. That check runs before the recipe tally, because a package merely missing a Packit entry used to return 1 first and hide the drift. **The recipe.** Provenance was already written at import and validated loosely. It is now mandatory and checked by form: a rawhide import must pin full 40-character commit and tree SHAs (an abbreviated SHA is ambiguous and is not what git resolved), and a direct-upstream import must name a non-empty remote and carry no dist-git snapshot. validate.py reports the split it validated -- 346 rawhide, 3 upstream -- so the direct-upstream form, the one easiest to leave unchecked, is visible in the gate output. **The source.** source_pipeline.py records how each accepted archive was verified: signature, sha256-manifest, or the locked sha512 alone. A checksum is not a signature, so factory_manifest.py counts each kind and names the packages in the checksum-only classes -- an exception list to shorten, not a property to grep for. **The output.** factory_manifest.py writes manifest.json beside the repository: built RPMs, source reports and their verification summary, buildroot snapshots, the lock, and the published OCI reference and digest. It is written once before the container build and again once the digest exists, and uploaded as the factory-manifest artifact. Reports are classified by shape -- schema+name+packages for a snapshot, package for a source report -- so unrelated JSON beside the repository is not folded in. docs/skills/supply-chain-provenance.md carries the conventions, including why --strict is not the rebuild default and why the duplicated digest is not resolved by deleting either copy. Carries the work from #158, rebased onto the #178 baseline the maintainer asked it to be re-cut against, with that review's findings closed. Closes #25 Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com> * fix(tests): keep the buildroot snapshot tests out of the prepare job's env The prepare job exports BUILDROOT_IMAGE for the whole job and runs the unit tests inside it. cmd_snapshot reads that variable as the running image when no --image or --digest is passed, so test_cmd_snapshot saw the factory's real digest instead of the lock's fake one and failed its own strict check. Strip the factory's variables in setUp so the tests only see what they pass. Also double the % signs in the --packages-from help text: argparse %-formats help strings, and Python 3.14 checks that eagerly at add_argument time, which made every buildroot_lock test error locally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK * fix(supply-chain): keep workflow expressions out of run scripts and stop the snapshot defaulting to the lock Two review findings on the buildroot lock. The two factory manifest steps in rebuild-rpms.yml inlined ${{ needs.prepare.outputs.build_list }} inside single quotes in run:, and the second one inlined steps.oci.outputs.ref and .digest as well. An expression in run: is substituted before bash parses the script, so a quote in a package directory name would end the argument early, in the publish job, which holds packages: write and id-token: write. The values now go through env: and the script reads "$BUILD_LIST", "$OCI_REF" and "$OCI_DIGEST", the way preflight already passes build_list. Every expression this change adds to the workflow is now in env: or with:. cmd_snapshot's fallback chain ended in `or locked_image`, so an empty BUILDROOT_DIGEST made the snapshot record the lock's own digest as the image it ran in, which is exactly the provenance the snapshot exists to check rather than assert. With nothing resolved it now records image: null and warns; under --strict that is an error, because an operator asking whether the root is exactly what was locked cannot be told yes. locked_image is unchanged. test_cmd_snapshot passes a digest now that --strict refuses an unknown image, and two tests cover the null case and the strict refusal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK Signed-off-by: Danathar <doug.baggett@gmail.com> * fix(lock): follow the fedora:44 buildroot digest Renovate moved on main The merge from main brought the Renovate bump of BUILDROOT_IMAGE in rebuild-rpms.yml to sha256:a659125..., and this branch's config/buildroot-lock.json still named the previous digest, so the PR's own drift gate (tools/validate.py) failed. Point the lock at the digest the workflow pins. Once this lands, renovate.json's build-root-image manager moves both copies in one PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com> * fix(supply-chain): drop snapshot env fallbacks and make the drift gate total Review findings on #186. cmd_snapshot resolved actual_image through undocumented fallbacks to ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE, contradicting the module docstring and docs/skills/supply-chain-provenance.md, which both promise image:null when no --image/--digest/BUILDROOT_DIGEST is given. Since validate.py forces the workflow's BUILDROOT_IMAGE to equal the lock digest, running snapshot inside the prepare job silently re-asserted the lock's own pin as the resolved image -- the failure mode this branch exists to remove. Both fallbacks are gone; --image, --digest and BUILDROOT_DIGEST are the only inputs. tests/test_supply_chain_lock.py setUp no longer strips the two variables (it acknowledged the hazard rather than removing it), and a new test exports both to prove they are ignored. check_buildroot_drift keyed workflow pins by full repo:tag, so a pin moved to a different tag (workflow fedora:45@X vs lock fedora:44@Y) shared no key with the lock and was skipped, and a deleted pin left nothing to compare: validate passed in both cases. The comparison is now per repository, and a locked buildroot that no workflow pins is drift rather than an absence of evidence. Nothing is required when .github/workflows does not exist. just check and just test both pass: 441 tests, 2127 subtests; validate reports 342 source RPMs (339 rawhide, 3 upstream). Refs #186 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com> * fix(supply-chain): make the snapshot honest about what it could not read Three reviewer findings, all about the tool misreporting its own confidence. compare() crashed instead of reporting when a lock entry had no `nevra`: `item.get("nevra")` put None into the expected set, it landed in `missing`, and `", ".join(missing[:20])` died with a TypeError. load_lock only validates that `packages` is a list -- the per-entry requirement lives in tools/validate.py -- so `buildroot_lock.py snapshot --strict` run standalone against a hand-edited lock hit this. The gate failed either way, but a traceback does not tell an operator which entry to fix. Malformed entries are now counted and named as a lock defect, and deliberately not reported as missing packages: they are not a divergence in the build root. parse_packages() dropped malformed `rpm -qa` lines without a word. For an attestation input that is the wrong default -- a truncated or corrupted capture makes the snapshot under-report what the root contained, and nothing downstream could tell. It now warns on stderr naming the dropped lines, and --strict refuses the snapshot outright: a capture that under-reports cannot answer "is this root exactly what was locked?", because the lines it could not read might be the divergence. The scheduled rebuild is unaffected; it does not pass --strict, and records what it saw. Also drops the unused `prefix` parameter from factory_manifest.reports(). 443 tests pass; `tools/validate.py` clean. Assisted-by: Claude Opus 5 via Claude Code Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com> * fix(supply-chain): stop the manifest attesting a previous run's buildroot `buildroots` is the one part of the manifest that attests rather than describes: it says which root this run built in. Publish seeds `repository/` from the previously published factory image, that image was built `COPY repository /repository` with the last run's `reports/` inside it, and preflight -- which writes the replacement -- is skipped outright when `build_list` is `[]` and downloads `continue-on-error`. A cleanup-only run, or any run whose preflight failed, therefore reached the manifest step with the previous run's `buildroot-fedora-44.json` on disk and folded it in with nothing marking it as someone else's: the manifest attesting a root this run never measured, which is the failure class the lock exists to close. Two independent stops, because either one alone is a step-ordering assumption: - the seed step deletes `repository/reports/buildroot-*.json` right after the `podman cp`, so only this run's `preflight-buildroot` artifact can put one back. Source reports stay: they describe the seeded RPMs, which really are in the repository being published. - `factory_manifest.py --buildroot-digest` keeps a snapshot only when the image it resolved carries the digest `prepare` resolved. Both manifest writes pass it. Discarded, not silently dropped. A rejected snapshot is named under `buildroots_discarded` with its report path, image and a reason, and warned about on stderr -- "no root was measured" and "a root was measured and not attested" are different answers to the question asked after a bad package ships. With no digest to compare against (standalone use, or a run whose own resolution failed) nothing is judged; the prune covers that case instead. Tests cover both halves, including the workflow wiring: the prune exists and precedes the preflight download, and both manifest writes pass the digest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(supply-chain): make the buildroot lock follow the mirror pin main moved the build root to the factory mirror (#252, #255): prepare now pulls exactly what config/buildroot-image pins, no workflow carries BUILDROOT_IMAGE, and Renovate no longer tracks quay.io/fedora/fedora:44. The drift gate here compared the lock against workflow pins, so after the merge it failed with "no workflow pins quay.io/fedora/fedora", and, as hanthor noted on #252, it would have failed every weekly refresh PR, because refresh-buildroot.yml only runs buildroot_pin.py set. - config/buildroot-lock.json names the mirror pin (moved with the tool). - tools/validate.py compares every locked root with config/buildroot-image, tag and digest; a missing pin is drift. - tools/buildroot_pin.py set moves the lock beside the pin. It parses the lock before writing either file, and leaves a locked packages list alone so snapshot --strict reports the divergence. - renovate.json goes back to main: the lock is no longer a Renovate target. - The validate output quoted in docs/architecture.md is the real line. * fix(supply-chain): publish the manifest from publish-repository.yml, and snapshot only the pinned image Publishing moved out of rebuild-rpms.yml into publish-repository.yml on main, one call per early wave plus the final one, so the manifest steps move with it: - the seed step still drops the previous run's buildroot-*.json, and the preflight snapshot is downloaded after it (now artifact-prefixed, as every artifact of a canary pass must be); - built RPMs now land in built/ and only RPMs are copied into the repository, so the source reports of the packages this publication replaced are carried into repository/reports; a pruned source loses its report with its RPMs; - both manifest writes judge snapshots against a new buildroot_digest input, which every publish job passes; every publish job also needs preflight, so an early one cannot run before the snapshot exists; - the artifact is factory-manifest, or factory-manifest-wave<N> for an early publication, since names are unique per run. The last review on 522a9fc found the snapshot described no real root: it was taken after preflight installed @buildsys-build from live repos, which is neither the pulled image nor any wave's root. rpm -qa now runs first, before preflight changes anything, so the NEVRA list is exactly what the recorded digest names. The skill doc says what it is not: each hermetic package's root is its lock-s<N>-<package> artifact, and the container lane's is the post-builddep rpm -qa the cache key hashes. --------- Signed-off-by: Danathar <Danathar@users.noreply.github.com> Signed-off-by: Danathar <doug.baggett@gmail.com> Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Held as a draft until the first publish lands, as agreed. Merging it changes nothing about the build root on its own. The pin only moves when the refresh PR it produces is merged, and that one cold rebuild happens then.
Why
quay.io garbage-collects
fedora:44digests within hours. Repeated-mistakes §7 is three failed attempts to pull it by digest: the third pin died four hours in and killed 37 jobs. Sopreparepulls the moving tag. That keeps runs alive, but it lets the build root change under the factory several times a day. The package cache key includes the root digest, so late stages almost never get a cache hit. webkitgtk built cold in runs 36112238153 and 36117470974 becausefedora:44moved from2cdfeddto94e175dbetween them. The recorded pin had already been collected by then.What
refresh-buildroot.yml: runs weekly (Mon 04:23 UTC) and on dispatch. It copiesfedora:44toghcr.io/projectbluefin/utah-buildroot:44-<date>-<upstream digest prefix>. The image is labelled with its source repository and the upstream digest. The job re-pulls the new digest from a clean cache to prove it works, then opens a PR that movesBUILDROOT_IMAGE.tools/buildroot_pin.py: reads and moves the pin. It refuses anything that isn't a mirrortag@sha256pin.prepare:buildroot_digeststays in it, and now changes only when a refresh PR is merged.check_build_container_pinning.py: exempts exactly(refresh-buildroot.yml, quay.io/fedora/fedora:44), the one place where a moving tag is the point. Tests confirm the same line fails in any other workflow.Tests: 503 pass,
just checkis clean, and actionlint is clean on the new workflow.After merge
org.opencontainers.image.sourcelabel.packagesscope, so I can't check or change either.PRs opened with
GITHUB_TOKENdon't trigger other workflows. The refresh PR is a one-line pin change, so it needs a human merge, or a close/reopen to run CI.🤖 Generated with Claude Code
https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv