Skip to content

feat(supply-chain): lock factory buildroots and enforce recipe provenance - #158

Closed
mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:feat/supply-chain-lock-buildroots
Closed

mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:feat/supply-chain-lock-buildroots

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Summary

This PR implements reviewably reproducible buildroot locking and immutable recipe provenance controls to satisfy supply-chain requirements for factory rebuilds:

  • Buildroot Locking: Adds config/buildroot-lock.json pinning buildroot image references by digest (fedora-44, fedora-rawhide, and hummingbird-bootc-os).
  • Buildroot NEVRA Snapshots & Verification: Implements tools/buildroot_lock.py to resolve locked image digests, snapshot live package NEVRAs via rpm -qa, and support --strict verification against locked package sets.
  • Factory Manifest Emission: Implements tools/factory_manifest.py to assemble complete build manifests combining binary RPMs, source verification reports, buildroot package snapshots, and the published OCI image digest.
  • Recipe Provenance & Lock Validation: Extends tools/validate.py and tools/package_inventory.py to enforce that all recipes carry provenance metadata, validating full 40-character commit/tree SHAs for rawhide imports, non-empty upstream remotes for direct-upstream imports, and valid buildroot locks.
  • Source Signature Reporting: Updates tools/source_pipeline.py to record verification status (signature, sha256-manifest, or sha512) and flag checksum-only exceptions in report outputs.
  • Pipeline Integration: Captures buildroot snapshots during preflight and publishes manifest.json during publish in .github/workflows/rebuild-rpms.yml.
  • Documentation & Skills: Documents supply-chain provenance conventions in docs/skills/supply-chain-provenance.md, indexes it in docs/SKILL.md, and updates AGENTS.md.

Closes #25

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 799d238

…ance

Implement supply-chain controls for reviewably reproducible factory rebuilds
and immutable recipe provenance:

- Lock factory buildroots by image digest in config/buildroot-lock.json.
- Add tools/buildroot_lock.py to query image digests and capture live
  package NEVRAs, supporting strict validation against expected packages.
- Add tools/factory_manifest.py to assemble an end-to-end manifest of RPMs,
  source reports, buildroot snapshots, and OCI image digest.
- Require recipe provenance and valid buildroot locks in tools/validate.py:
  enforce full 40-character commit/tree SHAs for rawhide imports and
  valid remotes for upstream imports.
- Record recipe provenance and branch in tools/package_inventory.py.
- Record verification status (signature, sha256-manifest, sha512) and
  checksum-only flags in tools/source_pipeline.py.
- Update rebuild-rpms.yml to snapshot buildroot packages during preflight
  and emit the factory manifest during publish.
- Document supply-chain provenance conventions in docs/skills/supply-chain-provenance.md
  and update docs/SKILL.md and AGENTS.md.

Closes projectbluefin#25

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 799d238)

1. The manifest records the locked digest, not the buildroot actually used — provenance can be false. tools/buildroot_lock.py cmd_snapshot writes "image": spec["image"] copied straight from config/buildroot-lock.json (tools/buildroot_lock.py:97), but the snapshot runs inside utah-buildroot:run, whose provenance is BUILDROOT_IMAGE in the workflow (rebuild-rpms.yml:85) — which the pre-existing comment at rebuild-rpms.yml:70 explicitly calls "the EXPECTED value, not the pull target", and the run continues with a warning when the tag has moved (rebuild-rpms.yml:140). Nothing in this PR pulls from the lock or cross-checks it against the running container, so buildroot-fedora-44.json and manifest.json can attest a digest that is not the buildroot the packages were built in. The snapshot should record/verify the actual image ID rather than echo the lock.

2. Duplicate pin with no drift check. The fedora-44 digest now lives in both rebuild-rpms.yml:85 (renovate-managed per the comment at rebuild-rpms.yml:68) and config/buildroot-lock.json:5 (not renovate-managed). They will silently diverge, and validate_buildroots (tools/validate.py:19) doesn't compare them — compounding finding 1.

3. Two of three lock entries are decorative. Only fedora-44 is ever consumed (rebuild-rpms.yml:215); fedora-rawhide and hummingbird-bootc-os (config/buildroot-lock.json:8-14) are referenced by nothing, despite the PR body claiming buildroot locking for all three. The image subcommand (tools/buildroot_lock.py:83) also has no callers.

4. Fragile buildroot/source report classification. factory_manifest.py:63-70 classifies any report whose name starts with fedora- or whose path contains buildroot as a buildroot snapshot; reports() also sweeps every *.json under the repository root recursively (tools/factory_manifest.py:22-41), so unrelated JSON published beside the repo would be folded into sources. Matching on the snapshot schema (schema+name+packages) would be robust.

Minor: the new prior step (build-stage.yml:51-56) fixes a dangling steps.prior.outputs.pattern reference that already existed on main — good, but unmentioned in the body; note rpm-s[0-%s]-* only works while stages stay single-digit ([0-10] is not a range). --strict (tools/buildroot_lock.py:108) is currently unreachable in CI: every lock entry ships packages: [] and the workflow never passes the flag.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…drift

Address reviewer feedback on PR projectbluefin#158:
- Capture and attest actual running container digest in tools/buildroot_lock.py
  via --digest/--image and BUILDROOT_DIGEST, recording both actual and locked images.
- Enforce drift check in tools/validate.py between config/buildroot-lock.json
  and rebuild-rpms.yml BUILDROOT_IMAGE.
- Prune unused lock entries from config/buildroot-lock.json.
- Make buildroot and package report classification in tools/factory_manifest.py
  schema-driven and limit directory scanning to reports directory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback in f2cb020:

  1. Live buildroot provenance: tools/buildroot_lock.py:cmd_snapshot now supports --digest / --image (and reads BUILDROOT_DIGEST), recording the actual running container image/digest in image and the expected pin in locked_image. .github/workflows/rebuild-rpms.yml forwards the resolved buildroot digest to the snapshot step. In --strict mode, any mismatch between actual and locked images fails the snapshot.
  2. Lock drift check: tools/validate.py:validate_buildroots now validates that the digest pin in config/buildroot-lock.json matches BUILDROOT_IMAGE in .github/workflows/rebuild-rpms.yml and fails if they diverge.
  3. Pruned unused entries: Removed unused fedora-rawhide and hummingbird-bootc-os entries from config/buildroot-lock.json, leaving fedora-44 as the active locked buildroot. The image subcommand is retained as a CLI query tool for inspectability.
  4. Report classification: tools/factory_manifest.py now scopes its scan to the reports/ subdirectory and classifies entries by snapshot schema (schema == 1, name, packages) and package reports (package in item), preventing unrelated repository JSON from leaking into the manifest.

🐝 Hive Agent: contributor | SHA: unknown

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review:

  • Buildroot locking & provenance: Implements config/buildroot-lock.json and tools/buildroot_lock.py to lock container digests and snapshot live buildroot package NEVRAs.
  • In-progress run: Head commit f2cb020 fixed the test_cmd_snapshot mock issue present in commit 4b29bb8 (where BUILDROOT_IMAGE in CI environment caused an un-mocked mismatch). Fresh run is currently queued/running in CI.
  • Outstanding: Awaiting CI completion of the updated commit and independent peer/human approval (+2) before landing.

@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

1 similar comment
@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

@hanthor hanthor closed this Sep 20, 2026
castrojo pushed a commit that referenced this pull request Sep 27, 2026
…ance (#186)

* feat(supply-chain): lock factory buildroots and require recipe provenance

A rebuild is reviewable only if it can say what it consumed. Three inputs
decide the output -- the recipe, the source archive, and the build root -- and
each was described by something that could move underneath a run without the
run noticing.

**The build root.** config/buildroot-lock.json declares it by digest, and the
preflight job now records what the root actually contained: every package
NEVRA, the digest prepare resolved, and the digest the lock expected. rpm -qa
runs inside the root and the snapshot is assembled on the runner, so the root
needs no interpreter -- installing one so it could inventory itself would
change the set being inventoried.

The rebuild does not pass --strict, deliberately. quay.io/fedora/fedora:44 is
republished several times a day and each previous digest is collected, so
failing on a moved tag recreates the outage repeated-mistakes.md section 7
records: thirty-seven jobs dead mid-run on a pin that was correct when the run
started. The mismatch is recorded in the snapshot and the manifest instead, so
a moved tag is visible rather than asserted away. --strict is there for an
operator asking whether a root is exactly what was locked.

The digest is now written twice -- in the lock and in BUILDROOT_IMAGE -- and
two copies of one digest drift. Renovate's build-root-image manager covers the
lock file as well, so both move in one pull request, and tools/validate.py
fails the gate when any workflow pins a locked buildroot to a digest the lock
does not name. That check runs before the recipe tally, because a package
merely missing a Packit entry used to return 1 first and hide the drift.

**The recipe.** Provenance was already written at import and validated
loosely. It is now mandatory and checked by form: a rawhide import must pin
full 40-character commit and tree SHAs (an abbreviated SHA is ambiguous and is
not what git resolved), and a direct-upstream import must name a non-empty
remote and carry no dist-git snapshot. validate.py reports the split it
validated -- 346 rawhide, 3 upstream -- so the direct-upstream form, the one
easiest to leave unchecked, is visible in the gate output.

**The source.** source_pipeline.py records how each accepted archive was
verified: signature, sha256-manifest, or the locked sha512 alone. A checksum
is not a signature, so factory_manifest.py counts each kind and names the
packages in the checksum-only classes -- an exception list to shorten, not a
property to grep for.

**The output.** factory_manifest.py writes manifest.json beside the
repository: built RPMs, source reports and their verification summary,
buildroot snapshots, the lock, and the published OCI reference and digest. It
is written once before the container build and again once the digest exists,
and uploaded as the factory-manifest artifact. Reports are classified by shape
-- schema+name+packages for a snapshot, package for a source report -- so
unrelated JSON beside the repository is not folded in.

docs/skills/supply-chain-provenance.md carries the conventions, including why
--strict is not the rebuild default and why the duplicated digest is not
resolved by deleting either copy.

Carries the work from #158, rebased onto the #178 baseline the maintainer
asked it to be re-cut against, with that review's findings closed.

Closes #25

Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(tests): keep the buildroot snapshot tests out of the prepare job's env

The prepare job exports BUILDROOT_IMAGE for the whole job and runs the
unit tests inside it. cmd_snapshot reads that variable as the running
image when no --image or --digest is passed, so test_cmd_snapshot saw
the factory's real digest instead of the lock's fake one and failed its
own strict check. Strip the factory's variables in setUp so the tests
only see what they pass.

Also double the % signs in the --packages-from help text: argparse
%-formats help strings, and Python 3.14 checks that eagerly at
add_argument time, which made every buildroot_lock test error locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK

* fix(supply-chain): keep workflow expressions out of run scripts and stop the snapshot defaulting to the lock

Two review findings on the buildroot lock.

The two factory manifest steps in rebuild-rpms.yml inlined
${{ needs.prepare.outputs.build_list }} inside single quotes in run:,
and the second one inlined steps.oci.outputs.ref and .digest as well.
An expression in run: is substituted before bash parses the script, so
a quote in a package directory name would end the argument early, in
the publish job, which holds packages: write and id-token: write. The
values now go through env: and the script reads "$BUILD_LIST",
"$OCI_REF" and "$OCI_DIGEST", the way preflight already passes
build_list. Every expression this change adds to the workflow is now
in env: or with:.

cmd_snapshot's fallback chain ended in `or locked_image`, so an empty
BUILDROOT_DIGEST made the snapshot record the lock's own digest as the
image it ran in, which is exactly the provenance the snapshot exists
to check rather than assert. With nothing resolved it now records
image: null and warns; under --strict that is an error, because an
operator asking whether the root is exactly what was locked cannot be
told yes. locked_image is unchanged. test_cmd_snapshot passes a digest
now that --strict refuses an unknown image, and two tests cover the
null case and the strict refusal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK
Signed-off-by: Danathar <doug.baggett@gmail.com>

* fix(lock): follow the fedora:44 buildroot digest Renovate moved on main

The merge from main brought the Renovate bump of BUILDROOT_IMAGE in
rebuild-rpms.yml to sha256:a659125..., and this branch's
config/buildroot-lock.json still named the previous digest, so the
PR's own drift gate (tools/validate.py) failed. Point the lock at the
digest the workflow pins. Once this lands, renovate.json's
build-root-image manager moves both copies in one PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(supply-chain): drop snapshot env fallbacks and make the drift gate total

Review findings on #186.

cmd_snapshot resolved actual_image through undocumented fallbacks to
ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE, contradicting the module
docstring and docs/skills/supply-chain-provenance.md, which both promise
image:null when no --image/--digest/BUILDROOT_DIGEST is given. Since
validate.py forces the workflow's BUILDROOT_IMAGE to equal the lock digest,
running snapshot inside the prepare job silently re-asserted the lock's own
pin as the resolved image -- the failure mode this branch exists to remove.
Both fallbacks are gone; --image, --digest and BUILDROOT_DIGEST are the only
inputs. tests/test_supply_chain_lock.py setUp no longer strips the two
variables (it acknowledged the hazard rather than removing it), and a new
test exports both to prove they are ignored.

check_buildroot_drift keyed workflow pins by full repo:tag, so a pin moved
to a different tag (workflow fedora:45@X vs lock fedora:44@Y) shared no key
with the lock and was skipped, and a deleted pin left nothing to compare:
validate passed in both cases. The comparison is now per repository, and a
locked buildroot that no workflow pins is drift rather than an absence of
evidence. Nothing is required when .github/workflows does not exist.

just check and just test both pass: 441 tests, 2127 subtests; validate
reports 342 source RPMs (339 rawhide, 3 upstream).

Refs #186

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* fix(supply-chain): make the snapshot honest about what it could not read

Three reviewer findings, all about the tool misreporting its own confidence.

compare() crashed instead of reporting when a lock entry had no `nevra`:
`item.get("nevra")` put None into the expected set, it landed in `missing`,
and `", ".join(missing[:20])` died with a TypeError. load_lock only validates
that `packages` is a list -- the per-entry requirement lives in
tools/validate.py -- so `buildroot_lock.py snapshot --strict` run standalone
against a hand-edited lock hit this. The gate failed either way, but a
traceback does not tell an operator which entry to fix. Malformed entries are
now counted and named as a lock defect, and deliberately not reported as
missing packages: they are not a divergence in the build root.

parse_packages() dropped malformed `rpm -qa` lines without a word. For an
attestation input that is the wrong default -- a truncated or corrupted
capture makes the snapshot under-report what the root contained, and nothing
downstream could tell. It now warns on stderr naming the dropped lines, and
--strict refuses the snapshot outright: a capture that under-reports cannot
answer "is this root exactly what was locked?", because the lines it could not
read might be the divergence. The scheduled rebuild is unaffected; it does not
pass --strict, and records what it saw.

Also drops the unused `prefix` parameter from factory_manifest.reports().

443 tests pass; `tools/validate.py` clean.

Assisted-by: Claude Opus 5 via Claude Code
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(supply-chain): stop the manifest attesting a previous run's buildroot

`buildroots` is the one part of the manifest that attests rather than
describes: it says which root this run built in. Publish seeds
`repository/` from the previously published factory image, that image was
built `COPY repository /repository` with the last run's `reports/` inside
it, and preflight -- which writes the replacement -- is skipped outright
when `build_list` is `[]` and downloads `continue-on-error`. A cleanup-only
run, or any run whose preflight failed, therefore reached the manifest step
with the previous run's `buildroot-fedora-44.json` on disk and folded it in
with nothing marking it as someone else's: the manifest attesting a root
this run never measured, which is the failure class the lock exists to
close.

Two independent stops, because either one alone is a step-ordering
assumption:

- the seed step deletes `repository/reports/buildroot-*.json` right after
  the `podman cp`, so only this run's `preflight-buildroot` artifact can put
  one back. Source reports stay: they describe the seeded RPMs, which really
  are in the repository being published.
- `factory_manifest.py --buildroot-digest` keeps a snapshot only when the
  image it resolved carries the digest `prepare` resolved. Both manifest
  writes pass it.

Discarded, not silently dropped. A rejected snapshot is named under
`buildroots_discarded` with its report path, image and a reason, and warned
about on stderr -- "no root was measured" and "a root was measured and not
attested" are different answers to the question asked after a bad package
ships. With no digest to compare against (standalone use, or a run whose own
resolution failed) nothing is judged; the prune covers that case instead.

Tests cover both halves, including the workflow wiring: the prune exists and
precedes the preflight download, and both manifest writes pass the digest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(supply-chain): make the buildroot lock follow the mirror pin

main moved the build root to the factory mirror (#252, #255): prepare now
pulls exactly what config/buildroot-image pins, no workflow carries
BUILDROOT_IMAGE, and Renovate no longer tracks quay.io/fedora/fedora:44.
The drift gate here compared the lock against workflow pins, so after the
merge it failed with "no workflow pins quay.io/fedora/fedora", and, as
hanthor noted on #252, it would have failed every weekly refresh PR,
because refresh-buildroot.yml only runs buildroot_pin.py set.

- config/buildroot-lock.json names the mirror pin (moved with the tool).
- tools/validate.py compares every locked root with config/buildroot-image,
  tag and digest; a missing pin is drift.
- tools/buildroot_pin.py set moves the lock beside the pin. It parses the
  lock before writing either file, and leaves a locked packages list alone
  so snapshot --strict reports the divergence.
- renovate.json goes back to main: the lock is no longer a Renovate target.
- The validate output quoted in docs/architecture.md is the real line.

* fix(supply-chain): publish the manifest from publish-repository.yml, and snapshot only the pinned image

Publishing moved out of rebuild-rpms.yml into publish-repository.yml on
main, one call per early wave plus the final one, so the manifest steps
move with it:

- the seed step still drops the previous run's buildroot-*.json, and the
  preflight snapshot is downloaded after it (now artifact-prefixed, as
  every artifact of a canary pass must be);
- built RPMs now land in built/ and only RPMs are copied into the
  repository, so the source reports of the packages this publication
  replaced are carried into repository/reports; a pruned source loses its
  report with its RPMs;
- both manifest writes judge snapshots against a new buildroot_digest
  input, which every publish job passes; every publish job also needs
  preflight, so an early one cannot run before the snapshot exists;
- the artifact is factory-manifest, or factory-manifest-wave<N> for an
  early publication, since names are unique per run.

The last review on 522a9fc found the snapshot described no real root: it
was taken after preflight installed @buildsys-build from live repos, which
is neither the pulled image nor any wave's root. rpm -qa now runs first,
before preflight changes anything, so the NEVRA list is exactly what the
recorded digest names. The skill doc says what it is not: each hermetic
package's root is its lock-s<N>-<package> artifact, and the container
lane's is the post-builddep rpm -qa the cache key hashes.

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Signed-off-by: Danathar <doug.baggett@gmail.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

supply-chain: lock factory buildroots and require recipe provenance

3 participants