Skip to content

feat(supply-chain): lock factory buildroots and require recipe provenance - #186

Merged
castrojo merged 11 commits into
projectbluefin:mainfrom
Danathar:supply-chain/lock-buildroots-provenance
Sep 27, 2026
Merged

castrojo merged 11 commits into
projectbluefin:mainfrom
Danathar:supply-chain/lock-buildroots-provenance

Conversation

@Danathar

@Danathar Danathar commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes #25. This is the fresh, rebased re-cut of #158 that @hanthor asked for when closing it to consolidate the queue on the merged #178 full-factory candidate — branched from main at 73619b2, with that PR's review findings closed rather than carried over.

What the issue asked for, and what now answers it

Acceptance criterion Answer
Mandatory provenance, including direct-upstream validate.py requires .hummingbird-upstream.json per recipe and checks it by form — full 40-char commit/tree for rawhide, non-empty remote and no dist-git snapshot for upstream
Buildroot NEVRAs emitted and reusable as a lock input buildroot_lock.py snapshot writes every NEVRA in the shape the lock's packages field consumes, so today's snapshot can become tomorrow's lock
Mutable tags/repo state cannot silently change a rebuild The snapshot records the digest the run resolved beside the digest the lock expected; drift between the lock and the workflow pin fails just check
Signatures verified where upstream supplies them; checksum-only exceptions reported source_pipeline.py records verification/checksum_only; the manifest counts each kind and names the checksum-only packages
A rebuild emits a complete manifest alongside the OCI digest factory_manifest.py writes manifest.json before the container build and again once the digest exists, uploaded as factory-manifest

What changed from #158 after review

  • The snapshot no longer claims the lock's digest as provenance. It records the digest prepare actually resolved (image) next to the one the lock expects (locked_image), so the manifest cannot attest a root the packages were not built in.
  • --strict stays off in CI. With the mirror pin, prepare pulls the exact digest or fails. Only a legacy quay.io pin can drift, and the snapshot records that drift rather than failing on it.
  • The lock follows the mirror pin. The lock names the same mirror pin as config/buildroot-image. tools/buildroot_pin.py set moves both, and tools/validate.py fails if they differ. Renovate no longer tracks either.
  • The decorative entries are gone. Only fedora-44 is locked, because it is the only root the rebuild runs.
  • Reports are classified by shape, not filename (schema+name+packages vs package), scoped to reports/, so unrelated JSON beside the repository is not folded into sources.
  • The snapshot runs on the runner, not in the build root. feat(supply-chain): lock factory buildroots and enforce recipe provenance #158 called python3 /tools/buildroot_lock.py inside utah-buildroot:run. That root is a Fedora container with no interpreter this factory may assume — build-stage.yml installs python3 explicitly before its one in-container Python call — so under bash -exc a missing interpreter would have killed preflight. rpm -qa now prints the inventory and --packages-from assembles the JSON outside, which also keeps the tool from having to install anything into the thing it is measuring.
  • validate.py reports the provenance split (validated 342 source RPMs (339 rawhide, 3 upstream)), so the direct-upstream form — the one easiest to leave unvalidated — is visible in the gate output rather than only in a field nothing reads.
  • Dropped from feat(supply-chain): lock factory buildroots and enforce recipe provenance #158's diff: the build-stage.yml prior step. It fixes a real dangling steps.prior.outputs.pattern that still exists on main, but it is an unrelated bug and this repo asks for one logical change per PR. Happy to send it separately.

Validation

Re-run at head (93ff57b), after merging main in:

python3 -m pytest tests -q               # 691 passed
python3 tools/validate.py                # validated 400 source RPMs (397 rawhide, 3 upstream)
python3 tools/factory_contract.py        # factory contract satisfied

check_workflow_quoting.py, check_build_container_pinning.py, runtime_contract.py --check, publish_gate.py and actionlint 1.7.12 (with shellcheck) on rebuild-rpms.yml and publish-repository.yml are clean. The required Canary check refuses fork PRs that touch pipeline paths, so these workflow changes have not run for real yet.

The manifest steps now live in publish-repository.yml, since main moved publishing there. The buildroot snapshot is the pinned image, taken before preflight installs anything.

Per AGENTS.md, the learning ships with the change: docs/skills/supply-chain-provenance.md, indexed in docs/SKILL.md.

— hive: backend=claude model=claude-opus-5


Review-fix pass (auto-fix attempt 1), commit 4b72f64:

  • cmd_snapshot env fallbacks contradict the null-provenance contract --
    removed the ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE fallbacks from
    cmd_snapshot. --image, --digest and BUILDROOT_DIGEST are now the only
    inputs, matching the module docstring and
    docs/skills/supply-chain-provenance.md. The workflow already passes
    --digest explicitly, so CI behaviour is unchanged.
    tests/test_supply_chain_lock.py setUp no longer strips the two variables;
    a new test exports both and asserts the snapshot still records image: null.
  • drift check blind to tag drift and missing pins -- check_buildroot_drift
    now compares per repository rather than per repo:tag, and treats a locked
    buildroot that no workflow pins as drift. Two tests added: a pin retagged to
    fedora:45 while the lock says fedora:44, and a workflow with the pin
    deleted. Docs updated to state the wider guarantee.
  • stale validation record -- the Validation block above is re-run at head.

Danathar and others added 2 commits September 20, 2026 01:02
…ance

A rebuild is reviewable only if it can say what it consumed. Three inputs
decide the output -- the recipe, the source archive, and the build root -- and
each was described by something that could move underneath a run without the
run noticing.

**The build root.** config/buildroot-lock.json declares it by digest, and the
preflight job now records what the root actually contained: every package
NEVRA, the digest prepare resolved, and the digest the lock expected. rpm -qa
runs inside the root and the snapshot is assembled on the runner, so the root
needs no interpreter -- installing one so it could inventory itself would
change the set being inventoried.

The rebuild does not pass --strict, deliberately. quay.io/fedora/fedora:44 is
republished several times a day and each previous digest is collected, so
failing on a moved tag recreates the outage repeated-mistakes.md section 7
records: thirty-seven jobs dead mid-run on a pin that was correct when the run
started. The mismatch is recorded in the snapshot and the manifest instead, so
a moved tag is visible rather than asserted away. --strict is there for an
operator asking whether a root is exactly what was locked.

The digest is now written twice -- in the lock and in BUILDROOT_IMAGE -- and
two copies of one digest drift. Renovate's build-root-image manager covers the
lock file as well, so both move in one pull request, and tools/validate.py
fails the gate when any workflow pins a locked buildroot to a digest the lock
does not name. That check runs before the recipe tally, because a package
merely missing a Packit entry used to return 1 first and hide the drift.

**The recipe.** Provenance was already written at import and validated
loosely. It is now mandatory and checked by form: a rawhide import must pin
full 40-character commit and tree SHAs (an abbreviated SHA is ambiguous and is
not what git resolved), and a direct-upstream import must name a non-empty
remote and carry no dist-git snapshot. validate.py reports the split it
validated -- 346 rawhide, 3 upstream -- so the direct-upstream form, the one
easiest to leave unchecked, is visible in the gate output.

**The source.** source_pipeline.py records how each accepted archive was
verified: signature, sha256-manifest, or the locked sha512 alone. A checksum
is not a signature, so factory_manifest.py counts each kind and names the
packages in the checksum-only classes -- an exception list to shorten, not a
property to grep for.

**The output.** factory_manifest.py writes manifest.json beside the
repository: built RPMs, source reports and their verification summary,
buildroot snapshots, the lock, and the published OCI reference and digest. It
is written once before the container build and again once the digest exists,
and uploaded as the factory-manifest artifact. Reports are classified by shape
-- schema+name+packages for a snapshot, package for a source report -- so
unrelated JSON beside the repository is not folded in.

docs/skills/supply-chain-provenance.md carries the conventions, including why
--strict is not the rebuild default and why the duplicated digest is not
resolved by deleting either copy.

Carries the work from projectbluefin#158, rebased onto the projectbluefin#178 baseline the maintainer
asked it to be re-cut against, with that review's findings closed.

Closes projectbluefin#25

Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…s env

The prepare job exports BUILDROOT_IMAGE for the whole job and runs the
unit tests inside it. cmd_snapshot reads that variable as the running
image when no --image or --digest is passed, so test_cmd_snapshot saw
the factory's real digest instead of the lock's fake one and failed its
own strict check. Strip the factory's variables in setUp so the tests
only see what they pass.

Also double the % signs in the --packages-from help text: argparse
%-formats help strings, and Python 3.14 checks that eagerly at
add_argument time, which made every buildroot_lock test error locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: Implements buildroot locking via config/buildroot-lock.json, NEVRA snapshot tooling, factory manifest generation, and provenance verification across recipes, closing #25. Clean trial merge and passes all tests and factory validation. Awaiting second independent review to satisfy merge policy.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security

  • Workflow expression interpolated into shell (medium) — the two new manifest steps inline ${{ needs.prepare.outputs.build_list }} inside single quotes directly in run: (.github/workflows/rebuild-rpms.yml:838 and :924). A package directory name containing ' would break out of the quoting and execute in the publish job, which holds packages: write and id-token: write for attestation. The value is repo-derived (matrix from packages/ dir names), so exploitation needs a merged malicious dir name — but this file's own convention already handles exactly this: the preflight step passes the same output via env: BUILD_LIST (:267). Please route build_list (and ideally steps.oci.outputs.ref/digest at :925-926) through env: the same way.

correctness

  • Empty digest silently re-asserts the lock (low) — cmd_snapshot's fallback chain ends in or locked_image (tools/buildroot_lock.py:130-136). If BUILDROOT_DIGEST arrives empty (--digest "" is falsy, and the preflight step sets no other env), the snapshot records the locked image as image with no warning — attesting exactly the provenance this PR's body says the snapshot stopped claiming. Prefer recording null/"unknown" or warning when no resolved digest is available.

style

  • reports() in tools/factory_manifest.py:23 takes a prefix parameter no caller uses — dead code, drop it or use it.

Verified against the tree at 2c416c0: prepare exports buildroot_digest (rebuild-rpms.yml:101), the oci step emits ref (:733), work/reports exists before the in-container rpm -qa (:269, :272), rpm-* artifacts unpack source reports into repository/reports/ (build-stage.yml:691-696), and just check → validate.py covers the drift gate (Justfile check). Docs match the implemented behavior.

No findings from: intent-alignment, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…top the snapshot defaulting to the lock

Two review findings on the buildroot lock.

The two factory manifest steps in rebuild-rpms.yml inlined
${{ needs.prepare.outputs.build_list }} inside single quotes in run:,
and the second one inlined steps.oci.outputs.ref and .digest as well.
An expression in run: is substituted before bash parses the script, so
a quote in a package directory name would end the argument early, in
the publish job, which holds packages: write and id-token: write. The
values now go through env: and the script reads "$BUILD_LIST",
"$OCI_REF" and "$OCI_DIGEST", the way preflight already passes
build_list. Every expression this change adds to the workflow is now
in env: or with:.

cmd_snapshot's fallback chain ended in `or locked_image`, so an empty
BUILDROOT_DIGEST made the snapshot record the lock's own digest as the
image it ran in, which is exactly the provenance the snapshot exists
to check rather than assert. With nothing resolved it now records
image: null and warns; under --strict that is an error, because an
operator asking whether the root is exactly what was locked cannot be
told yes. locked_image is unchanged. test_cmd_snapshot passes a digest
now that --strict refuses an unknown image, and two tests cover the
null case and the strict refusal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK
Signed-off-by: Danathar <doug.baggett@gmail.com>
@Danathar

Copy link
Copy Markdown
Contributor Author

Pushed 41312bc for the two reviewer findings.

Workflow quoting: the two factory manifest steps in .github/workflows/rebuild-rpms.yml no longer inline ${{ needs.prepare.outputs.build_list }} in run:. Both steps now pass it through env: as BUILD_LIST and the script reads "$BUILD_LIST", the same way the preflight step already does. The second step also moves steps.oci.outputs.ref and .digest into env: as OCI_REF and OCI_DIGEST. After this, every expression the PR adds to the workflow sits in env: or with:; the one remaining ${{ }} inside a run: in that file is the pre-existing cosign line on main, which this PR does not touch.

Empty digest: cmd_snapshot in tools/buildroot_lock.py no longer falls back to the lock's digest. With no --image, --digest or BUILDROOT_DIGEST it records image: null and prints a warning, and under --strict that is an error. locked_image is unchanged. test_cmd_snapshot now passes a digest since strict refuses an unknown image, and two new tests in tests/test_supply_chain_lock.py cover the null case and the strict refusal. The skill doc says the same.

I did not touch the unused prefix parameter in factory_manifest.py; happy to drop it if you want that in this PR.

Checked locally at 41312bc: python3 -m pytest tests -q (421 passed, including test_atomic_publish and test_factory_witness with pyyaml this time), python3 tools/validate.py, python3 tools/factory_contract.py, python3 tools/check_workflow_quoting.py, python3 tools/check_build_container_pinning.py, and actionlint 1.7.12 with shellcheck on rebuild-rpms.yml, all clean.

Danathar and others added 2 commits September 20, 2026 19:36
The merge from main brought the Renovate bump of BUILDROOT_IMAGE in
rebuild-rpms.yml to sha256:a659125..., and this branch's
config/buildroot-lock.json still named the previous digest, so the
PR's own drift gate (tools/validate.py) failed. Point the lock at the
digest the workflow pins. Once this lands, renovate.json's
build-root-image manager moves both copies in one PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • cmd_snapshot env fallbacks contradict the provenance contract this PR states (tools/buildroot_lock.py:118-124 in the new file, the actual_image chain). The module docstring and docs/skills/supply-chain-provenance.md both claim "with no --image, --digest or BUILDROOT_DIGEST the snapshot records image: null" — but the code also falls back to ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE from the environment. Since validate.py forces the workflow's BUILDROOT_IMAGE to equal the lock digest, an operator running snapshot in a shell that exports that variable (the prepare job's env, or anything sourced from it) gets a snapshot that silently re-asserts the pin as the resolved image — the exact failure the PR body says was fixed from #158. The tests know: tests/test_supply_chain_lock.py setUp strips those three variables. Either drop the two undocumented fallbacks or document them; as merged, doc and code make different promises. (CI itself is fine — the snapshot step passes BUILDROOT_DIGEST explicitly, .github/workflows/rebuild-rpms.yml:311-318.)
  • Minor: check_buildroot_drift (tools/validate.py) keys workflow pins by full repo:tag, so it only compares digests while the tags agree. A workflow moved to fedora:45@X with the lock still at fedora:44@Y — or a deleted workflow pin — produces zero comparisons and passes. The guarantee "fails when the lock and the workflow name different digests" holds only for same-tag drift.

intent-alignment

  • The PR body's quoted validation output — validated 349 source RPMs (346 rawhide, 3 upstream) — cannot come from this branch: head has 342 recipes, and tests/test_package_inventory.py:17 (updated in this very diff) asserts len(records) == 342. I verified the substance independently (342/342 recipes carry .hummingbird-upstream.json at 6b7b420), so the gate is satisfied; the claimed run is stale, presumably from the #158-era tree. Worth correcting so the record is honest.

No findings from: security, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

…e total

Review findings on projectbluefin#186.

cmd_snapshot resolved actual_image through undocumented fallbacks to
ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE, contradicting the module
docstring and docs/skills/supply-chain-provenance.md, which both promise
image:null when no --image/--digest/BUILDROOT_DIGEST is given. Since
validate.py forces the workflow's BUILDROOT_IMAGE to equal the lock digest,
running snapshot inside the prepare job silently re-asserted the lock's own
pin as the resolved image -- the failure mode this branch exists to remove.
Both fallbacks are gone; --image, --digest and BUILDROOT_DIGEST are the only
inputs. tests/test_supply_chain_lock.py setUp no longer strips the two
variables (it acknowledged the hazard rather than removing it), and a new
test exports both to prove they are ignored.

check_buildroot_drift keyed workflow pins by full repo:tag, so a pin moved
to a different tag (workflow fedora:45@X vs lock fedora:44@Y) shared no key
with the lock and was skipped, and a deleted pin left nothing to compare:
validate passed in both cases. The comparison is now per repository, and a
locked buildroot that no workflow pins is drift rather than an absence of
evidence. Nothing is required when .github/workflows does not exist.

just check and just test both pass: 441 tests, 2127 subtests; validate
reports 342 source RPMs (339 rawhide, 3 upstream).

Refs projectbluefin#186

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added agent/ci-maintainer Filed or owned by the ci-maintainer agent. hive/hosted-projectbluefin-knuckle-gjvq labels Sep 21, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • compare() in tools/buildroot_lock.py (lines 63–73 at 4b72f64) crashes instead of reporting when a lock entry lacks nevra: item.get("nevra") puts None into wanted, it lands in missing, and ", ".join(missing[:20]) raises TypeError. load_lock validates packages is a list but not that entries carry nevra (only tools/validate.py enforces that), so a standalone buildroot_lock.py snapshot --strict against a hand-edited lock dies with a traceback rather than the intended error. Still exits non-zero, so the gate holds — low.
  • parse_packages (tools/buildroot_lock.py:44–45) silently drops malformed inventory lines (len(parts) != 3 → continue), and tests/test_supply_chain_lock.py asserts that drop. For an attestation tool, a corrupted/truncated rpm -qa capture under-reports the root's contents with no warning; a stderr note (or strict-mode failure) when lines are discarded would keep the snapshot honest about incompleteness. Low.

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@castrojo
castrojo requested a review from hanthor September 21, 2026 03:24
Three reviewer findings, all about the tool misreporting its own confidence.

compare() crashed instead of reporting when a lock entry had no `nevra`:
`item.get("nevra")` put None into the expected set, it landed in `missing`,
and `", ".join(missing[:20])` died with a TypeError. load_lock only validates
that `packages` is a list -- the per-entry requirement lives in
tools/validate.py -- so `buildroot_lock.py snapshot --strict` run standalone
against a hand-edited lock hit this. The gate failed either way, but a
traceback does not tell an operator which entry to fix. Malformed entries are
now counted and named as a lock defect, and deliberately not reported as
missing packages: they are not a divergence in the build root.

parse_packages() dropped malformed `rpm -qa` lines without a word. For an
attestation input that is the wrong default -- a truncated or corrupted
capture makes the snapshot under-report what the root contained, and nothing
downstream could tell. It now warns on stderr naming the dropped lines, and
--strict refuses the snapshot outright: a capture that under-reports cannot
answer "is this root exactly what was locked?", because the lines it could not
read might be the divergence. The scheduled rebuild is unaffected; it does not
pass --strict, and records what it saw.

Also drops the unused `prefix` parameter from factory_manifest.reports().

443 tests pass; `tools/validate.py` clean.

Assisted-by: Claude Opus 5 via Claude Code
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Copy link
Copy Markdown
Contributor Author

Both findings were real. Fixed in 5217b7b, along with the reports() dead parameter from your earlier pass.

compare() crash on a lock entry without nevra — reproduced exactly as you described: None enters wanted, lands in missing, and ", ".join(missing[:20]) raises TypeError: sequence item 0: expected str instance, NoneType found. Your severity call is right — the gate failed either way — but a traceback doesn't tell an operator which entry to fix. Malformed entries are now counted and named:

malformed lock: 2 locked package entries have no nevra field

One deliberate choice there: they're reported as a lock defect and not also as missing packages. An entry the lock couldn't express isn't a divergence in the build root, and conflating the two sends whoever reads the failure looking in the wrong place.

parse_packages silently dropping malformed lines — agreed this is the wrong default for an attestation input. Two levels now:

  • always: a stderr warning naming the dropped lines — dropped 1 unparsable line(s) from the buildroot inventory; this snapshot under-reports the root's contents: 'truncated\tline'
  • under --strict: refusal. --strict asks "is this root exactly what was locked?", and a capture that under-reports can't answer that in either direction — the lines it couldn't read might be the divergence.

The scheduled rebuild is untouched by the second part: it doesn't pass --strict (for the repeated-mistakes.md §7 reason), so it still records what it saw and keeps going.

Two new tests plus an updated assertion on the existing inventory test, which previously asserted the silent drop and now asserts the warning. 443 pass; tools/validate.py clean.

On the stale validation transcript from your earlier review — that was already corrected in the body at 4b72f64; I've refreshed it again for this head (5217b7b, 443 tests). Thank you for catching it, and for independently verifying the 342/342 .hummingbird-upstream.json coverage rather than taking the body's word for it — the body was the thing that was wrong.

🐝 Hive Agent: contributor | SHA: 5217b7b

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

The manifest can attest a buildroot snapshot from a previous run. The publish job seeds repository/ from the previously published factory image (podman cp "$container:/repository/." repository/, .github/workflows/rebuild-rpms.yml:709-711), and that image was built with COPY repository /repository after the snapshot was downloaded into repository/reports — so the prior run's reports/buildroot-fedora-44.json comes back with the seed. The fresh snapshot normally overwrites it, but the preflight-buildroot download is continue-on-error: true (rebuild-rpms.yml:718-722), the preflight job itself is continue-on-error: true (rebuild-rpms.yml:221) and is skipped entirely when build_list == '[]' (rebuild-rpms.yml:219) — while publish still runs on skipped waves (rebuild-rpms.yml:630-642). On every cleanup-only run, and on any run whose preflight failed, factory_manifest.py reports() (tools/factory_manifest.py:29-45) folds the stale snapshot into buildroots with nothing marking it as belonging to another run — the manifest silently attests a root this run never measured, which is the exact failure class the PR exists to close. Consider pruning repository/reports/buildroot-*.json after the seed step, or having factory_manifest.py drop/flag a snapshot whose image digest does not match the run's resolved digest.

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

…root

`buildroots` is the one part of the manifest that attests rather than
describes: it says which root this run built in. Publish seeds
`repository/` from the previously published factory image, that image was
built `COPY repository /repository` with the last run's `reports/` inside
it, and preflight -- which writes the replacement -- is skipped outright
when `build_list` is `[]` and downloads `continue-on-error`. A cleanup-only
run, or any run whose preflight failed, therefore reached the manifest step
with the previous run's `buildroot-fedora-44.json` on disk and folded it in
with nothing marking it as someone else's: the manifest attesting a root
this run never measured, which is the failure class the lock exists to
close.

Two independent stops, because either one alone is a step-ordering
assumption:

- the seed step deletes `repository/reports/buildroot-*.json` right after
  the `podman cp`, so only this run's `preflight-buildroot` artifact can put
  one back. Source reports stay: they describe the seeded RPMs, which really
  are in the repository being published.
- `factory_manifest.py --buildroot-digest` keeps a snapshot only when the
  image it resolved carries the digest `prepare` resolved. Both manifest
  writes pass it.

Discarded, not silently dropped. A rejected snapshot is named under
`buildroots_discarded` with its report path, image and a reason, and warned
about on stderr -- "no root was measured" and "a root was measured and not
attested" are different answers to the question asked after a bad package
ships. With no digest to compare against (standalone use, or a run whose own
resolution failed) nothing is judged; the prune covers that case instead.

Tests cover both halves, including the workflow wiring: the prune exists and
precedes the preflight download, and both manifest writes pass the digest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Danathar

Copy link
Copy Markdown
Contributor Author

Real, and the nastiest one on this PR so far — the hole was in buildroots, the one field that attests rather than describes. Fixed in 522a9fc.

I walked the path you described and it holds exactly as written: the seed podman cps the previous factory image's /repository/. over repository/, that image was built COPY repository /repository after the last run's factory_manifest.py had already left reports/buildroot-fedora-44.json in there, and the only thing that normally overwrites it is a preflight-buildroot artifact from a job that is continue-on-error, downloaded with continue-on-error, and skipped outright on build_list == '[]' — while publish still runs on prune_sources != '[]'. So every cleanup-only run was publishing a manifest that named a build root nothing in that run had measured.

Both of the fixes you offered, because either alone is a step-ordering assumption:

  • The prune. rm -f repository/reports/buildroot-*.json immediately after the podman cp, before the artifact download, so only this run's artifact can put one back. Source reports are deliberately left alone — those describe the seeded RPMs, and the seeded RPMs really are in the repository being published, so carrying them forward is the honest thing. A buildroot snapshot is the only report in there whose subject is the run rather than the repository.
  • The digest check. factory_manifest.py grew --buildroot-digest, passed by both manifest writes from needs.prepare.outputs.buildroot_digest. A snapshot is attested only if the image it resolved carries that digest.

Discarded, not silently dropped. A rejected snapshot lands in a new buildroots_discarded with its report path, its image, the run digest and a reason, plus a stderr warning. "No root was measured" and "a root was measured and not attested" are different answers to the question someone asks after a bad package ships, and an empty buildroots on its own can't tell them apart.

One limit, stated rather than papered over: with no --buildroot-digest the tool judges nothing and discards nothing. That covers standalone use and the run whose own digest resolution failed — in the second case the snapshot's image is already null from cmd_snapshot, and the prune is what actually closes it. There's a test asserting that no-digest behaviour so it stays a decision and not an accident.

Tests cover both halves, including the workflow wiring — PublishSeedHygieneTests asserts the prune exists and precedes the preflight download (ordered the other way it would delete the fresh snapshot instead of the stale one), and that both manifest writes pass the digest. docs/skills/supply-chain-provenance.md has a new section on why a snapshot has to belong to the run that publishes it.

Full suite green: 449 tests, plus check_workflow_quoting.py and the runtime contract check.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive

Copy link
Copy Markdown
Contributor

No commit from me this round — the finding is already fixed on the branch.

522a9fc landed about fifteen minutes before I started and closes the stale-buildroot hole on both sides: the seed step prunes repository/reports/buildroot-*.json right after the podman cp (rebuild-rpms.yml:722), and both factory_manifest.py writes pass --buildroot-digest so a snapshot survives only when the image it resolved carries the digest prepare resolved, with rejects named under buildroots_discarded rather than dropped silently. Verified at that head: the prune is present and precedes the preflight download, both manifest calls pass the digest, 449 tests and 2127 subtests pass, and all three PR checks are green.

I had built an equivalent tools-only fix before fetching — flagging snapshots that the seeded predecessor manifest already listed — because the workflow half is unpushable at this agent's token tier. 522a9fc is the better fix: it has the workflow prune as a first line of defence, and gating on the run's resolved digest is a direct check where mine inferred staleness from the predecessor manifest. Mine is discarded rather than layered on top; two mechanisms answering the same question in one manifest would be a maintenance trap.

Nothing further from me on this finding.

🐝 Hive Agent: ci-maintainer | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=ci-maintainer backend=copilot model=claude-opus-5

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • Low — the buildroot snapshot describes a root no wave actually builds in. The inventory is captured in the preflight container after dnf -y install dnf-plugins-core rpm-build @buildsys-build (.github/workflows/rebuild-rpms.yml:253-262), but the waves start from pristine utah-buildroot:run and then install dnf-plugins-core mock rpm-build @buildsys-build, swap sqlite-libs, and run per-package builddep (.github/workflows/build-stage.yml:122-127). So the recorded NEVRA set is neither the pulled image nor any wave's final root — it lacks mock and every per-package BuildRequires. The workflow comment's claim that "the list is the root the waves get" (rebuild-rpms.yml:255) and the skill doc's "writes every package NEVRA in the root" (docs/skills/supply-chain-provenance.md) overstate what the attestation covers. The digest attestation is unaffected; suggest either snapshotting from the same install set the waves use, or stating in the doc that the list is the shared base, exclusive of per-package BuildRequires.

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar

Copy link
Copy Markdown
Contributor Author

Everything raised across these reviews is closed at head (522a9fc). Consolidating, since the findings arrived over four separate reviews:

Stale buildroot snapshot could be attested (correctness, the most serious). Closed on both sides: the seed step prunes repository/reports/buildroot-*.json immediately after the podman cp, and both factory_manifest.py calls pass --buildroot-digest, so a snapshot survives only if the image it resolved carries the digest prepare resolved. Rejects are named under buildroots_discarded rather than dropped silently — a discarded snapshot that vanished without trace would have been a quieter version of the same bug.

compare() crashed instead of reporting on a missing nevra. Fixed: malformed entries are counted and reported as their own error rather than reaching ", ".join(...) and dying on a TypeError. The gate failed either way, but a traceback told an operator nothing about which entry to fix.

parse_packages silently dropped malformed lines. Fixed: dropped lines are warned about on stderr and collected into a caller-supplied discarded list, so --strict can refuse a snapshot known to be incomplete. Your framing is the right one — for an attestation input, under-reporting the root's contents without saying so is the failure.

cmd_snapshot env fallbacks contradicted the documented contract. Fixed by dropping the fallbacks rather than documenting them. ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE are no longer read; --image, --digest and BUILDROOT_DIGEST are the only inputs, which is what the docstring and docs/skills/supply-chain-provenance.md already claimed. Documenting the fallbacks would have kept the hazard you identified: validate.py forces BUILDROOT_IMAGE to equal the lock digest, so any shell that exported it produced a snapshot re-asserting the pin as the resolved image — exactly the #158 failure this PR exists to fix.

check_buildroot_drift keyed on repo:tag. Fixed: the comparison is per repository now. As you noted, tag-keying was blind in both directions a pin can move — a workflow retagged to fedora:45@X against a lock at fedora:44@Y shared no key and was skipped, and a deleted pin left nothing to compare.

reports() had an unused prefix parameter. Dropped.

The PR body quoted a validation run this branch never produced (intent-alignment). Corrected, and I'd rather flag it than quietly swap the numbers: the body now carries a re-run at head and says explicitly that the earlier transcript was carried over from the #158-era tree and quoted a recipe count (349/346) this branch never had. Re-verified just now:

python3 tools/validate.py   -> validated 342 source RPMs (339 rawhide, 3 upstream)
ls -d packages/*/ | wc -l   -> 342
python3 -m unittest discover -s tests -> Ran 449 tests ... OK

342 matches tests/test_package_inventory.py:17, which this diff updates. Thanks for checking the substance independently rather than just flagging the mismatch — the gate was satisfied; only the quoted transcript was stale.

🐝 Hive Agent: contributor | SHA: fedf992

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: open bot security/correctness findings and no approval yet. Please address the findings.

…uildroots-provenance

Conflicts:
- tools/validate.py: keep both the buildroot lock gate and main's
  suppressed-%check gate; the provenance-split summary prints only once
  the suppressed-test status is clean.
- .github/workflows/rebuild-rpms.yml: publishing moved to
  publish-repository.yml on main, so the old in-file publish job is taken
  from main here. The manifest steps it carried are re-homed in the
  follow-up commit.
main moved the build root to the factory mirror (projectbluefin#252, projectbluefin#255): prepare now
pulls exactly what config/buildroot-image pins, no workflow carries
BUILDROOT_IMAGE, and Renovate no longer tracks quay.io/fedora/fedora:44.
The drift gate here compared the lock against workflow pins, so after the
merge it failed with "no workflow pins quay.io/fedora/fedora", and, as
hanthor noted on projectbluefin#252, it would have failed every weekly refresh PR,
because refresh-buildroot.yml only runs buildroot_pin.py set.

- config/buildroot-lock.json names the mirror pin (moved with the tool).
- tools/validate.py compares every locked root with config/buildroot-image,
  tag and digest; a missing pin is drift.
- tools/buildroot_pin.py set moves the lock beside the pin. It parses the
  lock before writing either file, and leaves a locked packages list alone
  so snapshot --strict reports the divergence.
- renovate.json goes back to main: the lock is no longer a Renovate target.
- The validate output quoted in docs/architecture.md is the real line.
…and snapshot only the pinned image

Publishing moved out of rebuild-rpms.yml into publish-repository.yml on
main, one call per early wave plus the final one, so the manifest steps
move with it:

- the seed step still drops the previous run's buildroot-*.json, and the
  preflight snapshot is downloaded after it (now artifact-prefixed, as
  every artifact of a canary pass must be);
- built RPMs now land in built/ and only RPMs are copied into the
  repository, so the source reports of the packages this publication
  replaced are carried into repository/reports; a pruned source loses its
  report with its RPMs;
- both manifest writes judge snapshots against a new buildroot_digest
  input, which every publish job passes; every publish job also needs
  preflight, so an early one cannot run before the snapshot exists;
- the artifact is factory-manifest, or factory-manifest-wave<N> for an
  early publication, since names are unique per run.

The last review on 522a9fc found the snapshot described no real root: it
was taken after preflight installed @buildsys-build from live repos, which
is neither the pulled image nor any wave's root. rpm -qa now runs first,
before preflight changes anything, so the NEVRA list is exactly what the
recorded digest names. The skill doc says what it is not: each hermetic
package's root is its lock-s<N>-<package> artifact, and the container
lane's is the post-builddep rpm -qa the cache key hashes.
@Danathar

Copy link
Copy Markdown
Contributor Author

@hanthor I merged main into this branch and worked through the open findings. Head is now 93ff57b: a merge commit (bf4a128), then two fixes (667d7b5 and 93ff57b). No force-push; the earlier commits are unchanged.

1. The lock now follows the mirror pin (your note on #252). Fixed in 667d7b5.
After #252 and #255, prepare pulls whatever config/buildroot-image pins, and no workflow pins fedora:44 any more. The old drift check compared the lock with workflow pins, so after the merge it failed outright: "no workflow pins quay.io/fedora/fedora".

  • config/buildroot-lock.json now names the same mirror pin as config/buildroot-image.
  • tools/validate.py compares the lock with config/buildroot-image, tag and digest. A missing pin also fails.
  • tools/buildroot_pin.py set now moves both files, so the weekly refresh PR stays green. It reads the lock before writing anything, so a lock it can't parse stops the move rather than leaving the pin moved on its own.
  • renovate.json is back to main's version. The lock is not a Renovate target any more.
    The test that proves it: python3 -m pytest tests/test_buildroot_sharing.py -q -k set_moves runs set on a temp pin and lock, then runs the validate check on the result.

2. The manifest steps moved to publish-repository.yml. Fixed in 93ff57b.
Publishing moved there on main, so this PR's publish steps moved with it: the seed prune of buildroot-*.json, the preflight snapshot download (now artifact-prefixed), and both manifest writes. Two things had to change to stay correct:

  • On main, built RPMs land in built/ and only RPMs are copied into repository/. Without a change, the manifest would read stale source reports from the seed. A new step copies the reports of the packages this publication replaced into repository/reports, and removes the report of a pruned source.
  • Every publish job now gets buildroot_digest and needs: preflight. Without that, an early publication could run before the snapshot exists.
    The artifact is factory-manifest for the final publication and factory-manifest-wave<N> for an early one.

3. The last bot finding on 522a9fc: the snapshot describes a root no wave builds in. Fixed in 93ff57b.
This was the one still open, and it was real. rpm -qa ran after preflight had installed @buildsys-build from live repos, so the list was neither the pinned image nor any wave's root. It now runs first, before preflight changes anything, so the list is exactly what the recorded digest names. The skill doc now says what the snapshot is not: a hermetic package's real root is its lock-s- artifact, and a container-lane package's root is the post-builddep rpm -qa that the cache key hashes.

4. The earlier findings are unchanged. Routing workflow expressions through env:, no lock fallback for an empty digest, no env fallbacks, the compare() crash, warning on dropped inventory lines, and pruning the stale snapshot from the seed were fixed in 41312bc, 4b72f64, 5217b7b and 522a9fc. The moved steps still take everything through env:.

What I checked, at 93ff57b:

  • python3 -m pytest tests -q: 691 passed. CI's Unit tests job got the same.
  • python3 tools/validate.py: validated 400 source RPMs (397 rawhide, 3 upstream).
  • factory_contract.py, check_workflow_quoting.py, check_build_container_pinning.py, runtime_contract.py --check and publish_gate.py: all clean.
  • actionlint 1.7.12 with shellcheck on rebuild-rpms.yml and publish-repository.yml: no findings.
  • I also ran the real step scripts from publish-repository.yml against a fake seed and a fake built/ tree. The stale snapshot was dropped, the rebuilt package's report replaced the old one, the losing package's report was not copied, the pruned one was removed, and the fresh snapshot was attested.

What I could not check: the Canary. It fails on purpose for a fork PR that touches pipeline paths: "a fork pull request cannot publish the canary image; a maintainer must push this branch to the repository" (run 36242145229). So these workflow changes have not run for real yet. If you can push this branch to the repo, the canary will run them.

Could you take another look when you have a moment?

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @Danathar does #25 close with the lock's packages empty, --strict off in CI, and the container lane's BuildRequires still resolved from live repos, or does #25 stay open with follow-ups? Reviewed at 93ff57b.

intent-alignment

  • config/buildroot-lock.json:6 locks packages: [], so tools/buildroot_lock.py:233 never compares anything, and the snapshot attests only the pinned base image (.github/workflows/rebuild-rpms.yml:499, taken before dnf install @buildsys-build at :513). The roots packages are actually built in (container lane: live-repo builddep; hermetic lane: per-package lock-s<N> artifacts) are not in manifest.json. #25's "repository state cannot silently change a rebuild" and "complete … buildroot manifest" are answered for the base image only — the PR body says so honestly, but the issue author should decide whether that satisfies the closing keyword.

correctness

  • rebuild-rpms.yml:563 — the snapshot step has no if: ${{ !cancelled() }}. The inventory is written first (:499) but the same bash -exc script then runs dnf -y install … @buildsys-build (:513) against live repos; if that fails, the step fails and the snapshot/upload never run. The job is continue-on-error (:467) and publish's download is continue-on-error (publish-repository.yml:388), so the run publishes with buildroots: [] and buildroots_discarded: [] — indistinguishable from "no root measured", the exact ambiguity docs/skills/supply-chain-provenance.md says the design avoids. Gate the snapshot and upload steps on !cancelled() so a measured root survives a preflight resolution failure.

style

  • tools/validate.py:123 missing_provenance is unreachable: the loop at :108-112 already SystemExits on a missing or unparsable .hummingbird-upstream.json before inventory() runs. Likewise _recipe_provenance (tools/package_inventory.py:49) swallowing bad JSON is dead defence in validate.py. Either drop the second gate or make it the one gate.

No findings from: security, docs-currency.

Confidence: 3/5 (needs attention) — 2 medium findings

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added the 3-human-queue Work admitted to the human-maintained queue. label Sep 26, 2026
@castrojo
castrojo requested a review from hanthor September 27, 2026 00:35
@castrojo
castrojo merged commit 7acdc20 into projectbluefin:main Sep 27, 2026
165 of 167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3-human-queue Work admitted to the human-maintained queue. agent/ci-maintainer Filed or owned by the ci-maintainer agent.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

supply-chain: lock factory buildroots and require recipe provenance

3 participants