Skip to content

fix(ci): verify cosign signature before seeding from the factory image - #215

Merged
castrojo merged 2 commits into
projectbluefin:mainfrom
Danathar:sec/cosign-verify-seed-image
Sep 23, 2026
Merged

castrojo merged 2 commits into
projectbluefin:mainfrom
Danathar:sec/cosign-verify-seed-image

Conversation

@Danathar

@Danathar Danathar commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #180 (same fix against #173), which hanthor closed on 2026-09-20
asking for "a fresh, rebased PR after that build establishes the current
baseline." #178 is merged; this branches from ef320c1 (current main).

Summary

In .github/workflows/rebuild-rpms.yml, the publish job's seed step
("Verify and seed repository from the prepare-time factory image") pulled
ghcr.io/<owner>/utah-packages:latest (or the current branch's own tag) and
extracted every RPM in it into repository/ without checking the cosign
signature this same workflow attaches to the image it publishes ("Sign the
published image" + actions/attest-build-provenance).

Anyone able to push to the utah-packages GHCR package once — a leaked
token, a compromised workflow holding packages: write, or a registry-side
compromise — could plant RPMs in the mutable tag and have every subsequent
run copy them forward, sign repomd.xml over them, publish a new signed
image, and attest provenance for it. The factory would launder an unverified
input into a verified one; the existing staleness check (comparing against
needs.prepare.outputs.factory_image) does not catch this because it only
detects the tag moving mid-run, not a bad image being present in the first
place.

Change

  • Installs cosign (same pinned v2.4.1 binary + checksum already used later in
    the job) before the seed step; the later re-install stays, since it is a
    harmless no-op and keeping it avoids coupling the sign-blob step to a side
    effect of an earlier, unrelated step.
  • After resolving which tag matched (latest vs. the branch tag) and pulling
    by digest, verifies cosign verify against this workflow's own keyless
    OIDC identity — refs/heads/main when the seed came from latest, the
    current ref otherwise — before creating a container or copying anything out
    of it. Verification runs on the digest actually pulled, never the mutable
    tag, so the tag cannot move between verification and extraction.
  • Logs in to GHCR with --authfile "$DOCKER_CONFIG/config.json" so cosign
    (which reads the Docker config, not podman's own auth store — see the
    existing comment on the "Publish the repository" step) actually sees the
    credentials the pull used.
  • First-run bootstrap (no prior image) is unchanged.
  • Adds regression tests in tests/test_atomic_publish.py asserting the seed
    step calls cosign verify with an OIDC issuer and certificate identity,
    verifies the resolved digest rather than a bare tag, does so before
    podman cp, logs in where cosign will look, and that cosign is installed
    before the seed step runs.
  • Documents the new cosign verify failure mode in
    .agents/skills/build-failure-triage/SKILL.md, and the underlying rule
    ("freshness is not trust") as entry 16 in docs/skills/repeated-mistakes.md.

Verification

  • python3 -m unittest discover -s tests — 416 tests, all pass (including
    the 5 new regression tests).
  • python3 tools/check_workflow_quoting.py — clean.
  • python3 tools/factory_contract.py / python3 tools/validate.py — both
    satisfied.
  • Ran the actual pinned cosign binary locally against the live published
    image with the exact identity this change now enforces:
    cosign verify ghcr.io/projectbluefin/utah-packages:latest \
      --certificate-oidc-issuer https://token.actions.githubusercontent.com \
      --certificate-identity https://github.com/projectbluefin/utah-packages/.github/workflows/rebuild-rpms.yml@refs/heads/main
    
    This succeeds against the current :latest (verified offline against the
    transparency log, signed by this repository's rebuild-rpms.yml on
    refs/heads/main, per the returned certificate claims), confirming the
    next scheduled publish run will not be broken by this check.

Fixes #173

— hive: backend=omp model=anthropic/claude-sonnet-5 effort=medium

@Danathar
Danathar force-pushed the sec/cosign-verify-seed-image branch from a77a759 to d7fdb3c Compare September 20, 2026 19:53

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: Properly verifies cosign signature before seeding from the factory OCI image in .github/workflows/rebuild-rpms.yml, and adds test coverage in tests/test_atomic_publish.py with documentation updates in docs/skills/repeated-mistakes.md. Clean trial merge against origin/main, passes factory contract check and test suite (416 passed, 2127 subtests passed). Awaiting second independent human review to satisfy merge policy.

The publish job's seed step pulled ghcr.io/<owner>/utah-packages:latest (or
the branch tag) and copied every RPM out of it without checking the cosign
signature this same workflow attaches to the image it publishes ('Sign the
published image' + actions/attest-build-provenance). Anyone able to push to
the GHCR package once -- a leaked token, a compromised workflow with
packages: write, or a registry-side compromise -- could plant RPMs in the
mutable tag and have every subsequent run copy them forward, sign
repomd.xml over them, publish a new signed image, and attest provenance for
it, laundering an unverified input into a verified one. The existing
staleness check (comparing against needs.prepare.outputs.factory_image) does
not catch this: it only detects the tag moving mid-run, not a bad image
already being current.

Install cosign before the seed step and verify the digest actually pulled
(never the mutable tag, which can move again after the check) against this
workflow's own keyless OIDC identity -- refs/heads/main when the seed came
from :latest, the current branch ref when it came from the branch tag --
before creating a container or copying anything out of it. Log in with an
authfile that matches DOCKER_CONFIG, the same auth-store split the publish
step's comment already documents, so cosign verify actually sees the
credentials the pull used.

Add regression tests asserting the seed step calls cosign verify with an
OIDC issuer and identity, verifies the resolved digest rather than a bare
tag, does so before podman cp, is installed ahead of the seed step, and
logs in where cosign will look. Document the new cosign-verify-failure
triage entry in build-failure-triage, and the underlying rule -- freshness
is not trust -- in repeated-mistakes.

Verified against the live image:
cosign verify ghcr.io/projectbluefin/utah-packages:latest \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity https://github.com/projectbluefin/utah-packages/.github/workflows/rebuild-rpms.yml@refs/heads/main
succeeds, so the identity this change pins matches how the workflow already
signs.

Fixes projectbluefin#173

Assisted-by: Claude Sonnet 5 via hive
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@castrojo
castrojo force-pushed the sec/cosign-verify-seed-image branch from d7fdb3c to d3faad7 Compare September 21, 2026 03:06
@castrojo
castrojo requested a review from hanthor September 21, 2026 03:24

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: cosign verify-before-seed in rebuild-rpms.yml plus skill doc. CI 3/3 green, CLEAN, bot no-findings, diff inspected. Good to merge once a second human confirms (per castrojo's note).

@castrojo
castrojo merged commit 9e0c1f1 into projectbluefin:main Sep 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[sec-check] rebuild-rpms.yml seed step consumes ghcr :latest without cosign verification — poisoned RPMs would be carried forward and re-signed

3 participants