feat(release): attest a CycloneDX bill of materials for each tag by flyingrobots · Pull Request #288 · flyingrobots/colorful-language

flyingrobots · 2026-08-01T15:48:12Z

What changed

Build provenance attestation already shipped (release.yml actions/attest);
the bill-of-materials half did not. The tag workflow now generates an attested
CycloneDX SBOM for every release.

Closes #227

Evidence

Workflow. Install SBOM tool pins cargo-cyclonedx@0.5.9 through the
established taiki-e/install-action pattern (same reviewed SHA already used for
cargo-mutants, cargo-llvm-cov, cargo-deny, and zizmor). Generate SBOM
builds a CycloneDX document from the locked dependency graph into dist/,
so it publishes as a release asset via the existing gh release create … dist/*.
The SBOM is added to the Attest Homebrew and editor artifacts subject-path, so
it carries the same GitHub/Sigstore provenance as the bytes it describes.

Ordering is load-bearing and enforced. Generation runs after the native
archives are downloaded, so the dependency graph ships beside the artifacts it
covers, and before attestation, so it cannot be published unattested. Both
constraints are expressed in REVIEWED_RELEASE_STEP_ORDER rather than left to
convention.

Policy. check-release-distribution.mjs gains DIST-9 enforcement.
Deterministic mutations reject each refused shape:

Mutation	Rejected because
generation step removed	no SBOM is produced
tool unpinned (`cargo-cyclonedx` with no version)	release inputs must be exact
output written outside `dist/`	asset would not publish
step reordered before the native download	graph would not describe shipped bytes
attestation omitting the SBOM	unattested artifact

Two existing cases asserted the previous attestation message and are updated
deliberately, since the attested set genuinely changed — flagged rather than
quietly edited.

Packet. docs/goalposts/v0.4.0/release.md carries #227 in both Must ship
and Scoped slices; the validator's symmetric inventory admits 34 scoped
issues (was 33). ROADMAP moves #227 parked → delivered: it was parked pending
"the distribution and signing authorities", which #245 and #251 established.

Gates.

node --test scripts/check-release-distribution.test.mjs — 33/33, 0 fail.
node scripts/check-release-distribution.mjs — 3 native platforms, 2 Homebrew
platforms, 3 editor registries.
node scripts/check-release-packet.mjs — 4 goalposts, 34 scoped issues.
node scripts/check-workflow-security.mjs — zizmor 1.28.0, 5 workflows clean.
node scripts/check-dependency-update-policy.mjs — satisfied (pin authority).
mise exec node@22.23.1 -- bash scripts/release-prep.sh — RELEASE PREP
PASSED on the pushed tree: 80 mutants with 63 caught, 17 unviable, zero
survivors; packaged editor smoke; 74 Markdown files, 0 errors.

Why now

The tag is immutable. An SBOM absent at tag time cannot be added to that
release, so this had to land before v0.4.0 rather than after.

Checklist

Living references describe only what is true on main —
docs/topics/distribution/README.md describes the mechanism, and makes no
public-URL or availability claim.
Planned cases marked implemented with evidence — DIST-9 requirement and
DIST-9a case added to docs/topics/distribution/test-plan.md.
CHANGELOG.md / ROADMAP.md updated.
cargo fmt, cargo clippy -D warnings, cargo test pass locally.

Build provenance attestation already shipped; the bill-of-materials half did not. The tag workflow now installs an exact pinned cargo-cyclonedx, generates a CycloneDX SBOM from the locked dependency graph into dist/ so it publishes as a release asset, and attests it alongside the Homebrew formula, VSIX, and Zed source archive so it carries the same GitHub/Sigstore provenance as the bytes it describes. Ordering is load-bearing and enforced: generation runs after the native archives are downloaded, so the dependency graph ships beside the artifacts it covers, and before attestation, so it cannot be published unattested. check-release-distribution gains DIST-9 topology enforcement. Deterministic mutations reject a removed generation step, an unpinned tool, output written outside dist/, a step reordered before the native download, and an attestation omitting the SBOM. Two existing cases asserted the previous attestation message and are updated deliberately, since the attested set changed. The v0.4.0 packet is amended to carry the SBOM slice in both Must ship and Scoped slices; the validator's symmetric inventory now admits 34 scoped issues. ROADMAP moves the slice from parked to delivered: it was parked pending the distribution and signing authorities, which #245 and #251 established. Evidence: 33/33 release-distribution cases; live checker passes 3 native platforms, 2 Homebrew platforms, 3 editor registries; zizmor clean across 5 workflows; workflow-security and dependency-update policies satisfied; packet admitted at 4 goalposts and 34 scoped issues; full release-prep passed with 80 mutants and zero survivors. Refs #227.

coderabbitai · 2026-08-01T15:48:20Z

Warning

Review limit reached

@flyingrobots, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 38 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bf0d0d0f-5678-4b0e-83c9-e8fa793ed98e

📥 Commits

Reviewing files that changed from the base of the PR and between a09b262 and 8d20c9c.

📒 Files selected for processing (4)

docs/RELEASING.md
docs/topics/distribution/test-plan.md
scripts/check-release-distribution.mjs
scripts/check-release-distribution.test.mjs

Summary by CodeRabbit

New Features
- Releases now include CycloneDX Software Bill of Materials (SBOM) files for the CLI and language server.
- SBOMs are generated from the locked dependency graph and included in release provenance attestations.
Documentation
- Updated release, distribution, roadmap, and verification documentation to describe SBOM generation and validation.
Tests
- Added checks for SBOM creation, required tooling, artifact placement, workflow ordering, and attestation coverage.

Walkthrough

The release pipeline now generates CycloneDX SBOMs for colorful and colorful-lsp, copies them into dist, attests them with release artifacts, and validates the workflow and published provenance.

Changes

Release SBOM provenance

Layer / File(s)	Summary
Generate and attest release SBOMs `.github/workflows/release.yml`, `.continuum/release.yml`	The workflow installs pinned `cargo-cyclonedx`, generates two JSON SBOMs, copies them into `dist`, and adds them to attestation subjects.
Enforce SBOM release policy `scripts/check-release-distribution.mjs`, `scripts/check-release-distribution.test.mjs`	Validation enforces the tool pin, exact commands, output paths, step ordering, and SBOM attestation. Tests cover valid and invalid workflow mutations.
Document publication verification `docs/...`, `CHANGELOG.md`, `ROADMAP.md`	Documentation records SBOM generation, JSON validation, attestation checks, release evidence, and delivery of issue `#227`.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

flyingrobots/colorful-language#287: Extends the same SBOM and attestation machinery to separate colorful and colorful-lsp artifacts.
flyingrobots/colorful-language#246: Introduced related release distribution and provenance validation.
flyingrobots/colorful-language#252: Modified the same release validation scripts and tests for artifact attestation.

Poem

Two manifests rise from the build,
CycloneDX records what was filled.
Pinned tools forge the trail,
Attestations guard the scale,
Release proof is sealed and stilled.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the primary change: attesting a CycloneDX SBOM for each tag release.
Description check	✅ Passed	The description covers implementation, evidence, policy checks, documentation updates, linked issue closure, and completed checklist items.
Linked Issues check	✅ Passed	The PR satisfies issue `#227` by generating per-binary SBOMs, publishing them with releases, and including them in provenance attestation.
Out of Scope Changes check	✅ Passed	The workflow, validation, documentation, roadmap, changelog, and release-packet changes directly support issue `#227` and the stated SBOM objectives.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0894209f51

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… command Three review findings, all valid. Register the SBOM in the release verification authorities. The workflow produced and attested an asset that no post-publication procedure checked, so a missing or invalid public attestation would have passed manual verification despite the new claim. .continuum/release.yml now carries the SBOM in the required-artifact inventory, docs/RELEASING.md verifies it is well-formed JSON and attestation-verified alongside the formula, and the v0.4.0 publication witness gains a bill-of-materials row. Validate the executable command rather than token presence. The previous check tested for 'cargo cyclonedx', '--locked', and a dist/ path independently, which an inert stand-in such as echo plus touch would have satisfied, letting an arbitrary empty file be attested and published as the SBOM. Generation is now matched as an exact normalized command sequence, mirroring the Homebrew check, which also binds --format json and the dist/ destination to the invocation. A new mutation case rejects inert commands, an unbound format, an output written outside dist/, and an extra command smuggled after the reviewed sequence. Restore DIST-8a's evidence. Inserting DIST-9a orphaned the trailing Evidence and Status fields onto it, which named only the Homebrew generator, leaving DIST-8a without its implemented evidence and DIST-9a with irrelevant evidence. The Homebrew block returns to DIST-8a and DIST-9a gains its own. Evidence: 34/34 release-distribution cases; live checker passes; release profile OK; packet admitted at 4 goalposts and 34 scoped issues; Markdown clean. Refs #227.

flyingrobots · 2026-08-01T19:53:33Z

Code Lawyer self-audit — P0: the SBOM step cannot execute

Self-audit of origin/main...HEAD. I ran the pinned tool (cargo-cyclonedx 0.5.9,
aarch64-apple-darwin release binary) against this workspace. The Generate SBOM step is
non-functional, and the policy gate I added enforces the broken command as "reviewed".

This would surface only at tag time, on an immutable tag.

#	Sev	File	Finding	Evidence
1	P0	`.github/workflows/release.yml:248`	`cargo cyclonedx` rejects `--locked`	`error: unexpected argument '--locked' found` (exit 2). Under `set -euo pipefail` the release job aborts.
2	P0	`.github/workflows/release.yml:249`	Output is 9 files, one per crate, in each crate's directory, named `sbom.json`	Run emitted `crates/{colorful-cli,colorful-core,colorful-ir,colorful-lexicon,colorful-lint,colorful-lsp,colorful-parse,colorful-projection,colorful-vale}/sbom.json`. No repo-root `sbom.cdx.json` exists, so `cp sbom.cdx.json …` fails on filename, path, and cardinality.
3	P0	`.github/workflows/release.yml:288`	Attest glob `dist/sbom.cdx.json` matches nothing*	Consequence of #2. The attestation subject set would be empty.
4	P1	design	A single crate's SBOM does not cover both shipped binaries	`colorful-cli` SBOM = 47 components; `logos`/`sha2`/`serde`/`colorful-core` present, `tower-lsp` absent — that is `colorful-lsp`'s dependency. The release ships `colorful` and `colorful-lsp`.
5	P2	design	SBOM bytes are non-deterministic	`metadata.timestamp` is emitted on every run.
6	P2	`scripts/check-release-distribution.mjs:113`	The gate locks in the broken command	`REVIEWED_SBOM_COMMANDS` asserts exact equality with a sequence that has never executed. Tightening the check (correctly, per review) hardened a command nobody ran.

Root cause

I bound the policy to an exact command sequence — which was the right hardening, and Codex was
right to ask for it — but never executed that sequence. Exactness was verified; executability
was assumed. --describe crate (the default) emits per-package SBOMs; there is no
aggregate-workspace mode in 0.5.9.

Proposed fix

Generate for each shipped binary and ship both:

set -euo pipefail
cargo cyclonedx --format json --all --manifest-path crates/colorful-cli/Cargo.toml --override-filename sbom
cargo cyclonedx --format json --all --manifest-path crates/colorful-lsp/Cargo.toml --override-filename sbom
cp crates/colorful-cli/sbom.json "dist/colorful-language-${GITHUB_REF_NAME}-colorful-sbom.cdx.json"
cp crates/colorful-lsp/sbom.json "dist/colorful-language-${GITHUB_REF_NAME}-colorful-lsp-sbom.cdx.json"

Requires updating REVIEWED_SBOM_COMMANDS, the DIST-9a case, .continuum/release.yml,
docs/RELEASING.md, and the v0.4.0 witness row to the two-asset shape.

Merge gate: LOCKED. cc @codex for a second opinion on the two-artifact design and on whether
metadata.timestamp non-determinism is acceptable for a release artifact in this repo.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d9e6d6d0bf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Self-audit P0. The Generate SBOM step could not run. Verified by downloading the pinned cargo-cyclonedx 0.5.9 release binary and executing it against this workspace: 1. cargo cyclonedx rejects --locked 'error: unexpected argument --locked found', exit 2. Under set -euo pipefail the release job aborts. 2. Output is nine files, one per crate, written beside each crate's manifest and named sbom.json. No repo-root sbom.cdx.json is ever produced, so the cp failed on filename, path, and cardinality at once. 3. The attestation glob dist/*sbom.cdx.json therefore matched nothing. 4. A single crate's SBOM does not cover both shipped binaries: colorful-cli resolves 47 components without tower-lsp, which is colorful-lsp's dependency. None of this was reachable before a tag, because nothing in CI executes the release workflow. Root cause: the previous commit hardened the gate to require an exact command sequence, which was the correct review outcome, but exactness was verified while executability was assumed. The fix generates one SBOM per shipped binary, since cargo-cyclonedx emits per-package documents and has no aggregate-workspace mode. The replacement sequence was executed end to end before being admitted: all four commands succeed, both assets land in dist/, and together they cover the shipped graph (colorful 47 components; colorful-lsp 112 including tower-lsp). The release profile, runbook verification commands, DIST-9a, the v0.4.0 publication witness, and CHANGELOG all move to the two-asset shape. The packet validator additionally rejected a first draft of the witness row for asserting completed evidence inside a section declared unavailable; the row now states required evidence rather than a claim. Evidence: 34/34 release-distribution cases; 49/49 release-packet cases; live distribution checker, release profile, packet, zizmor, and dependency-update policy all pass; Markdown clean. Refs #227.

coderabbitai

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/RELEASING.md`:
- Around line 731-735: Update docs/RELEASING.md lines 731-735 so the SBOM
verification procedure records both SBOM filenames, JSON validation outcomes,
and attestation verification outcomes. Update
docs/topics/distribution/test-plan.md lines 144-146 to assign public SBOM
evidence to DIST-9a or add a dedicated SBOM evidence row; do not assign it to
DIST-7a.

In `@scripts/check-release-distribution.mjs`:
- Line 88: The mutable cargo-cyclonedx version is duplicated outside the
canonical policy gate. In scripts/check-release-distribution.mjs:88-88, remove
the local EXPECTED_SBOM_TOOL ownership and expose or reuse the policy script’s
canonical pin; update the validator to read that value. In
scripts/check-release-distribution.test.mjs:20-20, replace the hard-coded
cargo-cyclonedx@0.5.9 fixture with the same canonical policy value so future pin
updates remain synchronized.
- Around line 85-105: Update validateReleaseDistribution and its validSnapshot
contract to include the .continuum/release.yml publish.artifacts inventory,
requiring both expected SBOM asset paths defined by EXPECTED_SBOM_ASSET or the
reviewed SBOM commands. Add failure tests covering removal or renaming of each
required SBOM artifact entry so either inventory change is rejected.
- Around line 431-445: The SBOM installer validation in the checks around
sbomInstall must also require sbomInstall.with?.fallback to equal "none",
alongside the existing tool and command checks. Add a negative test configuring
fallback as "cargo-binstall" and assert that validation rejects it.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 8a973bd6-d465-4f89-917c-3dc962192e1f

📥 Commits

Reviewing files that changed from the base of the PR and between 5bbbc85 and a09b262.

📒 Files selected for processing (11)

.continuum/release.yml
.github/workflows/release.yml
CHANGELOG.md
ROADMAP.md
docs/RELEASING.md
docs/goalposts/v0.4.0/release.md
docs/goalposts/v0.4.0/verification.md
docs/topics/distribution/README.md
docs/topics/distribution/test-plan.md
scripts/check-release-distribution.mjs
scripts/check-release-distribution.test.mjs

📜 Review details

⏰ Context from checks skipped due to timeout. (8)

GitHub Check: Generated IR and vocabulary drift
GitHub Check: Rust coverage
GitHub Check: Editor integrations (compile)
GitHub Check: Downstream discovery version-compat matrix
GitHub Check: Cargo package witness
GitHub Check: Rust (fmt, clippy, test)
GitHub Check: CodeQL (rust)
GitHub Check: CodeQL (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (10)

.continuum/release.yml