feat(ci): rivet-driven verification gate + sticky PR comment#221
Merged
Conversation
Makes artifacts/verification.yaml *executable* rather than purely descriptive.
A new CI job iterates every `type: feature` artifact, runs its
`fields.steps[].run` commands via the existing rivet CLI, aggregates per-
artifact pass/fail, and upserts a single marker-tagged PR comment with the
counts and failed artifact IDs.
Same script runs locally:
tools/run_verification.py --filter '(and (= type "feature") (has-tag "v093"))'
Per-PR override: add `Verify-Filter: <sexp>` to the PR body to scope what runs.
What landed:
- tools/run_verification.py — Python (stdlib only). Calls `rivet list`+
`rivet get`, executes each step under `bash -c`, writes a
verification-results.json with passed/failed/skipped lists.
- tools/post_verification_comment.py — finds the marker comment via
`gh api` and PATCHes the body; creates a new comment only on first run.
PR-comment markdown shows N/M passed, a table, and a `<details>` block
of failed IDs.
- .github/workflows/verification-gate.yml — new CI job on PRs. Pulls PR
body via env (avoiding script-injection on `${{ ... }}` interpolation).
- artifacts/verification.yaml: quote `cargo test ... enumerate::` step
commands (line ~1615) — unquoted trailing `::` made rivet's YAML
parser silently skip the entire file. Required for any rivet command
to see the verification artifacts at all.
- artifacts/{requirements,verification}.yaml: REQ-VERIFY-GATE-001 +
TEST-VERIFY-GATE-RUNNER documenting the new gate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…inary packages) The rivet workspace ships `rivet-cli` (the user-facing CLI binary `rivet`) and `rivet-fuzz` side-by-side. `cargo install --git ... --bin rivet` fails because cargo can't disambiguate which package owns the binary; add `--package rivet-cli` to select the right one. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
cargo install rejects --package; the crate name is the positional argument. Switch from `--package rivet-cli` to the trailing positional `rivet-cli`, which is the form cargo's error message itself suggests. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# Conflicts: # artifacts/requirements.yaml # artifacts/verification.yaml
…not rivet's The 4c06709 SHA I pinned earlier is etch's revision (the layout-engine sibling crate in the rivet workspace). It pre-dates rivet's `--filter` flag, so the verification gate ran against an old CLI that errored on `--filter`. Switch to b7a17be — rivet v0.7.0, the first release that ships `rivet list --filter <sexp>` (verified locally: `rivet --version` reports `0.7.0 (b7a17bef main 2026-04-30)`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…rrent milestone Two corrections to the verification gate's default filter: 1. `has-status` is not a rivet operator — rivet v0.7.0 supports the family and/or/not/implies/excludes/=/!=/>/</has-tag/has-field/in/matches/contains/ linked-*. Use `(= status "passing")` semantics where needed. 2. Running every `type: feature` artifact is 124 entries — they invoke `cargo test -p <crate>` across the whole workspace and easily exceed the 30-minute job timeout. Narrow the default to the current-milestone scope (v093 + v0100 tags = ~12 artifacts), which runs in minutes. Per-PR override via `Verify-Filter: <sexp>` in the PR body is unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ep name Two end-to-end fixes surfaced by the gate's first real run on PR #221: 1. The post-comment script used the `gh` CLI which is not installed on the self-hosted runners. Rewrote `post_verification_comment.py` to call the GitHub REST API directly via stdlib `urllib.request` — no external deps. Same sticky-comment behavior (marker lookup, PATCH or POST). 2. TEST-BINDING-NESTED-PATH's second step was `cargo test -p spar-cli applies_to_nested`, but the spar-cli crate's *package* name is `spar`, not `spar-cli`. The step never matched anything until the gate started actually executing it. Corrected to `cargo test -p spar --test applies_to_nested` — runs the 3 integration tests cleanly. This is exactly the drift the gate is built to catch: the YAML claimed a test was passing, but the test invocation was wrong all along. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rivet verification gate✅ 13/13 passed
Filter: Failed artifacts(none) Updated automatically by |
# Conflicts: # artifacts/requirements.yaml # artifacts/verification.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes
artifacts/verification.yamlexecutable rather than purely descriptive. A new CI job runs everytype: featureartifact'sfields.steps[].runcommands and posts a sticky PR comment with pass/fail counts + failed artifact IDs.The PR comment looks like:
On failure, the same comment lists the failing artifact IDs in a
<details>block.What's in this PR
tools/run_verification.py— Python (stdlib only). Callsrivet list+rivet get, executes each step underbash -c, writesverification-results.json. Same script runs locally for fast feedback.tools/post_verification_comment.py— finds the marker comment viagh apiand PATCHes the body; creates new only on first run. No comment spam across re-runs..github/workflows/verification-gate.yml— new CI job on PRs. Per-PR filter override: addVerify-Filter: <sexp>to the PR body.artifacts/verification.yaml: quotecargo test ... enumerate::step commands (line ~1615) — unquoted trailing::made rivet silently skip the whole file. Pre-existing parse error blocking the new gate.artifacts/{requirements,verification}.yaml:REQ-VERIFY-GATE-001+TEST-VERIFY-GATE-RUNNERdocumenting the new capability.Local use
Security note
The PR-body filter override (
Verify-Filter: <sexp>in the PR description) is passed via env-var indirection (env: PR_BODY: ${{ github.event.pull_request.body }}+printf '%s' "$PR_BODY"), not direct${{ }}interpolation in therun:block. This prevents the classic script-injection attack where a malicious PR body executes arbitrary commands on the runner. The security-reminder hook caught the original draft and flagged it.Test plan
tools/run_verification.py --filter '(and (= type "feature") (has-tag "classifier-match"))'runs 1 artifact, emits clean JSON, exits 0rivet validatepassesCo-Authored-By: Claude Opus 4.7 noreply@anthropic.com