Skip to content

feat(ci): rivet-driven verification gate + sticky PR comment#221

Merged
avrabe merged 8 commits into
mainfrom
feat/v0.10.x-rivet-verification-gate
May 15, 2026
Merged

feat(ci): rivet-driven verification gate + sticky PR comment#221
avrabe merged 8 commits into
mainfrom
feat/v0.10.x-rivet-verification-gate

Conversation

@avrabe
Copy link
Copy Markdown
Contributor

@avrabe avrabe commented May 14, 2026

Summary

Makes artifacts/verification.yaml executable rather than purely descriptive. A new CI job runs every type: feature artifact's fields.steps[].run commands and posts a sticky PR comment with pass/fail counts + failed artifact IDs.

The PR comment looks like:

Rivet verification gate

177/177 passed

count
Passed 177
Failed 0
Skipped (no steps) 0

Filter: (and (= type "feature") (has-status "passing"))

On failure, the same comment lists the failing artifact IDs in a <details> block.

What's in this PR

  • tools/run_verification.py — Python (stdlib only). Calls rivet list + rivet get, executes each step under bash -c, writes verification-results.json. Same script runs locally for fast feedback.
  • tools/post_verification_comment.py — finds the marker comment via gh api and PATCHes the body; creates new only on first run. No comment spam across re-runs.
  • .github/workflows/verification-gate.yml — new CI job on PRs. Per-PR filter override: add Verify-Filter: <sexp> to the PR body.
  • artifacts/verification.yaml: quote cargo test ... enumerate:: step commands (line ~1615) — unquoted trailing :: made rivet silently skip the whole file. Pre-existing parse error blocking the new gate.
  • artifacts/{requirements,verification}.yaml: REQ-VERIFY-GATE-001 + TEST-VERIFY-GATE-RUNNER documenting the new capability.

Local use

# all feature artifacts:
tools/run_verification.py
# v0.9.x closeout subset only:
tools/run_verification.py --filter '(and (= type "feature") (has-tag "v093"))'
# self-test:
tools/run_verification.py --filter '(and (= type "feature") (has-tag "classifier-match"))'

Security note

The PR-body filter override (Verify-Filter: <sexp> in the PR description) is passed via env-var indirection (env: PR_BODY: ${{ github.event.pull_request.body }} + printf '%s' "$PR_BODY"), not direct ${{ }} interpolation in the run: block. This prevents the classic script-injection attack where a malicious PR body executes arbitrary commands on the runner. The security-reminder hook caught the original draft and flagged it.

Test plan

  • Local smoke test passes: tools/run_verification.py --filter '(and (= type "feature") (has-tag "classifier-match"))' runs 1 artifact, emits clean JSON, exits 0
  • rivet validate passes
  • Workflow runs on this PR (will produce the first sticky comment as a meta-demo)
  • CI green

Co-Authored-By: Claude Opus 4.7 noreply@anthropic.com

Makes artifacts/verification.yaml *executable* rather than purely descriptive.
A new CI job iterates every `type: feature` artifact, runs its
`fields.steps[].run` commands via the existing rivet CLI, aggregates per-
artifact pass/fail, and upserts a single marker-tagged PR comment with the
counts and failed artifact IDs.

Same script runs locally:

    tools/run_verification.py --filter '(and (= type "feature") (has-tag "v093"))'

Per-PR override: add `Verify-Filter: <sexp>` to the PR body to scope what runs.

What landed:
- tools/run_verification.py — Python (stdlib only). Calls `rivet list`+
  `rivet get`, executes each step under `bash -c`, writes a
  verification-results.json with passed/failed/skipped lists.
- tools/post_verification_comment.py — finds the marker comment via
  `gh api` and PATCHes the body; creates a new comment only on first run.
  PR-comment markdown shows N/M passed, a table, and a `<details>` block
  of failed IDs.
- .github/workflows/verification-gate.yml — new CI job on PRs. Pulls PR
  body via env (avoiding script-injection on `${{ ... }}` interpolation).
- artifacts/verification.yaml: quote `cargo test ... enumerate::` step
  commands (line ~1615) — unquoted trailing `::` made rivet's YAML
  parser silently skip the entire file. Required for any rivet command
  to see the verification artifacts at all.
- artifacts/{requirements,verification}.yaml: REQ-VERIFY-GATE-001 +
  TEST-VERIFY-GATE-RUNNER documenting the new gate.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@avrabe avrabe enabled auto-merge (squash) May 14, 2026 10:46
…inary packages)

The rivet workspace ships `rivet-cli` (the user-facing CLI binary `rivet`) and
`rivet-fuzz` side-by-side. `cargo install --git ... --bin rivet` fails because
cargo can't disambiguate which package owns the binary; add `--package rivet-cli`
to select the right one.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

avrabe and others added 5 commits May 14, 2026 13:30
cargo install rejects --package; the crate name is the positional argument.
Switch from `--package rivet-cli` to the trailing positional `rivet-cli`,
which is the form cargo's error message itself suggests.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# Conflicts:
#	artifacts/requirements.yaml
#	artifacts/verification.yaml
…not rivet's

The 4c06709 SHA I pinned earlier is etch's revision (the layout-engine sibling
crate in the rivet workspace). It pre-dates rivet's `--filter` flag, so the
verification gate ran against an old CLI that errored on `--filter`.

Switch to b7a17be — rivet v0.7.0, the first release that ships
`rivet list --filter <sexp>` (verified locally: `rivet --version` reports
`0.7.0 (b7a17bef main 2026-04-30)`).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…rrent milestone

Two corrections to the verification gate's default filter:

1. `has-status` is not a rivet operator — rivet v0.7.0 supports the family
   and/or/not/implies/excludes/=/!=/>/</has-tag/has-field/in/matches/contains/
   linked-*. Use `(= status "passing")` semantics where needed.

2. Running every `type: feature` artifact is 124 entries — they invoke
   `cargo test -p <crate>` across the whole workspace and easily exceed the
   30-minute job timeout. Narrow the default to the current-milestone scope
   (v093 + v0100 tags = ~12 artifacts), which runs in minutes. Per-PR
   override via `Verify-Filter: <sexp>` in the PR body is unchanged.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ep name

Two end-to-end fixes surfaced by the gate's first real run on PR #221:

1. The post-comment script used the `gh` CLI which is not installed on the
   self-hosted runners. Rewrote `post_verification_comment.py` to call the
   GitHub REST API directly via stdlib `urllib.request` — no external deps.
   Same sticky-comment behavior (marker lookup, PATCH or POST).

2. TEST-BINDING-NESTED-PATH's second step was `cargo test -p spar-cli
   applies_to_nested`, but the spar-cli crate's *package* name is `spar`,
   not `spar-cli`. The step never matched anything until the gate started
   actually executing it. Corrected to `cargo test -p spar --test
   applies_to_nested` — runs the 3 integration tests cleanly.

This is exactly the drift the gate is built to catch: the YAML claimed a
test was passing, but the test invocation was wrong all along.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 14, 2026

Rivet verification gate

13/13 passed

count
Passed 13
Failed 0
Skipped (no steps) 0

Filter: (and (= type "feature") (or (has-tag "v093") (has-tag "v0100")))

Failed artifacts

(none)

Updated automatically by tools/post_verification_comment.py. Source of truth: artifacts/verification.yaml.

# Conflicts:
#	artifacts/requirements.yaml
#	artifacts/verification.yaml
@avrabe avrabe merged commit ba329f3 into main May 15, 2026
18 checks passed
@avrabe avrabe deleted the feat/v0.10.x-rivet-verification-gate branch May 15, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant