From 9460959c18f6e0fa610d00296d785e33ec66a8f2 Mon Sep 17 00:00:00 2001 From: squid-protocol Date: Mon, 31 Aug 2026 17:02:39 -0400 Subject: [PATCH] ci: rosetta-audit gate -- pinned cross-language consistency check on engine PRs (#2557) The engine had no rosetta gate: a PR shifting corpus-observed counts merged green here and detonated keyword-rosetta later on an unrelated PR (the #2552 cobol drift surfaced days late that way). rosetta-audit.yml now checks out keyword-rosetta at the pinned KEYWORD_ROSETTA_REF Actions variable (set: 7686b0a) and runs all 46 verify gates + the corpus's baseline-gated n/a review audit against the PR's engine build -- path-filtered to engine code, ~6 min, workflow_dispatch for manual runs. BUMPING_THE_ROSETTA_PIN.md documents the re-baseline protocol (the crucible-pin twin): corpus re-baseline PR with ENGINE_REF=pull//head -> merge -> engine PR bumps the var. ecosystem.md's rosetta-sweep row updated to the new choreography (no more draft-PR limbo / rerun dance). Cross-repo: companion keyword-rosetta PRs #6 (n/a semantics) + #7 (ENGINE_REF + na_check); the n/a audit step self-activates when the pin advances past #7. Closes #2557 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PUfXMUECX4Vq1vqh9mMQqt --- .github/workflows/rosetta-audit.yml | 78 +++++++++++++++++++++++ docs/ecosystem.md | 2 +- docs/self_scan/BUMPING_THE_ROSETTA_PIN.md | 46 +++++++++++++ 3 files changed, 125 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/rosetta-audit.yml create mode 100644 docs/self_scan/BUMPING_THE_ROSETTA_PIN.md diff --git a/.github/workflows/rosetta-audit.yml b/.github/workflows/rosetta-audit.yml new file mode 100644 index 000000000..53256f83b --- /dev/null +++ b/.github/workflows/rosetta-audit.yml @@ -0,0 +1,78 @@ +name: rosetta-audit + +# Cross-language consistency gate (#2557, Phase 5 of the control-corpus plan): +# checks out keyword-rosetta at the pinned KEYWORD_ROSETTA_REF and runs its +# verifier across ALL 46 language folders against THIS PR's engine build, plus +# the corpus's baseline-gated n/a review audit. An engine change that shifts +# corpus-observed counts, or nulls a rule without a validated ledger entry, +# fails HERE -- in the PR that caused it -- instead of days later in the +# corpus repo (how the #2552 cobol drift was found). +# +# Intentional drift is re-baselined via the cross-repo protocol in +# docs/self_scan/BUMPING_THE_ROSETTA_PIN.md (the corpus repo's ENGINE_REF +# escape hatch + a bump of KEYWORD_ROSETTA_REF in the engine PR). + +on: + pull_request: + branches: [main, v6-dev] + paths: + - "gitgalaxy/**" + - "pyproject.toml" + - ".github/workflows/rosetta-audit.yml" + workflow_dispatch: + +jobs: + rosetta-audit: + runs-on: ubuntu-latest + steps: + - name: Checkout GitGalaxy PR + uses: actions/checkout@v4 + with: + path: gitgalaxy + persist-credentials: false + + - name: Checkout keyword-rosetta (pinned) + uses: actions/checkout@v4 + with: + repository: squid-protocol/keyword-rosetta + ref: ${{ vars.KEYWORD_ROSETTA_REF || 'main' }} + path: keyword-rosetta + persist-credentials: false + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: "pip" + + # Full-precision deps: without all of them galaxyscope silently drops to + # Zero-Dependency Mode and observed counts drift from what the corpus + # manifests were locked against. + - name: Install GitGalaxy (full precision) + run: | + python -m pip install --upgrade pip + pip install PyYAML networkx tiktoken numpy pandas xgboost + pip install -e ./gitgalaxy + + - name: Run all language gates + working-directory: keyword-rosetta + env: + GITGALAXY_PATH: ${{ github.workspace }}/gitgalaxy + run: | + rc=0 + for d in data/*/; do + L=$(basename "$d") + python tools/verify_language.py "$L" || rc=1 + done + exit $rc + + - name: n/a review audit (baseline-gated) + working-directory: keyword-rosetta + env: + GITGALAXY_PATH: ${{ github.workspace }}/gitgalaxy + run: | + if [ -f tools/na_check.py ]; then + python tools/na_check.py --ci + else + echo "::notice::pinned keyword-rosetta ref predates tools/na_check.py -- bump KEYWORD_ROSETTA_REF to activate the n/a audit" + fi diff --git a/docs/ecosystem.md b/docs/ecosystem.md index eaab04623..47c739d3f 100644 --- a/docs/ecosystem.md +++ b/docs/ecosystem.md @@ -80,7 +80,7 @@ rule, sync all three. | Workflow | Repos touched (in merge order) | Documented in | |---|---|---| | **Crucible corpus growth → release → pin bump** | language-crucible (data PRs, tag per `RELEASING.md`) → gitgalaxy (`docs/self_scan/BUMPING_THE_CRUCIBLE_PIN.md`: regen golden masters + tri-comparison + tree-sitter artifacts, bump `LANGUAGE_CRUCIBLE_REF` + `PINNED_TAG`) | crucible `RELEASING.md`; gitgalaxy `BUMPING_THE_CRUCIBLE_PIN.md` | -| **Rosetta sweep** (work one language's bias issue) | gitgalaxy engine PR first (its CI is self-contained) → keyword-rosetta corpus PR **stays draft until the engine PR merges**, because rosetta CI checks out gitgalaxy *main*; then `gh run rerun --failed` + `gh pr ready`. Capstone lands back in gitgalaxy `docs/language_status/.md` §10. | keyword-rosetta `rosetta-language-sweep` skill | +| **Rosetta sweep** (work one language's bias issue) | gitgalaxy engine PR first (its `rosetta-audit` check fails at the source — expected) → keyword-rosetta corpus re-baseline PR with `ENGINE_REF=pull//head` (green immediately, no draft limbo) → restore `ENGINE_REF=main`, merge corpus → engine PR bumps `KEYWORD_ROSETTA_REF` var, merges green. Capstone lands back in gitgalaxy `docs/language_status/.md` §10. | keyword-rosetta `rosetta-language-sweep` skill; gitgalaxy `docs/self_scan/BUMPING_THE_ROSETTA_PIN.md` | | **Adding a language to the engine** | gitgalaxy (`standards/how_to_add_a_language.md`, includes authoring the rosetta control folder) → keyword-rosetta (`SPEC.md` shell + manifest) → optionally language-crucible (`expand-language-coverage`) | those three docs | | **Tri-comparison / accuracy verification** | gitgalaxy only (ledger, chart, `manual_verification.json`), but reads the pinned crucible corpus | gitgalaxy `docs/self_scan/tri_comparison_README.md` | | **README / evidence claims** | gitgalaxy README cites gitgalaxy-raw-output artifacts and the keyword-rosetta chart (embedded from that repo's raw main URL — it self-updates when rosetta main moves) | gitgalaxy `readme-maintenance` skill | diff --git a/docs/self_scan/BUMPING_THE_ROSETTA_PIN.md b/docs/self_scan/BUMPING_THE_ROSETTA_PIN.md new file mode 100644 index 000000000..fe282a1a1 --- /dev/null +++ b/docs/self_scan/BUMPING_THE_ROSETTA_PIN.md @@ -0,0 +1,46 @@ +# Bumping the keyword-rosetta pin (KEYWORD_ROSETTA_REF) + +The `rosetta-audit` CI check (`.github/workflows/rosetta-audit.yml`, issue #2557) runs the +[keyword-rosetta](https://github.com/squid-protocol/keyword-rosetta) control corpus's verifier — +all 46 language folders plus its baseline-gated n/a review audit — against every PR that touches +engine code, with the corpus checked out at the **`KEYWORD_ROSETTA_REF`** GitHub Actions +variable (a commit SHA or tag; falls back to `main` if unset). This is the cross-language +*consistency* twin of the crucible pin (`docs/self_scan/BUMPING_THE_CRUCIBLE_PIN.md`), and the +same principle applies: the pin makes the gate deterministic, and bumping it is a deliberate, +reviewed act — never a way to make a red check go away. + +## When rosetta-audit fails on your PR + +That is the gate working: your change shifts what the engine observes on the planted corpus +(or removes a signal rule nobody has ledgered). Decide which of these you are in: + +1. **Unintentional regression** — fix the engine change. Do not touch the pin. +2. **Intentional, corpus-visible improvement** (a rule fix/addition, a stripper change): the + expected values live in the corpus repo, so re-baseline there, then bump the pin here: + + 1. In keyword-rosetta, open the re-baseline PR per its `docs/GATING.md`: manifest edits + + a validated `deviation_ledger.json` entry (or `still_reproduces` flip) justifying every + changed number — with the committed **`ENGINE_REF`** file set to `pull//head` for + YOUR engine PR `N`, so its gates run against your unmerged build and go green now. + 3. Restore `ENGINE_REF` to `main` and merge the corpus PR (once your engine PR is approved). + 4. In your engine PR: update `KEYWORD_ROSETTA_REF` to the corpus repo's new main SHA — + `gh api -X PATCH repos/squid-protocol/gitgalaxy/actions/variables/KEYWORD_ROSETTA_REF -f value=` + (needs repo admin; in a review, request the maintainer do it) — and note the bump + the + companion corpus PR in the PR body's **Cross-repo** section. + 5. `rosetta-audit` reruns green → merge. + +3. **A new rule absence** (you nulled/removed a rule): `na_check --ci` fails until the corpus + PR ships a validated ledger entry naming the language and signal — absence is either real + morphology (ledger it) or a gap (don't ship it). Never regenerate `docs/na_baseline.json` + to absorb an unreviewed cell. + +## Invariants + +- The variable always points at a keyword-rosetta commit whose gates pass against the engine + `main` that existed when it was set (the brief corpus-ahead-of-engine window during step 2 + is covered by this gate's own pin). +- Never bump the pin and engine behavior in *unrelated* PRs simultaneously; the bump belongs + to the PR whose behavior change it validates. +- The corpus repo's independence is the point: expected values and their audit trail + (`deviation_ledger.json`) live there, under its gating rules — this repo only pins which + snapshot it holds itself accountable to.