fix(ci): guard corpus-pin audits against an empty LANGUAGE_CRUCIBLE_REF - #2637
Merged
Conversation
GitHub withholds repository variables -- like secrets -- from pull_request runs raised from a fork, so `vars.LANGUAGE_CRUCIBLE_REF` expands to an empty string there and the five corpus-backed workflows ran `git clone --branch ""`, failing with a bare `fatal: Remote branch not found in upstream origin` before any engine code was exercised. #2628 is the first fork PR this repo has seen in a long while, which is why this went unnoticed: it arrived with four red audits that had nothing to do with the contributor's change. Guard the clone instead of falling back to the corpus's `main`: these pins exist to make the audits deterministic, and silently auditing against a different corpus snapshot would trade a confusing failure for an authoritative-looking wrong answer. The guard fails with an explicit `::error::` explaining the fork limitation and pointing at CONTRIBUTING.md. Also document what a fork contributor should expect, and add the `rosetta-audit` cross-language gate (#2557) to the baselines section -- it shipped without a CONTRIBUTING entry, so the cross-repo re-baseline protocol it depends on was undiscoverable from here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Found via #2628, our first fork PR in a long while. It arrived with four red audits — both
crucible-auditmodes,tree-sitter-accuracy-audit,tri-comparison-audit— all dyingidentically at
fatal: Remote branch not found in upstream origin, before any engine code ran.Root cause: GitHub withholds repository variables (like secrets) from
pull_requestruns raisedfrom a fork, so
vars.LANGUAGE_CRUCIBLE_REFexpands to an empty string and the workflows rungit clone --branch "". The variable itself is fine (v1.2.0); it is simply not visible to therun.
rosetta-auditdodged this only because it happens to spell the lookupvars.KEYWORD_ROSETTA_REF || 'main'.Changes
::error::namingthe fork limitation and pointing at CONTRIBUTING.md, instead of a bare git error.
rosetta-auditas a fourth entry under CI Pipeline & Baselines.Why not
|| 'main'That is what
rosetta-auditdoes, and it would turn these four checks green on fork PRs — butthese pins exist to make the audits deterministic. Falling back to the corpus's
mainwould auditagainst a snapshot nobody chose and hand back an authoritative-looking wrong answer, in either
direction. A legible failure is better than a silent substitution. Making these audits actually
run for fork contributors is a separate policy question (label-gated re-run,
workflow_run,or a committed default pin) worth deciding on its own merits.
Note on
rosetta-auditIts
|| 'main'fallback has the same determinism gap — on a fork PR it silently audits againstcorpus
mainrather thanKEYWORD_ROSETTA_REF. It produced a correct result on #2628, so thisPR leaves it alone, but it is worth a follow-up.
Core Engine Modification Checklist
Not applicable — no engine code changes. CI configuration and contributor documentation only.
Cross-repo
None required.
Refs #2628, #2557