fix(swe): ONE env root — a second cache made the benchmark unable to report its own coverage - #2334
Merged
Merged
Conversation
…report its own coverage
Two SWE env roots existed and neither named the other:
~/.continuum/benchmarks/swe/envs swe_cache_dir()/envs — LIVE. 46 envs, 8 repos.
~/.continuum/cache/swe-envs legacy grade_local.py default — 14 envs, 3 repos,
untouched since Aug 4.
Nothing failed. Each root was internally consistent, so whichever one you looked at
answered confidently and never mentioned the other.
WHAT IT COST, 2026-08-17: `ls` on the retired root showed sympy/flask/requests only.
That became the reported finding — "77% of staged instances have no environment; the
env builder only works for three repos" — presented as the benchmark's root cause, with
a design approved on top of it. The live root held 46 envs across all 8 repo families:
95% coverage, astropy and django and pytest and pylint and sphinx all present. Same
question, two directories, opposite answers, and the wrong one drove a decision.
THE GENERAL DEFECT, which is why this gets a guard and not just a delete: a cache with
two roots cannot report its own coverage. Every reader picks one and receives a
self-consistent lie. That is not a stale-data problem — it is a missing constraint
([[the-same-bug-at-two-sites-is-a-missing-constraint-not-two-bugs]]).
THREE PARTS:
1. Retired root DELETED (154MB, 14 venvs). Nothing read it but the quarantined script.
2. `grade_local.py --env-root` now defaults to the canonical path, so even if the legacy
script is run it writes where the core reads. A second cache cannot re-form by
accident.
3. GUARD: `the_swe_env_root_has_exactly_one_spelling` scans crate source (comments
stripped) and fails if either root appears as a path literal anywhere. Needles are
ASSEMBLED at runtime — a literal needle matched its own declaration and the guard
failed on itself on first run, which is also its positive control: it demonstrably
scans rather than passing vacuously.
`swe_cache_dir()`'s doc now carries the incident and the rule: derive the envs dir from
the function, never spell the path. Corollary written down for the next person measuring
coverage — read the root from the code, never from a directory you found by name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two SWE env roots existed and neither named the other:
~/.continuum/benchmarks/swe/envs(swe_cache_dir()/envs)~/.continuum/cache/swe-envs(legacygrade_local.pydefault)Nothing failed. Each root was internally consistent, so whichever one you looked at answered confidently and never mentioned the other.
What it cost (2026-08-17): reading the retired root produced "77% of staged instances have no environment; the env builder only works for sympy/flask/requests" — reported as the benchmark's root cause, with a design approved on top of it. Truth was 95% coverage, all 8 families. Same question, two directories, opposite answers, and the wrong one drove a decision.
The general defect — why this gets a guard, not just a delete: a cache with two roots cannot report its own coverage. Every reader picks one and receives a self-consistent lie. That is a missing constraint, not stale data.
Three parts
grade_local.py --env-rootrepointed to the canonical path — if the legacy script runs, it writes where the core reads. A second cache can't re-form by accident.the_swe_env_root_has_exactly_one_spellingscans crate source (comments stripped) and fails if either root appears as a path literal. Needles are assembled at runtime — a literal needle matched its own declaration and the guard failed on itself first run, which doubles as its positive control.swe_cache_dir()'s doc carries the incident and the rule: derive the envs dir from the function, never spell the path.14 swe_bench tests green.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo