Repository navigation
G23-G25: the capability measurement chose, and a second lineage to ch… - #6
Merged
Merged
Conversation
…eck it
Points the shipped optimizer at ten repositories it did not author, builds the
capability that measurement said to build, and gives the correctness gate an
independent checker. Every speedup this repo has published was measured on data
that decayed during the timing loop; that is fixed, and the numbers are
re-baselined conservatively.
Gates closed:
G23 predicated execution (ADR-0015). `if (P) lhs = rhs;` is NORMALISED
rather than refused -- the predicate is lifted onto the statement and
the body becomes straight-line, so every downstream analysis applies
unchanged. The chooser gains a fifth family emitting the select form.
`body.control_flow` is RETIRED as a genus and replaced by its species:
a genus-level refusal stops being univocal the moment one species is
admitted.
G24 gate 3 measures a PROFILE, and says which one (ADR-0016). Fresh input
per measurement with the reset outside the timed region; two input
regimes with the worst reported; and a fourth answer, UNDECIDED, when
the regimes disagree and a probe confirms they genuinely differ. Every
family declares what its profitability depends on. Profitability is
reported as PASS / FAIL / XFAIL-HOLDS / XPASS-ALARM and stays ADVISORY.
G25 a tool's claim is not evidence (ADR-0017). Candidates emit preservation
witnesses; `tools/witness_check.py` re-decides them with its own driver,
its own inputs and its own comparison, importing nothing from the
chooser. Scored with SV-COMP's asymmetry: EXACT +2, CLOSE +1,
REFUTED -32. The foreign-witness lane GOVERNANCE.md has required since
G13.
Chosen by measurement, not by taste -- and then calibrated:
The capability model priced `body.control_flow` at +599 loops across a
held-out corpus of 10,777 loops from ten repositories. That is why G23 was
built rather than something else.
It delivered +14. Prices are quoted for a GENUS; capabilities are built for
a SPECIES; the gap here is 43x. The arithmetic was never wrong -- +599 is the
correct answer to a question nobody can act on. Most of that genus turned out
to be `body.nested_loop` (+2888), which is not conditionals at all. First
PROSPECTIVE test of the pricing instrument: its arithmetic survived, its
interpretation did not.
What leaving the home benchmark found, kept deliberately:
Five defects in the shipped lifter and chooser, each hiding behind a
coincidence uniform across the ten probe loops -- they all step by 1, all
ascend, all write their dead store last, all name arrays with plain
identifiers, and all stay inside the differential harness's buffers
(conformance/lift/repo-pins/, four pins with live witnesses).
The worst was not a transformation bug: the differential harness INDEXED PAST
ITS OWN BUFFERS, so wrong code from opus src/analysis.c:915 was scored EXACT
over undefined behaviour. Gate 2 is the safety property of the whole track
and it was quietly weaker than advertised.
Two more found while building G23, both refused rather than detected: a
POINTER-VALIDITY guard (`if (da) da[i] += x;` -- the select subscripts `da`
on both arms) and trapping guarded expressions. The correctness gate CANNOT
catch the first: the harness only ever passes valid pointers, so the input
that exposes it is not in the test distribution.
And one found by the new validator on its first run: dead-store elimination
replayed its removed store at `int i = n - 1`, writing index -1 at n == 0.
Invisible to a harness that measures only at n = 32000. Four witnesses
refuted, score -86; after the fix, 25 witnesses, 0 refuted, score +50.
Claims corrected rather than quietly restated:
- `p004` was published at ACCEPT 1.56x and is a 5.9x REGRESSION under fresh
inputs, confirmed independently outside the rig. Gate 3 inverted a
decision; it did not merely mis-estimate one.
- The repo-scale corpus was undercounted at 5,147 loops / 41% affine because
loops refused before recording accesses were dropped. Corrected: 10,777
loops / 19.6% affine. The zero-yield result is unaffected.
- `-mcpu=native` was hardcoded in the gated tools, making every measurement
ARM-only; `demo/run.sh` has probed for the right flag since G17 and the
tools now do too.
The gate applied to itself:
With every real witness CONFIRMED, a validator that printed CONFIRMED
unconditionally would have scored identically. conformance/lift/
witness-mutants/ carries five claims that are false on purpose, each wrong in
a way only ONE detector catches (value comparison, trip count, canary,
exactness, regime diversity). An XPASS-ALARM fails the gate: a mutant that
starts passing means that detector died, and the filename says which.
Verified discriminating -- restricting the validator to a single trip count
flips m002 and moves nothing else.
CI was a false witness, and is not any more:
`gates.yml` ran six tools while `just test` ran thirteen, and none of the
optimizer track was among them. The badge went green without executing a
single line of G17-G25. CI now runs the full gatekeeper, with the justfile as
the single source of truth so there is no third place to drift.
Zero candidates remain across all 10,777 loops. Two vorbis loops convert
bit-exactly (1.53x, 1.57x) -- EHMER_MAX is 56 and both are called once per
stream, so the number does not transfer, and ADR-0016 now records in writing
that no rig number licenses an upstream patch until an in-situ gate passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…eck it
Points the shipped optimizer at ten repositories it did not author, builds the capability that measurement said to build, and gives the correctness gate an independent checker. Every speedup this repo has published was measured on data that decayed during the timing loop; that is fixed, and the numbers are re-baselined conservatively.
Gates closed:
G23 predicated execution (ADR-0015).
if (P) lhs = rhs;is NORMALISEDrather than refused -- the predicate is lifted onto the statement and
the body becomes straight-line, so every downstream analysis applies
unchanged. The chooser gains a fifth family emitting the select form.
body.control_flowis RETIRED as a genus and replaced by its species:a genus-level refusal stops being univocal the moment one species is
admitted.
G24 gate 3 measures a PROFILE, and says which one (ADR-0016). Fresh input
per measurement with the reset outside the timed region; two input
regimes with the worst reported; and a fourth answer, UNDECIDED, when
the regimes disagree and a probe confirms they genuinely differ. Every
family declares what its profitability depends on. Profitability is
reported as PASS / FAIL / XFAIL-HOLDS / XPASS-ALARM and stays ADVISORY.
G25 a tool's claim is not evidence (ADR-0017). Candidates emit preservation
witnesses;
tools/witness_check.pyre-decides them with its own driver,its own inputs and its own comparison, importing nothing from the
chooser. Scored with SV-COMP's asymmetry: EXACT +2, CLOSE +1,
REFUTED -32. The foreign-witness lane GOVERNANCE.md has required since
G13.
Chosen by measurement, not by taste -- and then calibrated:
The capability model priced
body.control_flowat +599 loops across aheld-out corpus of 10,777 loops from ten repositories. That is why G23 was
built rather than something else.
It delivered +14. Prices are quoted for a GENUS; capabilities are built for
a SPECIES; the gap here is 43x. The arithmetic was never wrong -- +599 is the
correct answer to a question nobody can act on. Most of that genus turned out
to be
body.nested_loop(+2888), which is not conditionals at all. FirstPROSPECTIVE test of the pricing instrument: its arithmetic survived, its
interpretation did not.
What leaving the home benchmark found, kept deliberately:
Five defects in the shipped lifter and chooser, each hiding behind a
coincidence uniform across the ten probe loops -- they all step by 1, all
ascend, all write their dead store last, all name arrays with plain
identifiers, and all stay inside the differential harness's buffers
(conformance/lift/repo-pins/, four pins with live witnesses).
The worst was not a transformation bug: the differential harness INDEXED PAST
ITS OWN BUFFERS, so wrong code from opus src/analysis.c:915 was scored EXACT
over undefined behaviour. Gate 2 is the safety property of the whole track
and it was quietly weaker than advertised.
Two more found while building G23, both refused rather than detected: a
POINTER-VALIDITY guard (
if (da) da[i] += x;-- the select subscriptsdaon both arms) and trapping guarded expressions. The correctness gate CANNOT
catch the first: the harness only ever passes valid pointers, so the input
that exposes it is not in the test distribution.
And one found by the new validator on its first run: dead-store elimination
replayed its removed store at
int i = n - 1, writing index -1 at n == 0.Invisible to a harness that measures only at n = 32000. Four witnesses
refuted, score -86; after the fix, 25 witnesses, 0 refuted, score +50.
Claims corrected rather than quietly restated:
p004was published at ACCEPT 1.56x and is a 5.9x REGRESSION under fresh inputs, confirmed independently outside the rig. Gate 3 inverted a decision; it did not merely mis-estimate one.-mcpu=nativewas hardcoded in the gated tools, making every measurement ARM-only;demo/run.shhas probed for the right flag since G17 and the tools now do too.The gate applied to itself:
With every real witness CONFIRMED, a validator that printed CONFIRMED
unconditionally would have scored identically. conformance/lift/
witness-mutants/ carries five claims that are false on purpose, each wrong in
a way only ONE detector catches (value comparison, trip count, canary,
exactness, regime diversity). An XPASS-ALARM fails the gate: a mutant that
starts passing means that detector died, and the filename says which.
Verified discriminating -- restricting the validator to a single trip count
flips m002 and moves nothing else.
CI was a false witness, and is not any more:
gates.ymlran six tools whilejust testran thirteen, and none of theoptimizer track was among them. The badge went green without executing a
single line of G17-G25. CI now runs the full gatekeeper, with the justfile as
the single source of truth so there is no third place to drift.
Zero candidates remain across all 10,777 loops. Two vorbis loops convert bit-exactly (1.53x, 1.57x) -- EHMER_MAX is 56 and both are called once per stream, so the number does not transfer, and ADR-0016 now records in writing that no rig number licenses an upstream patch until an in-situ gate passes.