Skip to content

G23-G25: the capability measurement chose, and a second lineage to ch… - #6

Merged
cemphlvn merged 1 commit into
mainfrom
optimizer-track
Aug 26, 2026
Merged

cemphlvn merged 1 commit into
mainfrom
optimizer-track

Conversation

@cemphlvn

Copy link
Copy Markdown
Owner

…eck it

Points the shipped optimizer at ten repositories it did not author, builds the capability that measurement said to build, and gives the correctness gate an independent checker. Every speedup this repo has published was measured on data that decayed during the timing loop; that is fixed, and the numbers are re-baselined conservatively.

Gates closed:
G23 predicated execution (ADR-0015). if (P) lhs = rhs; is NORMALISED
rather than refused -- the predicate is lifted onto the statement and
the body becomes straight-line, so every downstream analysis applies
unchanged. The chooser gains a fifth family emitting the select form.
body.control_flow is RETIRED as a genus and replaced by its species:
a genus-level refusal stops being univocal the moment one species is
admitted.
G24 gate 3 measures a PROFILE, and says which one (ADR-0016). Fresh input
per measurement with the reset outside the timed region; two input
regimes with the worst reported; and a fourth answer, UNDECIDED, when
the regimes disagree and a probe confirms they genuinely differ. Every
family declares what its profitability depends on. Profitability is
reported as PASS / FAIL / XFAIL-HOLDS / XPASS-ALARM and stays ADVISORY.
G25 a tool's claim is not evidence (ADR-0017). Candidates emit preservation
witnesses; tools/witness_check.py re-decides them with its own driver,
its own inputs and its own comparison, importing nothing from the
chooser. Scored with SV-COMP's asymmetry: EXACT +2, CLOSE +1,
REFUTED -32. The foreign-witness lane GOVERNANCE.md has required since
G13.

Chosen by measurement, not by taste -- and then calibrated:
The capability model priced body.control_flow at +599 loops across a
held-out corpus of 10,777 loops from ten repositories. That is why G23 was
built rather than something else.

It delivered +14. Prices are quoted for a GENUS; capabilities are built for
a SPECIES; the gap here is 43x. The arithmetic was never wrong -- +599 is the
correct answer to a question nobody can act on. Most of that genus turned out
to be body.nested_loop (+2888), which is not conditionals at all. First
PROSPECTIVE test of the pricing instrument: its arithmetic survived, its
interpretation did not.

What leaving the home benchmark found, kept deliberately:
Five defects in the shipped lifter and chooser, each hiding behind a
coincidence uniform across the ten probe loops -- they all step by 1, all
ascend, all write their dead store last, all name arrays with plain
identifiers, and all stay inside the differential harness's buffers
(conformance/lift/repo-pins/, four pins with live witnesses).

The worst was not a transformation bug: the differential harness INDEXED PAST
ITS OWN BUFFERS, so wrong code from opus src/analysis.c:915 was scored EXACT
over undefined behaviour. Gate 2 is the safety property of the whole track
and it was quietly weaker than advertised.

Two more found while building G23, both refused rather than detected: a
POINTER-VALIDITY guard (if (da) da[i] += x; -- the select subscripts da
on both arms) and trapping guarded expressions. The correctness gate CANNOT
catch the first: the harness only ever passes valid pointers, so the input
that exposes it is not in the test distribution.

And one found by the new validator on its first run: dead-store elimination
replayed its removed store at int i = n - 1, writing index -1 at n == 0.
Invisible to a harness that measures only at n = 32000. Four witnesses
refuted, score -86; after the fix, 25 witnesses, 0 refuted, score +50.

Claims corrected rather than quietly restated:

  • p004 was published at ACCEPT 1.56x and is a 5.9x REGRESSION under fresh inputs, confirmed independently outside the rig. Gate 3 inverted a decision; it did not merely mis-estimate one.
  • The repo-scale corpus was undercounted at 5,147 loops / 41% affine because loops refused before recording accesses were dropped. Corrected: 10,777 loops / 19.6% affine. The zero-yield result is unaffected.
  • -mcpu=native was hardcoded in the gated tools, making every measurement ARM-only; demo/run.sh has probed for the right flag since G17 and the tools now do too.

The gate applied to itself:
With every real witness CONFIRMED, a validator that printed CONFIRMED
unconditionally would have scored identically. conformance/lift/
witness-mutants/ carries five claims that are false on purpose, each wrong in
a way only ONE detector catches (value comparison, trip count, canary,
exactness, regime diversity). An XPASS-ALARM fails the gate: a mutant that
starts passing means that detector died, and the filename says which.
Verified discriminating -- restricting the validator to a single trip count
flips m002 and moves nothing else.

CI was a false witness, and is not any more:
gates.yml ran six tools while just test ran thirteen, and none of the
optimizer track was among them. The badge went green without executing a
single line of G17-G25. CI now runs the full gatekeeper, with the justfile as
the single source of truth so there is no third place to drift.

Zero candidates remain across all 10,777 loops. Two vorbis loops convert bit-exactly (1.53x, 1.57x) -- EHMER_MAX is 56 and both are called once per stream, so the number does not transfer, and ADR-0016 now records in writing that no rig number licenses an upstream patch until an in-situ gate passes.

…eck it

Points the shipped optimizer at ten repositories it did not author, builds the
capability that measurement said to build, and gives the correctness gate an
independent checker. Every speedup this repo has published was measured on data
that decayed during the timing loop; that is fixed, and the numbers are
re-baselined conservatively.

Gates closed:
  G23  predicated execution (ADR-0015). `if (P) lhs = rhs;` is NORMALISED
       rather than refused -- the predicate is lifted onto the statement and
       the body becomes straight-line, so every downstream analysis applies
       unchanged. The chooser gains a fifth family emitting the select form.
       `body.control_flow` is RETIRED as a genus and replaced by its species:
       a genus-level refusal stops being univocal the moment one species is
       admitted.
  G24  gate 3 measures a PROFILE, and says which one (ADR-0016). Fresh input
       per measurement with the reset outside the timed region; two input
       regimes with the worst reported; and a fourth answer, UNDECIDED, when
       the regimes disagree and a probe confirms they genuinely differ. Every
       family declares what its profitability depends on. Profitability is
       reported as PASS / FAIL / XFAIL-HOLDS / XPASS-ALARM and stays ADVISORY.
  G25  a tool's claim is not evidence (ADR-0017). Candidates emit preservation
       witnesses; `tools/witness_check.py` re-decides them with its own driver,
       its own inputs and its own comparison, importing nothing from the
       chooser. Scored with SV-COMP's asymmetry: EXACT +2, CLOSE +1,
       REFUTED -32. The foreign-witness lane GOVERNANCE.md has required since
       G13.

Chosen by measurement, not by taste -- and then calibrated:
  The capability model priced `body.control_flow` at +599 loops across a
  held-out corpus of 10,777 loops from ten repositories. That is why G23 was
  built rather than something else.

  It delivered +14. Prices are quoted for a GENUS; capabilities are built for
  a SPECIES; the gap here is 43x. The arithmetic was never wrong -- +599 is the
  correct answer to a question nobody can act on. Most of that genus turned out
  to be `body.nested_loop` (+2888), which is not conditionals at all. First
  PROSPECTIVE test of the pricing instrument: its arithmetic survived, its
  interpretation did not.

What leaving the home benchmark found, kept deliberately:
  Five defects in the shipped lifter and chooser, each hiding behind a
  coincidence uniform across the ten probe loops -- they all step by 1, all
  ascend, all write their dead store last, all name arrays with plain
  identifiers, and all stay inside the differential harness's buffers
  (conformance/lift/repo-pins/, four pins with live witnesses).

  The worst was not a transformation bug: the differential harness INDEXED PAST
  ITS OWN BUFFERS, so wrong code from opus src/analysis.c:915 was scored EXACT
  over undefined behaviour. Gate 2 is the safety property of the whole track
  and it was quietly weaker than advertised.

  Two more found while building G23, both refused rather than detected: a
  POINTER-VALIDITY guard (`if (da) da[i] += x;` -- the select subscripts `da`
  on both arms) and trapping guarded expressions. The correctness gate CANNOT
  catch the first: the harness only ever passes valid pointers, so the input
  that exposes it is not in the test distribution.

  And one found by the new validator on its first run: dead-store elimination
  replayed its removed store at `int i = n - 1`, writing index -1 at n == 0.
  Invisible to a harness that measures only at n = 32000. Four witnesses
  refuted, score -86; after the fix, 25 witnesses, 0 refuted, score +50.

Claims corrected rather than quietly restated:
  - `p004` was published at ACCEPT 1.56x and is a 5.9x REGRESSION under fresh
    inputs, confirmed independently outside the rig. Gate 3 inverted a
    decision; it did not merely mis-estimate one.
  - The repo-scale corpus was undercounted at 5,147 loops / 41% affine because
    loops refused before recording accesses were dropped. Corrected: 10,777
    loops / 19.6% affine. The zero-yield result is unaffected.
  - `-mcpu=native` was hardcoded in the gated tools, making every measurement
    ARM-only; `demo/run.sh` has probed for the right flag since G17 and the
    tools now do too.

The gate applied to itself:
  With every real witness CONFIRMED, a validator that printed CONFIRMED
  unconditionally would have scored identically. conformance/lift/
  witness-mutants/ carries five claims that are false on purpose, each wrong in
  a way only ONE detector catches (value comparison, trip count, canary,
  exactness, regime diversity). An XPASS-ALARM fails the gate: a mutant that
  starts passing means that detector died, and the filename says which.
  Verified discriminating -- restricting the validator to a single trip count
  flips m002 and moves nothing else.

CI was a false witness, and is not any more:
  `gates.yml` ran six tools while `just test` ran thirteen, and none of the
  optimizer track was among them. The badge went green without executing a
  single line of G17-G25. CI now runs the full gatekeeper, with the justfile as
  the single source of truth so there is no third place to drift.

Zero candidates remain across all 10,777 loops. Two vorbis loops convert
bit-exactly (1.53x, 1.57x) -- EHMER_MAX is 56 and both are called once per
stream, so the number does not transfer, and ADR-0016 now records in writing
that no rig number licenses an upstream patch until an in-situ gate passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cemphlvn
cemphlvn merged commit bcf7129 into main Aug 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant