Script-size optimizer, all 7 tiers: p256-wallet 958,792 → 147,113 B (−84.7%), corpus −65.1% - #160
Open
icellan wants to merge 16 commits into
Open
Script-size optimizer, all 7 tiers: p256-wallet 958,792 → 147,113 B (−84.7%), corpus −65.1%#160icellan wants to merge 16 commits into
icellan wants to merge 16 commits into
Conversation
…ntation Nothing in the compiler could compare two candidate lowerings by the metric that actually matters — serialized locking-script bytes. `OP_DUP` and a 33-byte constant push are one instruction each and 34x apart in cost, so an instruction count cannot rank them. - `metrics/cost-model.ts`: `sizeOfStackOp` / `estimateScriptBytes`, routing every push through the same `push-encoding.ts` encoders the emit pass uses. Asserted byte-exact against `emitMethod` for every method of every fixture, before and after peephole — the model is a checked mirror of `06-emit.ts`, not a second opinion about encoding. An unknown opcode throws rather than costing zero. - `metrics/script-metrics.ts`: buckets a serialized script by what each byte is spent on. One rule worth stating: a push immediately consumed by OP_PICK / OP_ROLL is charged to stack access, not to constants, so `bringToTop`'s depth operands do not get blamed on the wrong optimizer. Worth 21,926 bytes of reclassification on p256-wallet alone. - `golden-invariance.test.ts`: every fixture that ships a `.runar.ts` reproduces its checked-in `expected-script.hex`. This is the TS-tier-only version of what the conformance runner checks across seven tiers, so a backend experiment can be shown byte-neutral in seconds without building six native toolchains. Read-only: no pass consults any of this and no emitted byte moves.
…uler
Both are opt-in and inert by default, so the 72 goldens,
conformance/script-size-baseline.json and cross-tier hex parity are untouched.
All 72 fixtures still reproduce their expected-script.hex byte-for-byte, and the
Go and Rust cross-compiler golden tests still pass.
--ec-constant-pool
`fieldMod` pushes the curve's field prime inline at every modular reduction:
20,025 pushes of a 34-byte literal in p256-wallet, 680,850 of its 958,792
bytes. ECTracker gains a pooled slot per constant, and `pushConst` compares the
emitted cost of picking that slot against re-pushing the literal and takes the
cheaper — so no individual call site can grow. Parameterized by CurveParams /
GroupParams, so secp256k1, P-256 and P-384 share one code path.
p256-wallet 958,792 -> 304,463 (-68.2%)
p384-wallet 1,963,300 -> 463,435 (-76.4%)
corpus 13,526,563 -> 6,285,154 (-53.5%), 9 fixtures, none grown
Proved equivalent on the real @bsv/sdk engine against OpenSSL signatures on
both curves, plus every SEC1 rejection case (r=0, s=0, r=n, s=n, all-zero
signature, wrong message, wrong parity, truncated and oversized inputs).
--stack-scheduler=liveness
Parks a result on the alt stack when the next binding does not consume it, so
the operands an ANF chain reads repeatedly stay at depth 0/1; the whole spill
group is restored in production order, which is the order an accumulation
reads it. Plus commutative operand ordering.
Ordering is scored by running the candidate op sequences through the real
peephole rather than a byte formula: two consumed operands at depths 1 and 0
emit OP_SWAP OP_SWAP, which `swap-swap` deletes outright — free — while the
cheaper-looking reversed order emits one real OP_SWAP and costs a byte. That
correction took `arithmetic` from 24 bytes to 18.
arithmetic 28 -> 18 B (-35.7%), the hand-derived optimum
bounded-loop 42 -> 37 B (-11.9%)
~30 mid-size fixtures -0.1%; 34 changed, none grown
Selection is per method: both schedules are lowered and the cheaper one,
measured after peephole, is kept — so "the scheduler never grows a method" is
structural rather than a property of the greedy heuristic.
Spilling is refused in any scope that still has control flow ahead of it.
Restoring immediately before an `if` leaves the parent stack in a shape
lowerIf's arm reconciliation, declared-result trim and Layer B/C depth
invariants were not written for, and it MISCOMPILED if-without-else-multi-temp
into accepting a witness the shipping compiler rejects. Byte counts, the other
goldens and 4,099 compiler unit tests all passed while that was true; the
conformance/witnesses corpus replayed through runDifferentialExecution caught
it, 2 of 86 cases.
Also:
- `conformance/runner/script-metrics.ts` — "where did the bytes go?", the
companion to script-size-check.ts's "did anything grow?". Compares named
compiler variants and never silently drops a fixture from a size report.
- `runDifferentialExecution` accepts both flags, so any experiment can be run
through the same source-vs-script oracle.
- `--stack-scheduler` rejects an unknown mode instead of falling back to the
default: a benchmark that quietly measured the shipping compiler while
reporting an experiment is worse than a crash.
Three reports under docs/experiments/, all reproducible with
`pnpm --filter runar-conformance run script-metrics`.
script-size-optimization-baseline.md
58% of every byte the compiler has ever emitted is a constant push, and 56% of
the whole 13.5 MB corpus is nine numbers — each curve's field prime — pushed
over and over. p256-wallet is 72.7% constant pushes, 71.0% of the fixture in a
single 33-byte literal repeated 20,025 times. Also records that brief Phase 3
(fix-point peephole) and Phase 15 (OP_PUSH_TX binding) already ship.
stack-scheduler-design.md
The current lowering algorithm with line references, the measured
inefficiencies, the byte-cost function, the correctness invariants, and the
benchmark plan — written before the prototype, then updated with what actually
happened, including the miscompile the witness corpus caught and why a passing
bisect can be vacuous.
script-size-optimizer-results.md
What is generic, what is not, and where the remaining bytes are. After
pooling, p256-wallet's 304,463 bytes are 70.6% stack traffic and 27.2%
arithmetic — all of it the modular-reduction sequence itself, ~20,000 times.
Reaching ~30 kB needs the algebra, not more scheduling: modular-domain
analysis first, then Straus/Shamir, then a fixed-base comb.
Two plan assumptions died on measurement and are recorded as dead rather than
quietly dropped:
- Eager dead-slot retirement. 657 of 387,749 pick/roll sites in the corpus are
deeper than 16; typical depths are 2-5, and depths 0-2 are single-byte
opcodes. Every drop would cost bytes to save nothing.
- That a generic scheduler could reach P-256 at all. The crypto emitters build
their own stack layout through ECTracker and never pass through
05-stack-lower.ts — 13.4 MB of the 13.5 MB corpus is out of its reach.
…rojecting it
§3.6 carried an estimate for the next step. It is now a measurement: `fieldMod`
/ `cFieldMod` were patched behind a throwaway switch to emit the short form, the
corpus was re-measured, and the patch was discarded.
p256-wallet 958,792 -> 304,463 (pool) -> 179,796 (+ sinking, -81.2%)
p384-wallet 1,963,300 -> 463,435 -> 272,584 (-86.1%)
ec-primitives 1,332,782 -> 433,880 -> 258,160 (-80.6%)
The sound variant captures 89% of the theoretical floor, so the analysis does
not need to be clever about subtraction. The two optimizations are also not
independent: without pooling the cheap `fieldSub` form pushes the prime twice
and p256-wallet gets LARGER (958,792 -> 999,371). Sinking only pays once the
prime is a 2-byte pick.
The more useful result is what the analysis actually has to prove (new §3.8).
The short-reduction variant passes 256 EC oracle assertions — OpenSSL signatures
on both curves, ec-on-curve-canonicity, ec-degenerate-add, ec-mul-scalars,
p256-p384-scalars, p256-p384-ecdsa-verify. It would have shipped looking green.
It is still unsound:
- multiply / add / mulconst need only `dividend >= 0`, which unsigned
coordinate decoding already gives — ~70% of reductions, trivial analysis;
- subtract needs `subtrahend < p`, which decoding 32 unsigned bytes does NOT
imply.
ecAdd((0, 1), (2^256 - 1, 1))
shipping : ...fffffffdfffff85f
sinking : ...0001000003d0 0x1000003d0 = 2^32 + 977 = 2^256 - p
Reachable only through the unguarded bare builtins; verifyECDSA_* and onCurve
run a canonicity guard first. So Phase 4/5 is a sign lattice plus a `< p` bit
that only subtrahends carry — materially smaller than a full modular-domain
lattice, and the difference between passing 256 oracle assertions and being
correct.
Also reorders §6. At 179,796 bytes the split is 69.6% stack-shuffle / 26.7%
arithmetic and OP_PICK (x36,683) is the largest opcode: once a reduction costs
3 bytes, ECTracker's own operand shuffling is the bottleneck, so the typed
field-element IR moves ahead of Straus/comb.
No code changes; the experiment branch was deleted.
…ion-sinking
`fieldMod` costs 10 bytes and runs ~20,000 times in a P-256 verify. Six of them
are a sign fix-up that exists only because OP_MOD takes the sign of the
dividend. Where the dividend is provably non-negative they are dead.
p256-wallet 958,792 -> 304,463 (pool) -> 179,890 (-81.2%)
p384-wallet 1,963,300 -> 463,435 -> 272,678 (-86.1%)
ec-primitives 1,332,782 -> 433,880 -> 258,303 (-80.6%)
ec-unit 479,716 -> 157,129 -> 93,678 (-80.5%)
That is within 94 bytes of the measured ceiling for this transformation
(179,796 on p256-wallet, docs/experiments §3.7), so the lattice recovers
essentially all of the available win. Opt-in and inert by default: all 72
goldens still reproduce byte-for-byte and script-size-check is 72/72 ok.
WHY A LATTICE AND NOT A REWRITE
The two paths need different facts, and conflating them is not hypothetical —
the ceiling measurement did exactly that, passed 256 EC oracle assertions, and
was still wrong:
- multiply / add / mulconst need `dividend >= 0`. Unsigned coordinate
decoding already gives it, so ~70% of reductions qualify immediately.
- subtract's cheap `a - b + p` form needs the strictly stronger
`subtrahend < p`, which OP_BIN2NUM of 32 unsigned bytes does NOT imply: a
coordinate may exceed p by up to 2^32 + 977.
ecAdd((0, 1), (2^256 - 1, 1))
correct : ...fffffffdfffff85f
blanket : ...0001000003d0 0x1000003d0 = 2^32 + 977 = 2^256 - p
IMPLEMENTATION
`Dom` is a three-point lattice (Unknown < NonNegative < Reduced) carried as
`ECTracker.dm`, a SLOT-parallel array to `nm` rather than a name-keyed map:
names are reused (`_fmul_prod` is written by every multiply) and the same name
can be resident twice, so a map would go stale in exactly the cases that matter.
Every `nm` mutation mirrors into `dm` with the same splice, external mutation
now goes through pushTracked/popTracked/removeSlotAt, and `domainOf` throws if
the arrays ever differ in length — a silent desync would hand a transfer
function a fact about the wrong slot, which is the one failure mode that yields
a smaller script that quietly computes something else.
Transfer functions: add/mul are non-negative iff both operands are; a square is
non-negative unconditionally; mulconst keeps the operand's sign for positive c;
every reduction result is Reduced; a decoded coordinate is NonNegative but never
Reduced. Anything a rawBlock or an OP_IF produces stays Unknown, so an
un-analysed value can only fall back to the shipping reduction. Group-order
reductions are deliberately left at NonNegative rather than Reduced, so a value
reduced mod n can never be mistaken for one reduced mod p.
The cheap subtraction references the prime twice, so whether to use it is a cost
comparison (`cheapSubPays`) against the pooled push cost, not a flag: without
`--ec-constant-pool` it would make p256-wallet larger.
TESTING
`ec-reduction-sinking.test.ts` is a differential sweep, not a signature check —
that question was already answered wrongly once. It runs every emitter over the
coordinate values on the boundary (0, 1, 2, p-1, p, p+1, 2^256-1, G.x) including
the full 8x8 cross product for ecAdd and p256Add, and requires the sunk script's
RESULT to match the shipping one on every combination, including ones no valid
curve point could produce. The counterexample above is pinned by name. An
absolute OpenSSL oracle then re-checks accept plus seven rejection cases.
One doubling and one conditional add per COLUMN instead of per bit, wherever
the base point is a compile-time constant: p256MulGen, p384MulGen, and the
u1*G half of ECDSA verification. u2*Q keeps the ladder — Q arrives in the
witness.
p256-wallet 958,792 -> 179,890 (sink) -> 147,113 (-84.7%)
p384-wallet 1,963,300 -> 272,678 -> 223,204 (-88.6%)
emitP256MulGen 90,676 -> 54,117 (-40.3%)
emitP384MulGen 136,599 -> 81,418 (-40.4%)
emitVerifyECDSA_P256 195,120 -> 158,560 (-18.7%)
Opt-in and inert by default: all 72 goldens still reproduce byte-for-byte and
script-size-check is 72/72 ok.
WHY NOT STRAUS
The obvious move is a joint ladder for u1*G + u2*Q. Measured, it does not pay.
The ladder's speed comes from the CHEAP incomplete mixed add, justified in
buildJacobianAddOrDoubleInline by an interval argument over c_i mod n — an
argument that holds because the accumulator is c_i*P and the addend is P, one
generator. A joint ladder makes the accumulator c_i*G + d_i*Q with Q supplied
by the caller; an attacker choosing Q = k*G solves c_i + d_i*k == 1 (mod n) for
k, so the exception becomes reachable at an arbitrary step. Completing the
addition costs a measured +299 B/round (a whole ladder goes 90,610 -> 167,410,
+84.8%), against 690 B/round for two independent ladders — so joint + complete
lands at ~700 B/round. A loss. The comb keeps a single generator, so the
argument survives.
SOUNDNESS
comb.ts re-derives the interval argument for the comb rather than assuming it,
as executable arithmetic:
- combParams searches for the scalar offset m with m*n >= 2^(w*d-1) and
(m+1)*n - 1 < 2^(w*d), so the first digit is never zero and the accumulator
never starts at infinity. For P-256 at w=3 that returns the same +3n the
ladder hardcodes; for P-384 at w=3 it returns +5n. Reusing +3n there would
have left the leading digit free to vanish.
- combSafeRounds proves, per round, that the pre-add accumulator cannot be 0,
+T[j] or -T[j] modulo n for any table entry, over the whole scalar domain.
Rounds it cannot prove get the complete add-or-double form. For P-256 at
w=3 it proves 81 of 86, so the fallback costs ~1.2 kB. `true` is never
assumed.
The window width is not hardcoded: cEmitCombBest renders w = 2, 3 and 4 in full
and keeps whichever estimateScriptBytes scores smallest.
TESTING
ec-comb.test.ts is a differential against the binary ladder on the real @bsv/sdk
engine over the scalars the argument turns on — 0, 1, small values, n-1, n, n+1,
2n, negatives, and the powers of two either side of each block boundary
(2^85, 2^86, 2^171, 2^172, 2^255) — plus a cross-check against the INDEPENDENT
generic-point ladder so a shared bug in the MulGen wrapper cannot hide. Then the
verifier under an OpenSSL oracle: genuine signature accepted, seven near-misses
rejected, comb and ladder agreeing on every one.
comb-table.test.ts pins the compile-time arithmetic against published vectors
(2G, n*G = infinity), checks every table entry is on the curve for both curves
and w in {2,3,4}, and checks the safety analysis is monotone under a widened
domain — a checker that proved every round would be broken, not clever.
NOT IN SCOPE
secp256k1. comb.ts is curve-generic, but the emitter uses the NIST codegen's
a = -3 doubling; ec-codegen.ts needs its own wiring, which is why ec-primitives,
ec-demo, schnorr-zkp, ec-unit and convergence-proof are unchanged here.
… stack
Measured with every flag on:
p256-wallet 958,792 -> 147,113 (-84.7%)
p384-wallet 1,963,300 -> 223,204 (-88.6%)
corpus 13,526,563 -> 4,906,225 (-63.7%), 43/72 changed, none grown
Adds the combined headline, the Straus/Shamir negative result with its
per-round arithmetic, and the two facts the comb's soundness work turned up
(the ladder's +3n offset is not portable to P-384 at w=3; the safety checker
has to be allowed to fail, and does, on 5 of 86 rounds).
Reorders the recommendations: pool, then sinking, then comb — sinking depends
on the pool, and the comb carries the heaviest proof obligation. secp256k1 comb
wiring moves to the top of "do next" since the analysis is already written and
4.5 MB of fixtures are still on the ladder.
Also runs the witness corpus under an `all` variant, so the flag combination a
user would actually turn on is proved together rather than only separately.
This was referenced Aug 29, 2026
`ecMulGen` was the last compile-time-known base still running the 257-round binary ladder. It now uses the same Lim-Lee comb the NIST curves got, cutting it from 84,203 to 52,237 bytes (-38.0%) and the conformance corpus from 4,906,225 to 4,726,567 (-3.7%, and -65.1% against the shipping baseline). `comb.ts` was already curve-generic, but `makeCurve` hardcodes the NIST a = -3. secp256k1 is y^2 = x^3 + 7, so `SECP256K1_COMB_CURVE` is written out rather than built from that template — a wrong `a` there does not produce an obviously broken table, it produces a table of points on a DIFFERENT curve that the other curve's on-curve check accepts. The published 2G vector is pinned for exactly that reason. The emitter is a twin of `cEmitCombMulGen`, not a share: secp256k1's `jacobianDouble` computes D = 3X^2 where the NIST version computes 3(X-Z^2)(X+Z^2). Only the compile-time table and the interval checker are common, and those read `a` from the curve record. `combSafeRounds` proves 81 of 86 rounds safe for the cheap incomplete add at w=3; the rest fall back to the complete add-or-double form. `combParams` independently re-derives the scalar offset (m=3, d=86 for secp256k1) instead of reusing the ladder's hardcoded +3n. `emitEcMul` is deliberately NOT combed: its base arrives at run time, and the interval argument does not cover an attacker-chosen point. Also adds `conformance/ec-flag-parity/`, the cross-tier target for the ports that follow. The size flags default off, so the ordinary conformance suite — which compiles with defaults — cannot see them; seven tiers could each ship a different `--ec-constant-pool` and stay green. The fixture pins the exact script the TS reference emits for all 24 EC emitters under all 4 flag combinations, and is re-derived in-process by its own test so it cannot go stale. Tests: 83 in ec-comb.test.ts (secp256k1 differential against the ladder over the interval argument's boundary scalars, plus absolute published k*G vectors), 14 in comb-table.test.ts, 4 in parity.test.ts. Default output unchanged.
Replays the whole TypeScript optimizer stack into `compilers/go`: the exact byte-cost model, the compile-time comb table and its interval checker, the sign lattice, the constant pool, reduction sinking, and the fixed-base comb for all three curves. Byte-exact against the TS reference for all 24 EC emitters under all 4 flag combinations, plus end-to-end through both CLIs (`ecMulGen` contract: 424,567 -> 50,157 bytes, identical hex from `runar-compiler-go --ec-fixed-base-comb` and the TS `--ec-fixed-base-comb`). New: `codegen/cost_model.go`, `codegen/comb.go`, and the flags `--ec-constant-pool`, `--ec-reduction-sinking`, `--ec-fixed-base-comb` on `runar-compiler-go`, threaded through `CompileOptions` -> `LowerToStackOptions` -> `loweringContext`. Three defects the parity fixture caught, none of which any Go-side test could have found on its own: 1. `cDecomposePoint` did not record that BIN2NUM of an unsigned coordinate is `NonNegative`, so the NIST tier proved fewer domains and emitted a LARGER script under `--ec-reduction-sinking` than the reference (P256Mul 94,137 vs 90,610). Silently correct, silently worse. 2. Six NIST entry points never released the pooled prime, leaving a 2-byte divergence per emitter. 3. The three `+n` pushes in `cEmitMul` were routed through the pool. That is strictly smaller (-96 B per P-256 ladder, -144 B per P-384) but the TS reference pushes them as literals, so the tiers would have diverged. Matched the reference and recorded the missed opportunity in a comment. Every direct `t.nm` mutation in `ec.go` and `p256_p384.go` now goes through `pushTracked`/`popTracked`/`removeSlotAt`, and `domainOf` panics if the name and lattice slices ever desynchronise — a silent desync would hand a transfer function a fact about the WRONG slot, which is the one failure mode that produces a smaller script that quietly computes something else. Default output is unchanged: `TestEcFlagsDefaultOffIsByteIdentical` pins that a nil options pointer reproduces the shipping hash for every emitter, and the existing op-count goldens are untouched. Full Go suite green.
Byte-exact against the TypeScript reference for all 24 EC emitters under all 4 flag combinations (`tests/ec_flag_parity_tests.rs`), and end-to-end through the CLI: `runar-compiler-rust --ec-fixed-base-comb` produces hex identical to the TS and Go compilers for the same contract. New: `codegen/cost_model.rs`, `codegen/comb.rs`, and the flags `--ec-constant-pool`, `--ec-reduction-sinking`, `--ec-fixed-base-comb`, threaded through `CompileOptions` -> `lower_to_stack_with_ec` -> `LoweringContext`. Three things this tier needed that Go did not: 1. The EC constants were pushed as pre-encoded script-number BYTE blobs because they exceed `i128`. `PushValue::Int` carries a `BigInt`, so the blob was never necessary — and it cost real things: a `Bytes` push is invisible to the peephole's constant folding (which is why `k + 3n` had to be hand-folded here and only here) and invisible to the sign lattice. Switched to `Int`, emitted `k + 3n` as the reference's three `+n` steps, and let fold-chain-add collapse them back. Shipped bytes unchanged; the raw op tree now AGREES WITH GO, where it was 4 ops short before. Op-count goldens restamped to the Go values with that explanation. 2. `p256_p384.rs` carried its own hand-copy of `ECTracker` — "duplicated since it's private there". That was tolerable for 200 lines of stack bookkeeping and stopped being tolerable once the tracker carried a sign lattice whose transfer functions decide which reduction shape is emitted. Two independently-maintained copies is two chances to prove `Reduced` where only `NonNegative` holds, and the resulting script is smaller, passes every local test, and is wrong. Widened the `ec.rs` tracker to `pub(crate)` and deleted the copy. 3. `compile_from_source_str_with_options` built its backend options with `..Default::default()`, silently dropping every caller-set field before stack lowering. `--ec-constant-pool` reached the frontend and vanished; the compile succeeded and emitted the unoptimized script. Now `..opts.clone()`. Also fixed here, as in Go: `c_decompose_point` did not record that BIN2NUM of an unsigned coordinate is `NonNegative`, and six entry points never released the pooled prime. Both were caught only by the parity fixture. Default output unchanged: `ec_flags_default_off_is_byte_identical` pins that `None` options reproduce the shipping hash for every emitter, and the conformance hex goldens are untouched. Full Rust suite green.
Byte-exact against the TypeScript reference for all 24 EC emitters under all 4 flag combinations (`tests/test_ec_flag_parity.py`, 48 assertions), and end-to-end through the CLI: `python3 -m runar_compiler --ec-fixed-base-comb` produces hex identical to the TS, Go and Rust compilers for the same contract. New: `codegen/cost_model.py`, `codegen/comb.py`, and the flags `--ec-constant-pool`, `--ec-reduction-sinking`, `--ec-fixed-base-comb`, threaded through `compile_from_source` -> `lower_to_stack` -> `_LoweringContext`. Two defects the parity fixture caught: the second (`u2*Q`) ladder inside `_c_emit_verify_ecdsa` was still called without options, so the verifier came out at 628,923 bytes where the reference emits 319,693 — the flag was applied to exactly half of it; and the verifier never released its two pooled constants, a 4-byte tail. Neither would have failed any Python-side test. Default output unchanged: `test_ec_flags_default_off_is_byte_identical` pins that `None` options reproduce the shipping hash for every emitter. Full Python suite green (1,208 passed, 1 skipped).
Byte-exact against the TypeScript reference for all 24 EC emitters under all 4 flag combinations (`test/codegen/test_ec_flag_parity.rb`), and end-to-end through the CLI: `runar-compiler-ruby --ec-fixed-base-comb` produces hex identical to the TS, Go, Rust and Python compilers for the same contract. New: `codegen/cost_model.rb`, `codegen/comb.rb`, and the flags `--ec-constant-pool`, `--ec-reduction-sinking`, `--ec-fixed-base-comb`, threaded through `compile_from_source` -> `lower_to_stack` -> `LoweringContext`. One tier-specific detail: Ruby's `Integer#pow` rejects a negative exponent, so there is no `x.pow(-1, m)` modular-inverse shortcut as in Python. `comb.rb` carries an explicit extended-Euclid `mod_inverse` instead. The dispatch tables split flag-aware emitters from the rest explicitly (`EC_FLAG_AWARE`, the `EncodeCompressed` check) rather than passing options to everything: an emitter the flags cannot reach taking an ignored argument is a silent no-op today and a latent divergence the day someone gives it a body. Default output unchanged: `test_ec_flags_default_off_is_byte_identical` pins that `nil` options reproduce the shipping hash for every emitter. Full Ruby suite green (67 test files).
Byte-exact against the TypeScript reference for all 24 EC emitters under all 4 flag combinations (`codegen/EcFlagParityTest`), and end-to-end through the CLI: `runar-java --ec-fixed-base-comb` produces hex identical to the TS, Go, Rust, Python and Ruby compilers for the same contract. New: `codegen/CostModel.java`, `codegen/Comb.java`, and the flags `--ec-constant-pool`, `--ec-reduction-sinking`, `--ec-fixed-base-comb`, threaded through `Cli.Args` -> `StackLower.run` -> `LoweringContext`. Every public emitter keeps its old one-argument signature as an overload that delegates with `null` options, so no existing caller or test changed. `Emit.OPCODES` is now public: the cost model has to be able to reject an unknown mnemonic loudly rather than costing it zero, or a codegen typo becomes a size report that is quietly wrong. The parity fixture is parsed by hand rather than by adding a JSON dependency to a module that has none — the shape is a fixed two-level map and the scan is anchored on the emitter key, so `EcMul` cannot match inside `EcMulGen`. The test also asserts the fixture is non-vacuous (the flags really do move the reference's bytes) and that the comb does NOT fire for `ecMul` / `p256Mul` / `p384Mul`, whose bases arrive at run time. Nested `LoweringContext`s for if-arms inherit the options explicitly; without that an EC call inside a branch would silently fall back to the shipping path. Default output unchanged: `ecFlagsDefaultOffIsByteIdentical` pins that `null` options reproduce the shipping hash for every emitter. Full Java suite green (683 tests).
Completes the seven-tier port. New: `passes/helpers/comb.zig`, `passes/helpers/ec_cost_model.zig`, and the flags `--ec-constant-pool`, `--ec-reduction-sinking`, `--ec-fixed-base-comb`, threaded through `CompileOptions` -> `stack_lower.lowerOpts` -> `LowerCtx`. End-to-end, `runar-zig --ec-fixed-base-comb` produces hex identical to the TypeScript, Go, Rust, Python, Ruby and Java compilers for the same contract — all seven tiers now agree byte for byte with every flag on. Two things are genuinely different here, and both are asserted rather than assumed: 1. This tier's `StackOp.roll` / `.pick` carry the depth in the op itself and `emitStackOp` writes the depth push while emitting them. Every other tier's tracker emits a separate depth `push` and charges the roll one byte. So the cost model here charges `scriptNumberCost(depth) + 1`. Same emitted bytes, different spelling; the cost-model equality is what keeps it honest. 2. `emitEcMul` keeps `k + 3n` pre-folded on the DEFAULT path. The reference emits three `+n` steps and lets its peephole reassociate them; this peephole reassociates only i64 `push_int` chains (rule 27) and a 256-bit constant is a `push_data` blob in this IR, so emitting three steps would ship 68 extra bytes rather than collapsing. Under `--ec-constant-pool` the three steps ARE emitted, each served from the pooled slot, which is what makes the pooled variants byte-identical to the reference. Because of (2) the Zig parity test gates on the raw byte COUNT with that single divergence asserted exactly (`allowedDelta`) — if it ever widens, or appears anywhere else, the test fails. The fixture now carries a `postPeephole` measurement alongside the raw one to make the distinction explicit, and the README explains which tiers assert which and why. `ec_cost_model.zig` re-exports the implementation that lives in `ec_emitters.zig` rather than duplicating it: the tracker's constant pool needs the estimator to price a call site before emitting anything, and a second copy would be free to drift from the one the pool actually consults. Also adds a test pinning that the comb's window-width search picks w=3, so a future change to the candidate set fails with a clear message rather than as an opaque byte count. Not ported: `nist_ec_emitters.zig`. Zig's secp256k1 side is complete; its NIST emitters keep their shipping path, and the parity fixture covers what is ported. Docs: `script-size-optimizer-results.md` gains §8 with the corpus figure updated for the secp256k1 comb (13,526,563 -> 4,726,567, -65.1 %), the six defects the parity fixture caught across the tiers, the tier-specific findings, and what remains before any of this could land as default. Default output unchanged: all 758 Zig tests pass, including the conformance goldens.
`gofmt -w codegen/` reformats the whole directory, and running it while porting pulled 43 unrelated files into the Go commit — babybear, bn254, koalabear, sp1_fri, wots, rabin, slh_dsa, blake3, and two frontend files, none of which the EC work touches. Every hunk in them is whitespace: statements split off shared lines, comment columns realigned. That is noise in a diff a reviewer has to read for correctness, and it puts unrelated crypto emitters in the blast radius of a change that has nothing to do with them. Restored to their pre-port bytes; the net diff for those files is now zero. Go builds and the full Go suite passes.
icellan
changed the title
…ters Completes the port: every EC emitter in all seven tiers now honours --ec-constant-pool, --ec-reduction-sinking and --ec-fixed-base-comb. `NistTracker` is deleted and aliased to `ec_emitters.ECTracker`. The old copy predates the sign lattice; keeping it would have meant two independently maintained lattices, which is two chances to prove `Reduced` where only `NonNegative` holds — and the resulting script is smaller, passes every local test, and is wrong. Its one curve-specific field became a parameter; nothing about a tracker is per-curve. The duplicate `beToUnsignedScriptNumAlloc` goes too. Field ops gain sinking, the pool, and the transfer functions. `decomposePoint` records `.non_negative` and NOT `.reduced` on both coordinates — a 0x00 sign byte before BIN2NUM proves >= 0 but not < p. `groupMod` deliberately neither sinks nor marks its result `.reduced`: a mod-n result can exceed p, and marking it reduced would license a later `fieldSub` to take the cheap path unsoundly. The p256 and p384 arms of `buildBuiltinOps` were two verbatim copies; folded into curve-generic helpers so the comb is wired once rather than twice. Byte-exact against the reference for all 14 NIST emitters under all 4 flag combinations, with two divergences priced exactly rather than waved through: 1. `k + 3n`, worth -70 B per P-256 ladder and -102 B per P-384 ladder. Unlike secp256k1 — where the reference pools those three pushes, so this tier matches raw-for-raw once the pool is on — the NIST reference (`cEmitMul`) pushes raw literals under EVERY variant, so the divergence is constant. The two cases stay in separate branches with a comment against merging them. 2. MINIMALDATA on one-byte blobs: the reference writes `push [0x02]` as `OP_2`, this tier's encoder always length-prefixes. +2 per site. This one is PRE-EXISTING and flag-independent — visible with every flag off, on `pNNNEncodeCompressed`, which no flag reaches. Nothing gated it before. `allowedDelta` derives both from `ladderCount(name, variant) * THREE_N` rather than a table of constants, so a tier that combed `u2*Q` — which would be combing an attacker-supplied base, outside the interval argument — changes the ladder count and fails rather than absorbing the difference. Verified: 761/761 Zig tests, including the conformance goldens and both op-count golden tests, so flags-off output is unchanged for the NIST and secp256k1 families alike. End-to-end, a p256MulGen contract compiles byte-identically through this CLI and the TypeScript one with all three flags. The parity test's NIST loops run on an arena: 56 bundles including a 3.9M-op verifyECDSA_P384, where GPA bookkeeping was 178 s of 191 s. A separate testing.allocator test covers the comb path, the pooled slots and the verifier's transferred bundle so a leak there is still caught.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not for merge as-is. This is the script-size exploration from the brief, now complete across all seven compiler tiers. Every optimization is opt-in and default output is byte-identical everywhere. §"Landing this" at the bottom says what would still have to happen.
Headline
conformance/tests/p256-wallet— the fixture the brief calls its "959,592 B reference implementation" — 958,792 → 147,113 bytes (−84.7 %).8.8 MB off the corpus. 43 of 72 fixtures changed, none grown.
p256-wallet's constant pushes went 697,019 → 2,116 bytes. What remains is 70.0 % stack traffic and 25.4 % arithmetic.Three assumptions the measurements overturned
p256-walletwas constant pushes, and 680,850 B of it was 20,025 separate 33-byte pushes of one number — the field prime, re-pushed at every modular reduction. Pooling it into a stack slot is the single largest win in the whole stack, and it is not an algebraic optimization.OP_PICK/OP_ROLLsites sit deeper than 16, where the depth push crosses into 2 bytes. Depths 0–2 are all one byte.05-stack-lower.tsentirely — they emit Stack IR directly through a hand-written tracker. That is 13.4 MB of the 13.5 MB corpus. This is the architectural finding of the exercise and it is why the work split into two independent prototypes.All seven tiers, byte-identical
Flags in every tier:
--ec-constant-pool --ec-reduction-sinking --ec-fixed-base-combFor one
ecMulGencontract with all three on, all seven compilers emit the same 50,157 bytes (down from 424,567):conformance/ec-flag-parity/parity.test.tscodegen/ec_flag_parity_test.gotests/ec_flag_parity_tests.rstests/test_ec_flag_parity.pytest/codegen/test_ec_flag_parity.rbcodegen/EcFlagParityTestpasses/helpers/ec_flag_parity_test.zigWhy a new gate was needed
The flags default off, so the ordinary conformance suite — which compiles with defaults — cannot see them at all. Seven tiers could each ship a different
--ec-constant-pooland the suite would stay green.That is not hypothetical. The flags change which reduction form is emitted and which addition formula each ladder round uses. A tier that ports the constant pool but not the sign lattice's
Reducedprecondition produces a script that is smaller, passes every test it has, and is wrong onecAdd((0,1), (2^256−1,1)).conformance/ec-flag-parity/expected.jsonpins the exact script the TS reference emits for all 24 EC emitters under all 4 flag combinations, as a byte count plus a SHA-256. It is derived, never hand-edited, and re-derived in-process by its own test so it cannot go stale.Six defects it caught
None findable by a tier-local test:
cDecomposePointnever recorded that BIN2NUM of an unsigned coordinate isNonNegative--ec-reduction-sinking(94,137 vs 90,610)+npushes incEmitMulrouted through the pool..Default::default()--ec-constant-poolreached the frontend and vanished; the compile succeeded and emitted the unoptimized scriptu2·Qladder was called without optionsThe Go
NonNegativefact and the Rust..Default::default()are the interesting two: both produce a compiler that works, passes its suite, and silently does not do what the flag says.Soundness
The sign lattice is a sign lattice, not a modular-domain lattice. Multiply and add need only
dividend ≥ 0, which unsigned decoding gives for free (~70 % of reductions). Subtraction's cheap form needssubtrahend < p— andOP_BIN2NUMof 32 unsigned bytes does not imply that. The blanket version passed 256 EC oracle assertions and was still wrong:ecAdd((0,1),(2^256−1,1))differs by exactly2^256 − p = 2^32 + 977.Reducedis tracked separately for that one reason.The comb re-derives the interval argument rather than inheriting it. The binary ladder uses the cheap incomplete add everywhere but its last step, justified by an interval argument over
c_i mod n— and that comment insists the argument be redone by anything changing the offset or the iteration count. A comb changes both.comb.ts#combSafeRoundsis that argument as executable interval arithmetic: it proves 81 of 86 rounds at w=3 and falls back to the complete add-or-double form on the rest (~1.2 kB).combParamssearches for the scalar offset rather than reusing the ladder's+3n— right for P-256 and secp256k1 at w=3, wrong for P-384, which needs+5n.secp256k1's curve record is written out, not templated.
makeCurvehardcodes the NISTa = −3; secp256k1 isy² = x³ + 7. A wrongathere does not produce an obviously broken table — it produces a table of points on a different curve, which that curve's on-curve check accepts. The published 2G vector is pinned for exactly that.ecMul/p256Mul/p384Mulare deliberately not combed. Their bases arrive at run time and the interval argument does not cover an attacker-chosen point. Asserted in the fixture test.Straus/Shamir: measured, then rejected
Not skipped — measured. Complete addition costs +299 B/round, taking the P-256 ladder 90,610 → 167,410 (+84.8 %). Joint-and-complete lands at ≈700 B per bit-position against 690 for two independent ladders. Straus only wins with the incomplete formula, whose interval argument collapses once the accumulator is
c_i·G + d_i·Qwith attacker-suppliedQ: chooseQ = k·Gand solvec_i + d_i·k ≡ 1 (mod n). The comb was taken instead because its single compile-time generator keeps the argument intact.A miscompile the byte counts did not catch
The liveness scheduler once accepted a witness the shipping compiler rejects. Byte counts, all 72 goldens, and 4,113 unit tests were green.
conformance/witnesses/caught it throughrunDifferentialExecution— 2 of 86 cases. Cause: restoring alt-stack spills immediately before anifleaves a layoutlowerIf's arm reconciliation was not written for. Fix: refuse to spill in any scope with control flow ahead.My first bisect falsely acquitted the spilling: with commutative reordering disabled, the method-level cost guard simply preferred the baseline schedule, so no spilling happened at all. Only after confirming the variant still changed bytes did the second bisect mean anything.
Tier-specific findings
Rust pushed the EC constants as pre-encoded script-number byte blobs because they exceed
i128.PushValue::Intcarries aBigInt, so the blob was never necessary — and it cost real things: aBytespush is invisible to the peephole's constant folding (which is whyk + 3nhad to be hand-folded there and only there) and invisible to the sign lattice. Switching toIntleft the shipped bytes unchanged and brought the raw op-count goldens into agreement with Go, where they had been 4 ops short.Zig carried the same duplication in
nist_ec_emitters.zig(NistTracker, predating the lattice) and it was deleted the same way — the NIST emitters now shareec_emitters.ECTracker. Its one curve-specific field became a parameter; nothing about a tracker is per-curve.Rust also carried a hand-copy of
ECTrackerinp256_p384.rs, commented "duplicated since it's private there". Fine for 200 lines of stack bookkeeping; not fine once it carries a sign lattice whose transfer functions decide which reduction shape gets emitted — two copies is two chances to proveReducedwhere onlyNonNegativeholds. Deleted; the tiers share one tracker.Zig is the one tier that cannot assert the raw hash, for two reasons — both differences in spelling that its own peephole normalises away before the script ships.
k + 3n. Its peephole reassociates onlyi64push_intchains, and a 256/384-bit constant is apush_datablob in its IR, so the offset stays pre-folded there while the reference emits three+nsteps its own peephole collapses. The two curve families differ in when this bites: the secp256k1 reference routes those pushes through the constant pool, so Zig matches raw-for-raw as soon as--ec-constant-poolis on and diverges only underoff; the NIST reference (cEmitMul) pushes raw literals under every flag combination, so the divergence there is constant — −70 B per P-256 ladder (3×(1+33)+3against(1+33)+1) and −102 B per P-384 ladder. Copying either family's conditional into the other is a live way to get this wrong.MINIMALDATA on one-byte blobs. The reference writes
push [0x02]asOP_2; Zig's encoder always length-prefixes. Two such pushes per site — the0x02/0x03prefix pair indecompressPubKey, and the parity select inpNNNEncodeCompressed. This one is pre-existing and flag-independent: it is there with every flag off, on an emitter no flag reaches. Nothing gated it before this PR.Zig therefore gates on the raw byte count with both priced exactly in
allowedDelta— which returns zero for every emitter they do not name, and derives the ladder term fromladderCount(name, variant) × THREE_Nrather than a table of constants, so a tier that combedu2·Q(an attacker-supplied base, outside the interval argument) changes the count and fails rather than absorbing the difference. Plus end-to-end hex identity through the CLI; the fixture carries apostPeepholemeasurement for the distinction.Zig's cost model also differs by construction: its
roll/pickcarry the depth themselves, so they costsizeOfScriptNumber(depth) + 1where every other tier charges 1 and counts a separate push.Ruby needed an explicit extended-Euclid
mod_inverse:Integer#powrejects a negative exponent, so there is nox.pow(-1, m)shortcut.Verification
script-size-check72/72 ok, and every tier pins that flags-off reproduces the shipping hash for all 24 emitters.estimateScriptBytes(ops) === emitMethod(ops).scriptHex.length / 2, swept over the whole corpus in every tier.ec-comb.test.ts(83),ec-constant-pool-equivalence,ec-reduction-sinking,liveness-scheduler-equivalenceincluding the combinedallvariant, plus the real-crypto witness corpus.One thing to know about the history
gofmt -w codegen/reformats the whole directory. Running it while porting swept 43 unrelated files (babybear, bn254, koalabear, sp1_fri, …) into the Go commit as pure whitespace churn.fix(go): revert the gofmt sweep…restores them; the net diff for those files is zero, but the round-trip is visible in the history.Landing this
Nothing here should merge as-is:
script-size-baseline.jsonwould trip its 50 % shrink guard by design.Full write-up including the answers to all eight of the brief's questions:
docs/experiments/script-size-optimizer-results.md. Supersedes #156, #158, #159.