fix(kv-unified): exact token key in the leaf engine (#109); tests from the #110 review

[theaspirational]

Follow-up to #110 with the items from its review thread. Closes #109.

What changes

Leaf engine fix (#109). The leaf engine now groups labels by the exact token count (stateKey(label, 0, k, f)). Before, a token bucket put labels with different token counts in one group and dropped the one with more tokens. But fewer tokens is not always better: budgetPenalty punishes going under budget. The engine has no error envelope, so it returned a worse cut and still reported a bound of 0. Continuity and fidelity buckets stay, because they only split groups; dominance still compares the real values. This is fix 1 from #109.

Who it affects. After #110 the auto route never uses the leaf engine. Only an explicit engine: 'leaf' reaches it, and nothing in src/ sets that, so the default route is unchanged. A caller that does pick leaf gets more labels and may need a larger labelCeiling (numbers below).

Latent-demand approximate flag. It used to check whether bucket sizes were configured. Explicit options require all three to be positive, so it was always true, even for exact results. It now reads the solve's reported score error bound: approximate only when the bound is above 0.

Tests

Each new test was checked by breaking the line it guards; it fails every time.

• kv-unified leaf engine: token-bucketed dominance can discard the better cut while reporting a zero regret bound #109 repro: the bucketed leaf engine picks the oracle's cut, with or without a relevant cache and with or without a protected hole. The auto route's regret stays within its reported bound.
• Random sweep on equal-size chunks (60 cases, tie-heavy): leaf score equals the oracle score. It fails on the old key.
• Packed and object DAG storage both throw SparseLabelCeilingError (labelCeiling: 1).
• Selective scoring: a candidate whose lower bound equals the best upper bound is still evaluated (catches > → >= at kv-unified-selective.ts:121).
• PackedTraceArena: every assignment is right across the 2^18 page boundary.
• Latent demand: exact results are not marked approximate, on both the auto and the leaf engine.

npm test: 844/844.

Cost of the fix (leaf engine only)

Measured in review round 2: explicit engine: 'leaf', buckets 100/100/100, maxTokens = 60% of raw.

chunks	labels before → after	time before → after
24	2,425 → 15,876	12 → 55 ms
28	5,392 → 41,961	19 → 117 ms
32	9,360 → 96,059	30 → 354 ms

It grows faster than the forest. At 32 chunks with labelCeiling: 20000, the old key returned a (possibly wrong) cut; the new key throws SparseLabelCeilingError. This is the input for the open question below.

Review rounds

Round 1: GPT-6 Astra, xhigh. Approve.

• The page-boundary test only checked the count and one step, so it missed a wrong-page lookup. Fixed: it checks every step now.
• The leaf engine reported the configured token bucket though it ignores it. Fixed differently in round 2 (see the flag above).
• Ties: when two cuts score the same, the leaf engine can keep a different one than the oracle. Not changed: this is older than this PR, and the score is the same.

Round 2: Claude Opus, xhigh. Approve.

• The label growth kv-unified leaf engine: token-bucketed dominance can discard the better cut while reporting a zero regret bound #109 asked to measure was not measured. Done: table above, plus a changelog note about labelCeiling.
• Round 1's flag fix did not work: the approximate flag was still always true. Fixed: the flag now follows the reported bound.
• Nothing tested round 1's stats change. Covered: the new flag test.
• The changelog did not say the fix needs engine: 'leaf'. Fixed.
• The kv-unified leaf engine: token-bucketed dominance can discard the better cut while reporting a zero regret bound #109 guard relied on one hand-built case; random sweeps with mixed chunk sizes don't catch the bug. Fixed: added the equal-size sweep.
• It also confirmed: the fix is sound, the auto route never reaches the leaf engine, and 793 random cases match the oracle (the old key failed 79 of them).

Open, not in this PR

• Ties: exactness here means the lowest score, not the oracle's tie-picked cut. This can happen whenever the policy gives a dominance dimension zero weight (continuityLambda: 0, or cacheLambda: 0 with a relevant cache).
• Protected holes back to the leaf engine? Now possible without the kv-unified leaf engine: token-bucketed dominance can discard the better cut while reporting a zero regret bound #109 bug, but the table above shows the cost. Your call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q1H4PTNHqyTg3G74YnFK9e

[Anarchid]

Verdict: approve

Reviewed at 8bc3788 against main eca70da (0.11.0).

The #109 fix is sound, the tests actually pin the behaviour, and the scope is stated honestly. The one follow-up is the approximate flag: it still reports true in every live steady-state solve, so the second changelog line promises more than it delivers. CI hasn't run yet: this is a fork PR, and both workflow runs are waiting for maintainer approval.

Minor

1. In live solves the approximate flag is still always true. (Sol) approximationBound adds hysteresis = adoptEpsilon whenever options.presentation is set (kv-unified-pareto.ts:700-709), and the live adapter passes both. So with any positive adoptEpsilon, once the first presentation is accepted, every auto-route latent-demand evaluation reports approximate: true, even when nothing was pruned. The new test sets neither presentation nor adoptEpsilon. Fix: split the pruning error from the hysteresis allowance in the propagation stats and key the flag on pruning only. Otherwise, reword the comment and changelog as "nonzero regret bound, hysteresis included".
2. The flag reads only the conservative solve. (Sol) expectedImprovement (also used as a ranking tiebreak) and the shared baseline can come from a solve with a nonzero bound while the flag says false. Sol has a deterministic case: expected bound 2226.68, conservative 0 → approximate = false. Fix: OR the same predicate across baseline, expected and conservative.
   Neither #1 nor #2 changes any decision. Nothing in CM, AF or connectome-host reads approximate; it's only exposed through lastDemandEvaluations.
3. A recipe can reach engine: 'leaf'. KvUnifiedOptions extends ParetoSolveOptions. connectome-host passes the whole kvUnified object through without rejecting unknown keys, and it's spread into solve() (kv-unified.ts:64). A recipe with kvUnified.engine: "leaf" therefore gets the extra label growth, and SparseLabelCeilingError is uncaught. That's already true on every engine, and the body and changelog disclose the ceiling change. No recipe we know of sets engine. Follow-up: treat engine as solver-internal and pin it to 'auto' at the adapter.

Verified fine

• stateKey(label, 0, …) means the exact renderedTokens, with no division (:960-968). dominates() compares real values, so with an exact token key no cross-token prune can happen. The continuity and fidelity buckets only partition comparisons, dominance still uses the real losses, and those are monotone in policyScore. So the engine stays exact.
• Routing: omitted, auto and dag all go to the DAG branch, and only the literal 'leaf' reaches solveLeaf (:152-166). Nothing in CM, AF or connectome-host src/ sets it.
• The 60-case sweep uses a fixed seed and a local LCG, with no timing assertions. The #109 repro and the latent-demand test fail on the old logic.
• main + #114–#118 merged together builds and passes 852/852 (node 22), including this PR's new tests. That's the only test run so far, since GitHub CI is still pending approval.

Method

Two reviewers in parallel. Sol (Codex gpt-5.6-sol, xhigh, read-only, no network; ~10 min, 56 commands, 2.8M input tokens (2.5M cached), 26k output) and Claude (the combined-tree build and run, recipe and CI state, and a check of every Sol citation). Sol only: #1, #2, and the pass-through detail in #3. Claude only: the cause of the empty status rollup (fork approval) and the check that no recipe sets engine. Sol rated #3 major; it was lowered because ceiling errors are already fatal on every engine and the change is disclosed.