Repository navigation
kolibri1 TT: score the b2ii argmax chain at decode positions — the golden index was off by input_len - #3453
Merged
Conversation
lu-zero
force-pushed
the
row/tt-b2ii-index
branch
from
October 12, 2026 11:53
2c4efce to
7eac167
Compare
…ions — the golden index was off by input_len The b2ii token gate's ARGMAX CHAIN compared the d-th decoded token against generated_ids[step], but `step` is the global walk step: since the one-token-per-step prompt walk (9124217) the decode steps start at input_len, while the goldens' generated_ids is laid out over decode positions only (32 entries, no prompt prefix — kolibri1_goldens.json, max_new_tokens=32). Every scored position was therefore off by input_len, and the gate's counts were not comparable to the W3 methodology's baseline (test_kolibri1_w3.cpp:314 indexes generated_ids[decode_step]). Three sites in tests/vllm/models/test_kolibri1_tt_b2ii.cpp carried the same class of bug; all three are fixed: the ARGMAX CHAIN comparison (its flip records now name the decode position); the near-tie instrument's teacher-forced walk, which fed generated_ids[step-1] — a shifted chain, so pass 1 never ran on the golden prefix and its teacher-forced flip counts were meaningless — now feeds generated_ids[d-1]; and the SCRATCH dbg TF walk, which fed generated_ids[i - input_len - 1] — off by one and an out-of-bounds read at the first decode step — now feeds the golden token at position i. Red-first: with the wrong index (branch HEAD), both arms reproduce the recorded 0/8 verdicts — every prompt flips at its first compared position, recorded at step = input_len with want = generated_ids[input_len]. With the fix the counts move: ON 0/8 -> 3/11 (3 near-ties, 5 hard), OFF 0/8 -> 12/20 (6 near-ties, 2 hard), two identical runs per arm with byte-identical streaming counters, which re-checks the nondeterminism acceptance on the corrected indexing. The gate's CHECKs stay RED by design: the flips are the known B2b-ii residual. The issue Resolution also corrects the record: the 26/33 baseline was itself a wrong-index measurement (recorded after 9124217; under the wrong index even a perfect engine scores 48/54), not a pre-walk baseline. Issue: ISSUE-LOCAL-01M4P21TZ1Q9CA7P66XBH0BQTX FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/tt-b2ii-index
branch
from
October 12, 2026 13:18
7eac167 to
ca60650
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The b2ii token gate's ARGMAX CHAIN compared the d-th decoded token against
generated_ids[step], butstepis the global walk step: since the one-token-per-step prompt walk (9124217) the decode steps start atinput_len, while the goldens'generated_idsis laid out over decode positions only (32 entries, no prompt prefix). Every scored position was off byinput_len, so the gate's counts were not comparable to the W3-methodology baseline — the recorded 26/33 baseline was itself a wrong-index measurement (under the wrong index even a perfect engine scores 48/54).Three sites carried the same class of bug, all fixed in tests/vllm/models/test_kolibri1_tt_b2ii.cpp: the ARGMAX CHAIN comparison (flip records now name the decode position); the near-tie instrument's teacher-forced walk, which fed
generated_ids[step-1]— a SHIFTED chain, so its pass 1 never ran on the golden prefix and its teacher-forced flip counts were meaningless (this invalidates the earlier 47-hard-flip measurement — the residual classification that rested on it needs a re-measurement with the fixed instrument, recorded as owed); and the SCRATCH dbg TF walk, which fedgenerated_ids[i - input_len - 1]— off by one and an out-of-bounds read at the first decode step.Corrected verdicts (operator-verified; two identical runs per arm with byte-identical streaming counters, which re-checks the #3452 nondeterminism acceptance on the corrected indexing): ON 0/8 -> 3/11 (3 near-ties, 5 hard); OFF 0/8 -> 12/20 (6 near-ties, 2 hard). The OFF arm was much better than the record said. The gate's CHECKs stay red by design: the remaining flips are the known B2b-ii residual, now correctly measured.
Red-first: with the wrong index both arms reproduce the recorded 0/8 verdicts (every prompt flips at its first compared position, recorded at step = input_len).
Stacks on #3452 (the nondeterminism fix). Issue: ISSUE-LOCAL-01M4P21TZ1Q9CA7P66XBH0BQTX.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]