Repository navigation
Conversation
…cted walk The prior hard-flip classification was computed on a shifted chain (generated_ids[step-1] fed instead of generated_ids[d-1]), so its per-step teacher-forced numbers answered the wrong question. This re-runs the adjudication on 2c4efce's corrected walk and re-derives the residual. Both arms reproduce the corrected counts exactly (ON 69 tf flips: 29 near-tie, 40 HARD, worst 10.617 nats; OFF 44: 27/17, worst 8.647) - the residual is real. The drift origin is unchanged (L0 MoE input, first divergent stage at every decode step from step 0, 1-bf16-ULP class), and the router-gain mechanism HOLDS: CPU-vs-TT top-6-of-384 expert-set overlap averages 3.0-3.7 of 6 at every layer under the golden prefix, and the new route_scores probe shows the top6-top7 selection margin below 0.05 nats at 26-44 of 50 layers at every hard flip, while top1-top2 stays robust. The flips SPREAD across 7 of 8 prompts and the whole step range in both arms, so the band-2.5 residual is classified STRUCTURAL (drift x router gain), not a single-site repair; the levers are host-free shadow reconciliation (halves the hard count) and f32 device attention/norm. A scratch route_scores dump (sorted top-8 biased selection scores behind VT_KOLIBRI1_TT_STAGE_DUMP) accompanies the existing stage probes; the adjudication table lives in docs/bench-evidence/kolibri1-tt-readjudication-20261012.md and the issue resolution records the supersession. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…on evidence and parsers
The review on the P150 readjudication found the SCRATCH dbg TF log's raw
TFstep label is not the gate's decode position: the walk prints
d = i - input_len before the +1 that the logits' context position implies,
so raw `TFstep N` is decode position N+1 (the last prompt token's
TFstep -1 line is decode position 0). The test now prints both labels and
one short prompt-4 device leg verified the new wording. The doc gains an
explicit mapping section; its SCRATCH-derived step numbers are restated as
decode positions (the tf-p legs' decode-position-0 forward diverges at L0
dh2 like every later one, so the origin claim survives the relabel), and
the Q2 example is restated as the walk's first prompt forward, not a step.
The structural verdict is restated to what the evidence supports: the flip
spread across 7 of 8 prompts and the whole decode range is consistent with
the shared L0-origin drift amplification, and it rules out single-site
patches — it does not rule out that a chain-wide f32 fix (lever 2) clears
most of the residual, since that targets the origin itself.
The evidence the tables derive from is now durable: docs/bench-evidence/
kolibri1-readj/ carries the parsed OFF/ON.json and the corrected-label
tf_origin.json, the tf-p{1,5,4}.log raw legs gzipped, and the three
parsers that regenerate every doc table from them (the 22-98 MB full-gate
arm logs stay off-tree with their sizes and regeneration recipe recorded).
parse_overlap.py and parse_tf_origin.py were rerun against the committed
logs and reproduce the doc's Q1/Q2 numbers exactly.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
…e add-norm kernel bias, not the shadow reconciliation The lever-1 premise (host-free shadow reconciliation is the largest single lever behind the 40-vs-17 teacher-forced hard-flip arm delta) is falsified on the current base. The delta reproduces exactly (ON 40 / OFF 17 TF HARD, gate 5/2), so the localai-org#3452 UAF fixes did not close it, but the stage-trace localization puts the first divergence at `L0 dhn` — the input_layernorm — on bit-identical inputs (emb hash dd76406a, gam sum 858.502781 in both arms): the host f32 loop outputs 9.846392, the device arm 9.765238 (ratio 0.9918). VT_TT_NORM_DEBUG proves the device norm's inputs are bit-identical while its output reads 0.7% low, and the committed rmsnorm micro on the same patched substrate measures 0.9997 clean on random inputs — the kernel bias is input-distribution-dependent. Bisect: forced-host rope (byte-identical flip set) and forced-host PA (identical chain) are ruled out; forced-host residual reads WORSE (7 hard) because the host round-trips desync the device shadow, so it is not a clean probe and the pre-localai-org#3452 "host residual is the accurate arm" finding is dead. The composed f32 rms_norm repair remains substrate-blocked in-model (Invalid subtile broadcast type, binary_ng_device_operation.cpp:224, re-confirmed on this stack). MECHANISM: host-free decode forces the device rms_norm at T=1 for capture; that kernel carries a ~0.7% systematic input-dependent bias on kolibri1 activations; 50 layers x 2-3 norm sites compounds it into multi-nat router-logit shifts and the 40-vs-17 arm delta. Priced fix options and the recommended next lever (feed the in-model emb/gam row through the micro to split input-dependence from program provenance) are recorded in the evidence file and the issue. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lever 1 of the #3454 adjudication (host-free shadow reconciliation) is FALSIFIED as the arm-delta mechanism, and the real one is pinned: the host-free decode path forces the DEVICE rms_norm at T=1 (for capture), and that kernel carries a ~0.7% systematic INPUT-DEPENDENT bias on kolibri1 activations — inputs bit-identical across arms (emb hash, gamma sum, VT_TT_NORM_DEBUG probes on dev_x/norm_f32), output 0.7% low (host f32 9.846392 vs device 9.765238) — while the committed rmsnorm micro on the same patched substrate measures 0.9997 clean on random inputs of the same shape/eps. The kernel bias is input-distribution-dependent. 50 layers x 2-3 norm sites compound it into multi-nat router-logit shifts: the 40-vs-17 teacher-forced hard-flip arm delta.
Bisect: forced-host rope (byte-identical flip set) and forced-host PA (identical chain) RULED OUT; forced-host residual reads worse (7 hard) because host round-trips desync the device shadow — not a clean probe, and the pre-#3452 "host residual is the accurate arm" finding is dead. The composed f32 rms_norm repair remains substrate-blocked in-model (Invalid subtile broadcast type, binary_ng_device_operation.cpp:224, re-confirmed on this stack).
The #3452 UAF fixes are confirmed correct but orthogonal: they fixed crashes; the arm delta is kernel precision on real distributions, and it reproduces exactly on the current base (40/17).
Priced next diagnostic (recorded in the evidence file + issue): feed the in-model emb/gam row through the standalone micro to split INPUT-DEPENDENCE (a kernel precision bug triggered by real activation distributions — fix in the substrate kernel) from PROGRAM PROVENANCE (the in-model call selects a different kernel config than the micro — fix at the call site). Either way the fix site is the tt-metal rms_norm path this stack already patches.
Issue: ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY (files the arm-delta re-measurement ownership; Resolution carries the re-attribution).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]