diff --git a/.agents/issues/BACKEND-TENSTORRENT/ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY.md b/.agents/issues/BACKEND-TENSTORRENT/ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY.md new file mode 100644 index 0000000000..78a8c7d856 --- /dev/null +++ b/.agents/issues/BACKEND-TENSTORRENT/ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY.md @@ -0,0 +1,46 @@ +ID: ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY +Title: kolibri1 TT host-free ON arm produces more teacher-forced hard flips than OFF (40 vs 17) +Row: BACKEND-TENSTORRENT +State: OPEN +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-15 +Updated: 2026-10-15 +Closed: - + +## Problem + +The #3454 readjudication attributes the kolibri1 P150 hard-flip count to device bf16 1-ULP drift x router gain, with the ARM DELTA (host-free ON 40 teacher-forced HARD flips vs OFF 17) left as the largest single lever: the host-free path's shadow handling (WeightViewShadows/Bfp8WeightShadows reconciliation) is suspected to introduce more drift than the eager path. The #3452 UAF fixes (memo eviction order + shadow sweeps at UnregisterHostBuffer/MarkScratchAcquired) landed after that measurement; this issue owns re-measuring the delta on the current base, bisecting which host-free-specific path carries the extra drift (rope/residual/PA force-host knobs, expert-weight shadow residency), and fixing it if it is a defect (stale shadow serve) or pricing it precisely if it is inherent precision difference. + +## Resolution + +- 2026-10-15 (row/tt-kolibri-shadow-recon): the delta REPRODUCES exactly on the + current base (ON 40 / OFF 17 TF HARD, gate 5/2). The shadow-reconciliation + hypothesis is FALSIFIED: both arms share the shadow maps and the #3452 + sweeps, and the first diverging stage in the run is `L0 dhn` — the + input_layernorm — on BIT-IDENTICAL inputs (emb hash `dd76406a`, gam sum + 858.502781 both arms): host f32 loop 9.846392 vs device 9.765238 (ratio + 0.9918). `VT_TT_NORM_DEBUG` proves the device norm's inputs are bit-identical + (`dev_x`/`norm_f32` −20.613504, `wf32` 858.502781) while `dev_y` reads + 9.777145: the deficit is in the tt-metal rms_norm kernel ON THE MODEL'S + ACTIVATION DISTRIBUTION — the committed micro (random inputs, same shape/eps, + patched `fp32_acc=true` substrate, verified installed) measures 0.9997 clean. + Bisect: `VT_TT_FORCE_HOST_ROPE` byte-identical flip set (ruled out); + `VT_TT_FORCE_HOST_PA` identical chain (ruled out); `VT_TT_FORCE_HOST_RESIDUAL` + reads 7 HARD — WORSE — because the host round-trips desync the device shadow + (the recorded 1e37 collapse class), so it is not a clean probe and the "host + residual is more accurate" reading of the pre-#3452 bisect is dead. + `VT_TT_RMSNORM_COMPOSED=1` re-confirmed substrate-blocked in-model + (`Invalid subtile broadcast type`, binary_ng_device_operation.cpp:224). + MECHANISM: host-free forces the device rms_norm at T=1 for capture; that + kernel carries a ~0.7% systematic input-dependent bias on kolibri1 + activations; 50 layers x 2-3 norm sites compounds into multi-nat router-logit + shifts and the 40-vs-17 arm delta. Evidence: + docs/bench-evidence/kolibri1-readj/shadow-recon-bisect-20261015.md. + NEXT LEVER: substrate characterization — feed the in-model emb/gam row + through the micro to split input-dependence from program provenance + (program-cache/config under the model call pattern), then fix tt-metal or + force the clean config in-model. STAYS OPEN (the delta is not fixed; it is + re-attributed). diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.md b/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.md index b8adb9fe94..c5622c9ed8 100644 --- a/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.md +++ b/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.md @@ -195,6 +195,34 @@ B2b-i landed the dense-resident device forward with the routed-expert tier delib driver is the upstream drift class. CLASSIFICATION: selection flips DRIVEN by continuous bf16 device-arm precision drift through the attention/norm chain; per the row's adjudication rule this is an - upstream adjudication decision (f32-enabling the device attention/norm + upstream adjudication decision ( f32-enabling the device attention/norm chain or accepting the band), not a local patch. ISSUE STAYS OPEN on the device gates and the OOM debt. + +- 2026-10-12 (branch row/tt-kolibri-readj, off row/tt-b2ii-index + @ 2c4efce88): the teacher-forced adjudication RE-RUN on the CORRECTED + instrument — the prior classification was computed on a SHIFTED chain + (generated_ids[step-1] fed instead of generated_ids[d-1]), so every + prior per-step teacher-forced number is invalid as step evidence. The + corrected walk reproduces the #3453-era corrected counts exactly (ON 69 + tf flips: 29 near-tie, 40 HARD, worst 10.617 nats; OFF 44: 27 near-tie, + 17 HARD, worst 8.647) — the residual is REAL, not an instrument + artifact. Drift origin UNCHANGED: L0 dh2 (the L0 MoE input) is the first + divergent stage at EVERY decode step from step 0 on all three diverging + prompts (SCRATCH dbg TF prompt, CPU-vs-TT bit hashes), 0.03-2.5% rel per + step — the 1-bf16-ULP-class L0 input-norm origin stands. Router-flip + mechanism CONFIRMED: per-layer CPU-vs-TT top-6-of-384 selected-expert + overlap under the corrected walk averages 3.0-3.7 of 6 (full agreement + only 6-7% of layer-steps) at EVERY layer including L0, median router + logit_sum rel diff 6.7-9.7%; the route_scores probe shows top1-top2 + margins are robust (median 0.13-0.48) while top6-top7 margins sit below + 0.05 nats at 26-41 of 50 layers at every hard-flip step — the incoming + noise exceeds the rank-6/7 boundary margin at most layers. Distribution: + hard flips SPREAD, not concentrated — 7 of 8 prompts, opening bursts + (steps 0-4) on some prompts and isolated late flips (steps 14-26) on + others, both arms. Gate-standard input: the P150 band-2.5 residual is + STRUCTURAL (continuous step-0-resident drift x router gain onto a + sub-0.05-nat selection boundary), not a single-site repair; levers in + order: host-free shadow reconciliation (17 vs 40 hard shows it halves + the count), then f32 device attention/norm chain. Evidence: + docs/bench-evidence/kolibri1-tt-readjudication-20261012.md. diff --git a/docs/bench-evidence/kolibri1-readj/OFF.json b/docs/bench-evidence/kolibri1-readj/OFF.json new file mode 100644 index 0000000000..a95c60fc39 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/OFF.json @@ -0,0 +1,257 @@ +{ + "arm": "OFF", + "summary": [ + 158, + 44, + 27, + 17, + 8.64696 + ], + "chain": [ + 12, + 20, + 8, + 6, + 2 + ], + "hard_flips": [ + { + "prompt": "der Mond scheint hell \u00fcber den Bergen", + "step": 0, + "golden": 948, + "tfarg": 46, + "gap": 3.181908, + "intop5": "no", + "layer_top12_min": 0.0016739999999999533, + "layer_top12_med": 0.24162299999999998, + "layer_top67_min": 0.0002620000000002065, + "layer_top67_med": 0.028259000000000034, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 6 + }, + { + "prompt": "der Mond scheint hell \u00fcber den Bergen", + "step": 1, + "golden": 46252, + "tfarg": 3389, + "gap": 2.273598, + "intop5": "no", + "layer_top12_min": 0.0012659999999999894, + "layer_top12_med": 0.167883, + "layer_top67_min": 0.0007520000000000009, + "layer_top67_med": 0.020367000000000024, + "n_layers_gap67_below_0p05": 40, + "n_layers_gap12_below_0p05": 12 + }, + { + "prompt": "The capital of Australia is", + "step": 18, + "golden": 75361, + "tfarg": 543, + "gap": 3.553511, + "intop5": "yes", + "layer_top12_min": 0.016432000000000002, + "layer_top12_med": 0.484023, + "layer_top67_min": 0.0007150000000000212, + "layer_top67_med": 0.04172799999999999, + "n_layers_gap67_below_0p05": 28, + "n_layers_gap12_below_0p05": 5 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 1, + "golden": 936, + "tfarg": 420, + "gap": 1.92732, + "intop5": "no", + "layer_top12_min": 0.003607000000000027, + "layer_top12_med": 0.21200000000000002, + "layer_top67_min": 0.00011499999999997623, + "layer_top67_med": 0.04392200000000002, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 2, + "golden": 2973, + "tfarg": 936, + "gap": 2.048908, + "intop5": "no", + "layer_top12_min": 0.0017440000000000788, + "layer_top12_med": 0.316585, + "layer_top67_min": 0.0001020000000000465, + "layer_top67_med": 0.044141999999999904, + "n_layers_gap67_below_0p05": 30, + "n_layers_gap12_below_0p05": 10 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 3, + "golden": 476, + "tfarg": 735, + "gap": 3.018167, + "intop5": "yes", + "layer_top12_min": 0.008242, + "layer_top12_med": 0.19427700000000003, + "layer_top67_min": 0.0009399999999999964, + "layer_top67_med": 0.02517999999999998, + "n_layers_gap67_below_0p05": 41, + "n_layers_gap12_below_0p05": 8 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 14, + "golden": 18331, + "tfarg": 502, + "gap": 7.306021, + "intop5": "no", + "layer_top12_min": 0.0006349999999999412, + "layer_top12_med": 0.34341299999999997, + "layer_top67_min": 0.0001620000000001065, + "layer_top67_med": 0.034442000000000084, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 13 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 17, + "golden": 936, + "tfarg": 286, + "gap": 3.616066, + "intop5": "yes", + "layer_top12_min": 0.002801999999999999, + "layer_top12_med": 0.19718500000000005, + "layer_top67_min": 0.0009519999999998419, + "layer_top67_med": 0.036826000000000025, + "n_layers_gap67_below_0p05": 33, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "Translation to German: the weather is beautiful today", + "step": 5, + "golden": 4484, + "tfarg": 286, + "gap": 3.819496, + "intop5": "yes", + "layer_top12_min": 0.0007070000000000132, + "layer_top12_med": 0.287526, + "layer_top67_min": 0.000280000000000058, + "layer_top67_med": 0.03786900000000004, + "n_layers_gap67_below_0p05": 33, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "Translation to German: the weather is beautiful today", + "step": 10, + "golden": 5445, + "tfarg": 2302, + "gap": 3.030571, + "intop5": "no", + "layer_top12_min": 0.003015999999999991, + "layer_top12_med": 0.19559200000000004, + "layer_top67_min": 5.100000000002325e-05, + "layer_top67_med": 0.027216000000000018, + "n_layers_gap67_below_0p05": 35, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "Translation to German: the weather is beautiful today", + "step": 22, + "golden": 5445, + "tfarg": 2302, + "gap": 1.899872, + "intop5": "no", + "layer_top12_min": 0.0015570000000000306, + "layer_top12_med": 0.29370300000000005, + "layer_top67_min": 0.001670999999999978, + "layer_top67_med": 0.042990000000000084, + "n_layers_gap67_below_0p05": 31, + "n_layers_gap12_below_0p05": 5 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 7, + "golden": 51092, + "tfarg": 2459, + "gap": 3.285328, + "intop5": "yes", + "layer_top12_min": 0.004497999999999891, + "layer_top12_med": 0.278218, + "layer_top67_min": 0.0005789999999998852, + "layer_top67_med": 0.04965699999999995, + "n_layers_gap67_below_0p05": 26, + "n_layers_gap12_below_0p05": 6 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 0, + "golden": 12843, + "tfarg": 14031, + "gap": 6.156881, + "intop5": "no", + "layer_top12_min": 0.008246999999999893, + "layer_top12_med": 0.170104, + "layer_top67_min": 0.0010459999999999914, + "layer_top67_med": 0.02381000000000011, + "n_layers_gap67_below_0p05": 39, + "n_layers_gap12_below_0p05": 13 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 1, + "golden": 12843, + "tfarg": 14031, + "gap": 8.646961, + "intop5": "no", + "layer_top12_min": 0.002374999999999905, + "layer_top12_med": 0.24289000000000005, + "layer_top67_min": 0.0004789999999998962, + "layer_top67_med": 0.028886000000000023, + "n_layers_gap67_below_0p05": 38, + "n_layers_gap12_below_0p05": 12 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 2, + "golden": 12843, + "tfarg": 14031, + "gap": 5.815979, + "intop5": "no", + "layer_top12_min": 0.004530999999999952, + "layer_top12_med": 0.128412, + "layer_top67_min": 0.00014899999999995472, + "layer_top67_med": 0.018912999999999958, + "n_layers_gap67_below_0p05": 41, + "n_layers_gap12_below_0p05": 10 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 3, + "golden": 12843, + "tfarg": 14031, + "gap": 4.99181, + "intop5": "yes", + "layer_top12_min": 0.01208100000000012, + "layer_top12_med": 0.24573, + "layer_top67_min": 0.00014399999999997748, + "layer_top67_med": 0.030302999999999997, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 10 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 4, + "golden": 12843, + "tfarg": 14031, + "gap": 3.630234, + "intop5": "yes", + "layer_top12_min": 0.000136000000000025, + "layer_top12_med": 0.20996, + "layer_top67_min": 0.0004939999999999944, + "layer_top67_med": 0.033832, + "n_layers_gap67_below_0p05": 32, + "n_layers_gap12_below_0p05": 10 + } + ] +} \ No newline at end of file diff --git a/docs/bench-evidence/kolibri1-readj/ON.json b/docs/bench-evidence/kolibri1-readj/ON.json new file mode 100644 index 0000000000..28ccf229e4 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/ON.json @@ -0,0 +1,579 @@ +{ + "arm": "ON", + "summary": [ + 170, + 69, + 29, + 40, + 10.617 + ], + "chain": [ + 3, + 11, + 8, + 3, + 5 + ], + "hard_flips": [ + { + "prompt": "der Mond scheint hell \u00fcber den Bergen", + "step": 0, + "golden": 948, + "tfarg": 44, + "gap": 1.746723, + "intop5": "no", + "layer_top12_min": 0.0020540000000000003, + "layer_top12_med": 0.18128000000000002, + "layer_top67_min": 0.00017800000000001148, + "layer_top67_med": 0.021506999999999943, + "n_layers_gap67_below_0p05": 36, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "der Mond scheint hell \u00fcber den Bergen", + "step": 1, + "golden": 46252, + "tfarg": 44, + "gap": 0.933052, + "intop5": "no", + "layer_top12_min": 0.0009640000000000759, + "layer_top12_med": 0.144598, + "layer_top67_min": 0.00016499999999999848, + "layer_top67_med": 0.02083099999999999, + "n_layers_gap67_below_0p05": 38, + "n_layers_gap12_below_0p05": 14 + }, + { + "prompt": "The capital of Australia is", + "step": 0, + "golden": 75361, + "tfarg": 262, + "gap": 10.523823, + "intop5": "no", + "layer_top12_min": 0.0047680000000001055, + "layer_top12_med": 0.16785, + "layer_top67_min": 0.0008289999999999687, + "layer_top67_med": 0.02781699999999998, + "n_layers_gap67_below_0p05": 35, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "The capital of Australia is", + "step": 11, + "golden": 7789, + "tfarg": 262, + "gap": 9.26522, + "intop5": "no", + "layer_top12_min": 0.0015049999999999786, + "layer_top12_med": 0.24800300000000008, + "layer_top67_min": 0.00040699999999982417, + "layer_top67_med": 0.03004399999999996, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "The capital of Australia is", + "step": 14, + "golden": 7252, + "tfarg": 6547, + "gap": 5.249352, + "intop5": "no", + "layer_top12_min": 0.007746999999999948, + "layer_top12_med": 0.21314500000000003, + "layer_top67_min": 0.0007470000000000011, + "layer_top67_med": 0.03867900000000002, + "n_layers_gap67_below_0p05": 32, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "The capital of Australia is", + "step": 15, + "golden": 13919, + "tfarg": 735, + "gap": 3.886755, + "intop5": "no", + "layer_top12_min": 0.008272000000000057, + "layer_top12_med": 0.262342, + "layer_top67_min": 0.00020199999999981344, + "layer_top67_med": 0.03825799999999999, + "n_layers_gap67_below_0p05": 32, + "n_layers_gap12_below_0p05": 4 + }, + { + "prompt": "The capital of Australia is", + "step": 16, + "golden": 37112, + "tfarg": 735, + "gap": 8.259396, + "intop5": "no", + "layer_top12_min": 0.001306999999999947, + "layer_top12_med": 0.184584, + "layer_top67_min": 0.0014459999999998363, + "layer_top67_med": 0.032416, + "n_layers_gap67_below_0p05": 37, + "n_layers_gap12_below_0p05": 13 + }, + { + "prompt": "The capital of Australia is", + "step": 18, + "golden": 75361, + "tfarg": 543, + "gap": 4.044167, + "intop5": "no", + "layer_top12_min": 0.011316, + "layer_top12_med": 0.35889899999999997, + "layer_top67_min": 0.0008639999999999759, + "layer_top67_med": 0.0232, + "n_layers_gap67_below_0p05": 40, + "n_layers_gap12_below_0p05": 4 + }, + { + "prompt": "The capital of Australia is", + "step": 21, + "golden": 6980, + "tfarg": 9255, + "gap": 8.207735, + "intop5": "no", + "layer_top12_min": 0.0045799999999998064, + "layer_top12_med": 0.32849699999999993, + "layer_top67_min": 0.00031500000000006523, + "layer_top67_med": 0.02279600000000004, + "n_layers_gap67_below_0p05": 36, + "n_layers_gap12_below_0p05": 5 + }, + { + "prompt": "The capital of Australia is", + "step": 26, + "golden": 13919, + "tfarg": 1366, + "gap": 2.124249, + "intop5": "no", + "layer_top12_min": 0.0022460000000004143, + "layer_top12_med": 0.26252699999999995, + "layer_top67_min": 0.0005800000000000249, + "layer_top67_med": 0.02806800000000001, + "n_layers_gap67_below_0p05": 31, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 1, + "golden": 936, + "tfarg": 420, + "gap": 8.78104, + "intop5": "no", + "layer_top12_min": 0.006489, + "layer_top12_med": 0.37050799999999995, + "layer_top67_min": 6.0000000000004494e-05, + "layer_top67_med": 0.038656000000000024, + "n_layers_gap67_below_0p05": 33, + "n_layers_gap12_below_0p05": 8 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 3, + "golden": 476, + "tfarg": 735, + "gap": 6.007673, + "intop5": "no", + "layer_top12_min": 0.0035390000000000144, + "layer_top12_med": 0.277022, + "layer_top67_min": 0.0006679999999998909, + "layer_top67_med": 0.020653, + "n_layers_gap67_below_0p05": 38, + "n_layers_gap12_below_0p05": 10 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 4, + "golden": 1821, + "tfarg": 353, + "gap": 3.539688, + "intop5": "yes", + "layer_top12_min": 0.00018099999999998673, + "layer_top12_med": 0.330202, + "layer_top67_min": 0.00022099999999980469, + "layer_top67_med": 0.02795700000000001, + "n_layers_gap67_below_0p05": 38, + "n_layers_gap12_below_0p05": 8 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 6, + "golden": 18331, + "tfarg": 1584, + "gap": 7.250818, + "intop5": "no", + "layer_top12_min": 0.010554000000000001, + "layer_top12_med": 0.318635, + "layer_top67_min": 0.0006050000000001887, + "layer_top67_med": 0.031596, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 5 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 7, + "golden": 936, + "tfarg": 44, + "gap": 3.047478, + "intop5": "yes", + "layer_top12_min": 0.0013809999999998546, + "layer_top12_med": 0.28210399999999997, + "layer_top67_min": 0.00027700000000008274, + "layer_top67_med": 0.028172000000000003, + "n_layers_gap67_below_0p05": 37, + "n_layers_gap12_below_0p05": 11 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 11, + "golden": 938, + "tfarg": 1584, + "gap": 3.704597, + "intop5": "no", + "layer_top12_min": 0.010965999999999976, + "layer_top12_med": 0.2533479999999999, + "layer_top67_min": 5.900000000003125e-05, + "layer_top67_med": 0.026286000000000004, + "n_layers_gap67_below_0p05": 37, + "n_layers_gap12_below_0p05": 8 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 14, + "golden": 18331, + "tfarg": 2973, + "gap": 7.035637, + "intop5": "no", + "layer_top12_min": 0.02868000000000004, + "layer_top12_med": 0.31353, + "layer_top67_min": 0.00024399999999991095, + "layer_top67_med": 0.04563100000000003, + "n_layers_gap67_below_0p05": 27, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 16, + "golden": 26177, + "tfarg": 502, + "gap": 7.852909, + "intop5": "no", + "layer_top12_min": 0.014342999999999995, + "layer_top12_med": 0.31583300000000003, + "layer_top67_min": 0.00022700000000000498, + "layer_top67_med": 0.03534400000000004, + "n_layers_gap67_below_0p05": 36, + "n_layers_gap12_below_0p05": 5 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 17, + "golden": 936, + "tfarg": 46, + "gap": 3.945122, + "intop5": "no", + "layer_top12_min": 0.008265999999999996, + "layer_top12_med": 0.255118, + "layer_top67_min": 0.000756000000000201, + "layer_top67_med": 0.0350379999999999, + "n_layers_gap67_below_0p05": 31, + "n_layers_gap12_below_0p05": 6 + }, + { + "prompt": "Wissen ist Macht, aber Zeit ist Geld", + "step": 21, + "golden": 938, + "tfarg": 4314, + "gap": 1.905293, + "intop5": "no", + "layer_top12_min": 0.0024179999999999757, + "layer_top12_med": 0.24708200000000002, + "layer_top67_min": 0.0006170000000000897, + "layer_top67_med": 0.03241100000000019, + "n_layers_gap67_below_0p05": 32, + "n_layers_gap12_below_0p05": 12 + }, + { + "prompt": "x1 = 3, x2 = 7, x3 = 11, x4 =", + "step": 2, + "golden": 51, + "tfarg": 44, + "gap": 2.728287, + "intop5": "yes", + "layer_top12_min": 0.002854999999999941, + "layer_top12_med": 0.13317800000000002, + "layer_top67_min": 0.0007670000000001842, + "layer_top67_med": 0.01722800000000002, + "n_layers_gap67_below_0p05": 44, + "n_layers_gap12_below_0p05": 11 + }, + { + "prompt": "x1 = 3, x2 = 7, x3 = 11, x4 =", + "step": 4, + "golden": 14169, + "tfarg": 32, + "gap": 3.113139, + "intop5": "yes", + "layer_top12_min": 0.001422999999999952, + "layer_top12_med": 0.11886, + "layer_top67_min": 7.299999999998974e-05, + "layer_top67_med": 0.018066000000000006, + "n_layers_gap67_below_0p05": 40, + "n_layers_gap12_below_0p05": 14 + }, + { + "prompt": "x1 = 3, x2 = 7, x3 = 11, x4 =", + "step": 5, + "golden": 53, + "tfarg": 49, + "gap": 3.466087, + "intop5": "no", + "layer_top12_min": 0.00025999999999992696, + "layer_top12_med": 0.15920900000000004, + "layer_top67_min": 2.6000000000081513e-05, + "layer_top67_med": 0.02266199999999996, + "n_layers_gap67_below_0p05": 39, + "n_layers_gap12_below_0p05": 12 + }, + { + "prompt": "x1 = 3, x2 = 7, x3 = 11, x4 =", + "step": 9, + "golden": 53, + "tfarg": 44, + "gap": 3.590601, + "intop5": "yes", + "layer_top12_min": 0.000276000000000054, + "layer_top12_med": 0.14829700000000012, + "layer_top67_min": 0.00012700000000000038, + "layer_top67_med": 0.02942200000000006, + "n_layers_gap67_below_0p05": 37, + "n_layers_gap12_below_0p05": 14 + }, + { + "prompt": "Translation to German: the weather is beautiful today", + "step": 4, + "golden": 33341, + "tfarg": 262, + "gap": 3.414822, + "intop5": "yes", + "layer_top12_min": 0.003141999999999978, + "layer_top12_med": 0.16148600000000002, + "layer_top67_min": 0.00012999999999996348, + "layer_top67_med": 0.023805000000000076, + "n_layers_gap67_below_0p05": 37, + "n_layers_gap12_below_0p05": 13 + }, + { + "prompt": "Translation to German: the weather is beautiful today", + "step": 5, + "golden": 4484, + "tfarg": 46, + "gap": 2.804047, + "intop5": "no", + "layer_top12_min": 0.0044839999999999325, + "layer_top12_med": 0.313788, + "layer_top67_min": 0.0003990000000000382, + "layer_top67_med": 0.02496900000000002, + "n_layers_gap67_below_0p05": 38, + "n_layers_gap12_below_0p05": 6 + }, + { + "prompt": "Translation to German: the weather is beautiful today", + "step": 10, + "golden": 5445, + "tfarg": 262, + "gap": 3.590538, + "intop5": "no", + "layer_top12_min": 0.003094000000000041, + "layer_top12_med": 0.30454800000000004, + "layer_top67_min": 0.0008709999999996221, + "layer_top67_med": 0.03434500000000007, + "n_layers_gap67_below_0p05": 30, + "n_layers_gap12_below_0p05": 6 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 0, + "golden": 30135, + "tfarg": 46, + "gap": 10.616974, + "intop5": "no", + "layer_top12_min": 0.0032340000000000146, + "layer_top12_med": 0.18133, + "layer_top67_min": 0.00021800000000005149, + "layer_top67_med": 0.01924999999999999, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 12 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 2, + "golden": 30134, + "tfarg": 518, + "gap": 4.228184, + "intop5": "no", + "layer_top12_min": 0.0009990000000001942, + "layer_top12_med": 0.21190799999999999, + "layer_top67_min": 0.0003449999999998177, + "layer_top67_med": 0.04044200000000009, + "n_layers_gap67_below_0p05": 31, + "n_layers_gap12_below_0p05": 8 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 4, + "golden": 1496, + "tfarg": 1914, + "gap": 1.356123, + "intop5": "no", + "layer_top12_min": 0.023004000000000024, + "layer_top12_med": 0.30889, + "layer_top67_min": 0.0003929999999998657, + "layer_top67_med": 0.031081000000000025, + "n_layers_gap67_below_0p05": 32, + "n_layers_gap12_below_0p05": 3 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 7, + "golden": 51092, + "tfarg": 2459, + "gap": 4.369095, + "intop5": "yes", + "layer_top12_min": 0.0042279999999998985, + "layer_top12_med": 0.31256, + "layer_top67_min": 0.0006299999999999084, + "layer_top67_med": 0.030577, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 9, + "golden": 10304, + "tfarg": 939, + "gap": 1.740644, + "intop5": "no", + "layer_top12_min": 0.002264000000000016, + "layer_top12_med": 0.266214, + "layer_top67_min": 9.900000000007125e-05, + "layer_top67_med": 0.04258800000000007, + "n_layers_gap67_below_0p05": 30, + "n_layers_gap12_below_0p05": 4 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 11, + "golden": 30135, + "tfarg": 420, + "gap": 4.770554, + "intop5": "no", + "layer_top12_min": 0.00031799999999992945, + "layer_top12_med": 0.247517, + "layer_top67_min": 0.00018899999999999473, + "layer_top67_med": 0.023880000000000012, + "n_layers_gap67_below_0p05": 35, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 13, + "golden": 30134, + "tfarg": 518, + "gap": 2.812277, + "intop5": "no", + "layer_top12_min": 0.0018609999999998905, + "layer_top12_med": 0.34269700000000003, + "layer_top67_min": 0.00038899999999975066, + "layer_top67_med": 0.02781100000000003, + "n_layers_gap67_below_0p05": 35, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 22, + "golden": 30135, + "tfarg": 420, + "gap": 7.391319, + "intop5": "no", + "layer_top12_min": 0.0021180000000000643, + "layer_top12_med": 0.273671, + "layer_top67_min": 0.0017829999999999999, + "layer_top67_med": 0.029463000000000017, + "n_layers_gap67_below_0p05": 31, + "n_layers_gap12_below_0p05": 11 + }, + { + "prompt": "Ein gutes Buch liest man am besten", + "step": 24, + "golden": 30134, + "tfarg": 518, + "gap": 3.878605, + "intop5": "no", + "layer_top12_min": 0.0019319999999996007, + "layer_top12_med": 0.214069, + "layer_top67_min": 0.00018199999999968242, + "layer_top67_med": 0.04209099999999999, + "n_layers_gap67_below_0p05": 29, + "n_layers_gap12_below_0p05": 7 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 0, + "golden": 12843, + "tfarg": 4241, + "gap": 4.733225, + "intop5": "no", + "layer_top12_min": 0.007288000000000072, + "layer_top12_med": 0.144122, + "layer_top67_min": 0.0009310000000000151, + "layer_top67_med": 0.017272999999999983, + "n_layers_gap67_below_0p05": 43, + "n_layers_gap12_below_0p05": 10 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 1, + "golden": 12843, + "tfarg": 4241, + "gap": 4.437166, + "intop5": "no", + "layer_top12_min": 0.00541400000000003, + "layer_top12_med": 0.21224299999999996, + "layer_top67_min": 0.0007369999999999877, + "layer_top67_med": 0.023745000000000016, + "n_layers_gap67_below_0p05": 34, + "n_layers_gap12_below_0p05": 10 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 2, + "golden": 12843, + "tfarg": 8445, + "gap": 3.724331, + "intop5": "no", + "layer_top12_min": 0.0016089999999999716, + "layer_top12_med": 0.13612800000000003, + "layer_top67_min": 0.00010799999999999699, + "layer_top67_med": 0.018015000000000003, + "n_layers_gap67_below_0p05": 38, + "n_layers_gap12_below_0p05": 9 + }, + { + "prompt": "counting: one two three four five six seven", + "step": 3, + "golden": 12843, + "tfarg": 8445, + "gap": 2.756411, + "intop5": "no", + "layer_top12_min": 0.0019770000000000065, + "layer_top12_med": 0.192143, + "layer_top67_min": 0.00025399999999997647, + "layer_top67_med": 0.024882000000000015, + "n_layers_gap67_below_0p05": 39, + "n_layers_gap12_below_0p05": 9 + } + ] +} \ No newline at end of file diff --git a/docs/bench-evidence/kolibri1-readj/parse_adjudication.py b/docs/bench-evidence/kolibri1-readj/parse_adjudication.py new file mode 100644 index 0000000000..f53d36ebb9 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/parse_adjudication.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +"""Derive the adjudication tables (off.json / on.json) from a full-gate log. + +Input: a `test_kolibri1_tt_b2ii --test-case='*device token gate*'` log with +VT_KOLIBRI1_TT_STAGE_DUMP=1 (the readjudication run's arm-on3.log / +arm-off.log class — full stage dump, both passes of the near-tie +instrument). Output: .json with the summary counters, the argmax-chain +counters, and one record per HARD teacher-forced flip carrying the +route_scores probe gaps at that forward. + +STEP LABELS: the gate's `HARD flip ... step d` and the flip records use +d = walk_step - input_len, which IS the decode position (the logits at +that forward predict generated_ids[d]); no offset applies here. Only the +SCRATCH dbg TF prompt legs print the pre-increment raw `TFstep` label — +see parse_tf_origin.py for the +1 mapping. + +The committed OFF.json / ON.json in this directory were produced by: + + python3 parse_adjudication.py /tmp/readj/arm-off.log OFF + python3 parse_adjudication.py /tmp/readj/arm-on3.log ON + +(the raw arm logs are 95.7 MB / 97.7 MB and are not committed). + +Usage: python3 parse_adjudication.py [repo/root] [outdir] +""" +import json +import os +import re +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = sys.argv[3] if len(sys.argv) > 3 else os.path.join(HERE, "..", "..", "..") + + +def forwards(path): + fwd, cur = [], None + dh2re = re.compile(r'\[STAGE\]\[TT\] L(\d+) dh2 sum=(\S+) sq=(\S+) hash=(\w+) max=(\S+)') + rsre = re.compile(r'\[STAGE\]\[TT\] L(\d+) route_scores tok=(\d+) top8=\[([^\]]*)\]') + with open(path, errors='replace') as f: + for ln, line in enumerate(f): + if '[STAGE][TT] L0 dh2 ' in line: + cur = {'line': ln, 'dh2': {}, 'scores': {}} + fwd.append(cur) + if cur is None: + continue + m = dh2re.search(line) + if m: + cur['dh2'][int(m.group(1))] = (m.group(2), m.group(3), m.group(4), m.group(5)) + m = rsre.search(line) + if m: + cur['scores'][int(m.group(1))] = [float(x) for x in m.group(3).split(',')] + return fwd + + +def load_prompts(): + d = json.load(open(os.path.join( + ROOT, 'tests', 'vllm', 'models', 'kolibri1_goldens.json'))) + return [(p['prompt'], len(p['input_ids']), len(p['generated_ids'])) + for p in d['prompts']] + + +def parse_results(path): + flips, summ, chain = [], None, None + for line in open(path, errors='replace'): + m = re.search(r'NEAR-TIE INSTRUMENT: (\d+) free-running divergences, ' + r'(\d+) teacher-forced flips \((\d+) near-ties, ' + r'(\d+) HARD\), worst teacher-forced gap (\S+) nats', line) + if m: + summ = tuple(int(m.group(i)) for i in (1, 2, 3, 4)) + (float(m.group(5)),) + m = re.search(r"REAL hard flip: '(.+)' step (\d+): golden (\d+) " + r"tf-argmax (\d+), gap (\S+) nats, in-top5=(\w+)", line) + if m: + flips.append({'prompt': m.group(1), 'step': int(m.group(2)), + 'golden': int(m.group(3)), 'tfarg': int(m.group(4)), + 'gap': float(m.group(5)), 'intop5': m.group(6)}) + m = re.search(r'ARGMAX CHAIN: (\d+)/(\d+) compared positions match.*' + r'flips (\d+) \(near-ties (\d+), hard (\d+)\)', line) + if m: + chain = tuple(int(m.group(i)) for i in (1, 2, 3, 4, 5)) + return flips, summ, chain + + +def main(path, arm): + ps = load_prompts() + fwd = forwards(path) + # doctest buffers MESSAGEs, so line position is useless; the empirical + # structure: every forward dumps 50 dh2 lines in the GATE phase (prefill + # + decode) and in the INSTRUMENT decode phases only. total = + # (74 + compared) + 2*(74+256) - 2*74. + flips, summ, chain = parse_results(path) + gate_forwards = 74 + (chain[1] if chain else 0) + if len(fwd) != gate_forwards + 8 * 64: + print(f'WARN forward count {len(fwd)} != {gate_forwards + 8*64}; ' + f'router-gap table unreliable') + print('SUMMARY divergences,tfflips,near-ties,hard,worst =', summ) + print('CHAIN matched/compared, flips, near-ties, hard =', chain) + out = [] + for fl in flips: + pi = next(i for i, p in enumerate(ps) if p[0] == fl['prompt']) + off = sum((q[1] + q[2]) * 2 for q in ps[:pi]) + fi = gate_forwards + pi * 64 + fl['step'] + if fi >= len(fwd): + print('MISSING forward for', fl) + continue + f = fwd[fi] + gaps12, gaps67 = [], [] + for Lay in sorted(f['scores']): + s = f['scores'][Lay] + if len(s) >= 7: + gaps12.append(s[0] - s[1]) + gaps67.append(s[5] - s[6]) + rec = dict(fl) + rec['layer_top12_min'] = min(gaps12) + rec['layer_top12_med'] = sorted(gaps12)[len(gaps12) // 2] + rec['layer_top67_min'] = min(gaps67) + rec['layer_top67_med'] = sorted(gaps67)[len(gaps67) // 2] + rec['n_layers_gap67_below_0p05'] = sum(1 for g in gaps67 if g < 0.05) + rec['n_layers_gap12_below_0p05'] = sum(1 for g in gaps12 if g < 0.05) + out.append(rec) + for r in out: + print(json.dumps(r)) + dest = sys.argv[4] if len(sys.argv) > 4 else HERE + json.dump({'arm': arm, 'summary': summ, 'chain': chain, 'hard_flips': out}, + open(os.path.join(dest, f'{arm}.json'), 'w'), indent=1) + + +if __name__ == '__main__': + main(sys.argv[1], sys.argv[2]) diff --git a/docs/bench-evidence/kolibri1-readj/parse_overlap.py b/docs/bench-evidence/kolibri1-readj/parse_overlap.py new file mode 100644 index 0000000000..2ee7873856 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/parse_overlap.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Derive the CPU-vs-TT top-6-of-384 expert-set overlap table (doc Q2). + +Input: the tf-p{1,5,4}.log(.gz) teacher-forced legs committed in this +directory (the `[STAGE] ... route ... ids=[...]` lines). Labels: rows span +the WHOLE one-token-per-forward walk (prompt tokens first, then the +teacher-forced chain), so a row index is a walk position, not a decode +position — the aggregate table is not step-labelled and needs no offset. +The single-step example quoted in the doc is the walk's first forward +(prompt token 1). + +Usage: python3 parse_overlap.py [repo/root] +""" +import gzip +import json +import os +import re +import statistics as st +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "..", "..") +r = re.compile(r"\[STAGE\]\[(CPU|TT)\] L(\d+) route tok=(\d+) logit_sum=(-?[0-9.eE+-]+) ids=\[([0-9,]+)\]") +LEGS = (("p1", 1), ("p5", 5), ("p4", 4)) + + +def open_log(which): + p = os.path.join(HERE, f"tf-{which}.log.gz") + if os.path.exists(p): + return gzip.open(p, "rt", errors="replace") + return open(os.path.join(HERE, f"tf-{which}.log"), errors="replace") + + +def main(): + ps = json.load(open(os.path.join( + ROOT, "tests", "vllm", "models", "kolibri1_goldens.json")))["prompts"] + for which, pi in LEGS: + rows = [] + for l in open_log(which): + m = r.search(l) + if m: + rows.append((m.group(1), int(m.group(2)), float(m.group(4)), + set(map(int, m.group(5).split(","))))) + cpu = [x for x in rows if x[0] == "CPU"] + tt = [x for x in rows if x[0] == "TT"] + ovl, noise, per = [], [], {} + for c, t in zip(cpu, tt): + o = len(c[3] & t[3]) + ovl.append(o) + per.setdefault(c[1], []).append(o) + noise.append(abs(c[2] - t[2]) / abs(c[2]) if c[2] else 0) + print(f"prompt {pi} ({ps[pi]['prompt'][:35]}): pairs={len(ovl)} " + f"mean_overlap={st.mean(ovl):.2f}/6 " + f"full_match={sum(1 for o in ovl if o == 6) / len(ovl) * 100:.1f}% " + f"median_logitsum_reldiff={st.median(noise) * 100:.2f}%") + print(" per-layer mean overlap:", + {L: round(st.mean(v), 1) for L, v in sorted(per.items())}) + + +if __name__ == "__main__": + main() diff --git a/docs/bench-evidence/kolibri1-readj/parse_tf_origin.py b/docs/bench-evidence/kolibri1-readj/parse_tf_origin.py new file mode 100644 index 0000000000..02900e68a4 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/parse_tf_origin.py @@ -0,0 +1,102 @@ +#!/usr/bin/env python3 +"""Derive the L0-drift-origin table from the SCRATCH dbg TF prompt logs. + +Input: the tf-p{1,5,4}.log(.gz) teacher-forced legs of +`test_kolibri1_tt_b2ii --test-case='*SCRATCH dbg TF prompt*'` +(VT_KOLIBRI1_TT_STAGE_DUMP=1), CPU and TT arms in one process, as committed +in this directory. Output: tf_origin.json, one row per teacher-forced +forward on the three instrumented prompts. + +STEP-LABEL MAPPING (the walk in test_kolibri1_tt_b2ii.cpp, SCRATCH dbg TF +prompt): walk iteration i feeds input_ids[i] for i < input_len and +generated_ids[i - input_len] afterwards, and its logits sit at context +position i, so they predict generated_ids[i - input_len + 1]. The log's +raw `TFstep` counter prints d = i - input_len BEFORE that +1, therefore + raw `TFstep N` == DECODE position N + 1, +and the last prompt token (raw TFstep -1) produces decode-position-0 +logits. This script labels every row with the decode position, matching +the gate's decode-step numbering (the `HARD flip ... step d` lines, where +d is already the decode position). + +Usage: python3 parse_tf_origin.py [path/to/repo/root] +Writes tf_origin.json next to this file. +""" +import gzip +import json +import os +import re +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = sys.argv[1] if len(sys.argv) > 1 else os.path.join(HERE, "..", "..", "..") +h = re.compile(r"\[STAGE\]\[(CPU|TT)\] L(\d+) (\S+) sum=(\S+) .*?hash=(\w+)") +LEG_STEPS = (("p1", 1, 25), ("p5", 5, 23), ("p4", 4, 10)) + + +def open_log(which): + for cand in (f"tf-{which}.log", f"tf-{which}.log.gz"): + p = os.path.join(HERE, cand) + if os.path.exists(p): + return gzip.open(p, "rt", errors="replace") if cand.endswith(".gz") \ + else open(p, errors="replace") + raise SystemExit(f"no tf-{which}.log(.gz) under {HERE}") + + +def main(): + ps = json.load(open(os.path.join( + ROOT, "tests", "vllm", "models", "kolibri1_goldens.json")))["prompts"] + res = {} + for which, pi, steps in LEG_STEPS: + plen = len(ps[pi]["input_ids"]) + segs, cur = [], None + for l in open_log(which): + m = h.search(l) + if not m: + continue + arm, L, tag, su, hs = m.groups() + if tag == "dh2" and L == "0": + cur = {"arm": arm, "stages": {}} + segs.append(cur) + if cur is None: + continue + cur["stages"].setdefault((int(L), tag), (hs, float(su))) + pairs = [(segs[i], segs[i + 1]) for i in range(0, len(segs) - 1, 2)] + rows = [] + # pair index fi == walk iteration i. Decode position 0 is the last + # prompt-token forward (fi = plen - 1, raw TFstep -1); every + # teacher-forced forward fi >= plen is decode position fi-plen+1. + for fi in range(plen - 1, min(plen + steps, len(pairs))): + c, t = pairs[fi] + if (0, "dh2") not in c["stages"] or (0, "dh2") not in t["stages"]: + continue + dh2c = c["stages"][(0, "dh2")] + dh2t = t["stages"][(0, "dh2")] + rel = abs(dh2c[1] - dh2t[1]) / abs(dh2c[1]) if dh2c[1] else 0 + idflips = 0 + lid = [] + for k in c["stages"]: + if k[1].startswith("moe_ex"): + if c["stages"][k][0] != t["stages"].get(k, ("x",))[0]: + idflips += 1 + lid.append(k[0]) + rows.append({ + "decode_position": fi - plen + 1, + "raw_tfstep_label": fi - plen, # the pre-fix log label + "l0_dh2_rel_drift_pct": round(rel * 100, 2), + "l0_dh2_hash_cpu": dh2c[0], + "l0_dh2_hash_tt": dh2t[0], + "expert_set_flips": idflips, + "first_flip_layers": lid[:6], + }) + res[f"p{pi}"] = rows + diverged = all(r["l0_dh2_hash_cpu"] != r["l0_dh2_hash_tt"] for r in rows) + print(f"prompt {pi} ({ps[pi]['prompt'][:35]}): " + f"{len(rows)} decode positions, L0 dh2 diverges at every " + f"position: {diverged}") + out = os.path.join(HERE, "tf_origin.json") + json.dump(res, open(out, "w"), indent=1) + print("wrote", out) + + +if __name__ == "__main__": + main() diff --git a/docs/bench-evidence/kolibri1-readj/shadow-recon-bisect-20261015.md b/docs/bench-evidence/kolibri1-readj/shadow-recon-bisect-20261015.md new file mode 100644 index 0000000000..dc8722a384 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/shadow-recon-bisect-20261015.md @@ -0,0 +1,72 @@ +# kolibri1 P150 host-free shadow-reconciliation lever — bisect outcome (2026-10-15) + +Worktree `row/tt-kolibri-shadow-recon` (base: origin/main 50d425bec, includes the +#3452 memo-eviction + shadow-sweep fixes). Gate: `test_kolibri1_tt_b2ii +--test-case='*device token gate*'`, `VT_KOLIBRI1_TT_STAGE_DUMP=1`, model +`/mnt/models/Aleph-Alpha/Kolibri-1`, clean `tt-smi -r 0` + cache clear per leg. +TF HARD counts from the near-tie instrument line (parse_adjudication.py). + +## The arm delta reproduces exactly on the current base + +| arm | gate HARD | argmax chain | TF instrument | +|---|---|---|---| +| OFF (`VT_TT_HOST_FREE_DECODE=0`) | 2 | 12/20, flips 8 (6 near, 2 hard) | 44 tf flips (27 near, **17 HARD**), worst 8.65 nats | +| ON (host-free default) | 5 | 3/11, flips 8 (3 near, 5 hard) | 69 tf flips (29 near, **40 HARD**), worst 10.62 nats | + +40/17 — identical to the #3454 readjudication. The #3452 UAF fixes did not +close the delta (expected: they fixed crashes, not drift). + +## Bisect matrix (forced-host knobs, ON arm) + +| leg | gate HARD | outcome | +|---|---|---| +| ON + `VT_TT_FORCE_HOST_ROPE=1` | 5 | flip set byte-identical to ON — rope ruled out | +| ON + `VT_TT_FORCE_HOST_RESIDUAL=1` | 7 | WORSE; host round-trips desync the device shadow (the recorded 1e37 collapse class, tenstorrent_ops.cpp:466-478) — NOT a clean probe, and it falsifies "host residual is the accurate path" | +| ON + both | 7 | identical to +fres | +| ON + `VT_TT_FORCE_HOST_PA=1` | 5 | chain identical (3/11, 5 hard) — PA ruled out | + +## Mechanism (stage-trace localization, forward 0 layer 0) + +The first diverging stage in the entire run is `L0 dhn` (the input_layernorm), +on BIT-IDENTICAL inputs: `L-2 emb` hash `dd76406a` and `gam` sum 858.502781 in +both arms. OFF (host f32 loop, `rmsnorm-host call=1`) outputs sum 9.846392; +ON (device add-norm, forced at T=1 for capture) outputs sum 9.765238 — ratio +0.9918, the 0.9899-class systematic bias. + +`VT_TT_NORM_DEBUG=1` on the ON arm shows the device norm's INPUTS are +bit-identical (`dev_x`/`norm_f32` sum −20.613504, `wf32` sum 858.502781) while +`dev_y` sums 9.777145 — so the deficit is in the tt-metal rms_norm kernel ON +THE MODEL'S ACTIVATION DISTRIBUTION. The committed micro +("SCRATCH dbg rmsnorm micro", random nd(0,0.5) inputs, same [1,2560], same eps, +patched substrate with `fp32_acc=true`) measures 0.9997 — clean. The bias is +input-distribution-dependent, not a config or staging difference. + +The composed f32 repair is still substrate-blocked IN-MODEL: re-confirmed today +(`VT_TT_RMSNORM_COMPOSED=1` gate leg) — `TT_THROW: Invalid subtile broadcast +type` from binary_ng (binary_ng_device_operation.cpp:224) on the +`repeat([rows,1] -> [rows,d])` row scale. + +## Verdict + +The shadow-reconciliation hypothesis is FALSIFIED as the lever: both arms share +WeightViewShadows/Bfp8WeightShadows and their #3452 sweeps, and the arm delta +originates upstream of any shadow traffic, at the first add-norm. The host-free +arm is noisier because it forces the device rms_norm at T=1 (capture requires +no host ops), and that kernel carries a ~0.7% systematic, input-dependent bias +on kolibri1's activations; 50 layers x 2-3 norm sites compounds it into +multi-nat router-logit shifts and 40 vs 17 teacher-forced hard flips. + +## Fix options priced + +1. **Substrate characterization (recommended next lever).** The micro is clean + where the model is not; find the in-model program difference (program-cache + provenance, compute-config, CB data format under the model's call pattern) + and either fix tt-metal or force the clean config in-model. Cheap to probe: + the micro passes the same shapes/eps, so a reproducing in-model input dump + (`emb` row + `gam`) fed through the micro isolates input-dependence vs + program provenance in one run. +2. **Composed f32 chain** — blocked by the binary_ng subtile broadcast refusal; + needs an upstream broadcast primitive or a matmul-shaped row scale. Priced + before (ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN), still refused on this stack. +3. **Host add-norm inside host-free** — falsified: the host round-trips desync + the device shadow (1e37 collapse, measured 7 HARD today). diff --git a/docs/bench-evidence/kolibri1-readj/tf-p1.log.gz b/docs/bench-evidence/kolibri1-readj/tf-p1.log.gz new file mode 100644 index 0000000000..7d60443e0c Binary files /dev/null and b/docs/bench-evidence/kolibri1-readj/tf-p1.log.gz differ diff --git a/docs/bench-evidence/kolibri1-readj/tf-p4.log.gz b/docs/bench-evidence/kolibri1-readj/tf-p4.log.gz new file mode 100644 index 0000000000..8c7c58577a Binary files /dev/null and b/docs/bench-evidence/kolibri1-readj/tf-p4.log.gz differ diff --git a/docs/bench-evidence/kolibri1-readj/tf-p5.log.gz b/docs/bench-evidence/kolibri1-readj/tf-p5.log.gz new file mode 100644 index 0000000000..55b4c1311e Binary files /dev/null and b/docs/bench-evidence/kolibri1-readj/tf-p5.log.gz differ diff --git a/docs/bench-evidence/kolibri1-readj/tf_origin.json b/docs/bench-evidence/kolibri1-readj/tf_origin.json new file mode 100644 index 0000000000..1b36391205 --- /dev/null +++ b/docs/bench-evidence/kolibri1-readj/tf_origin.json @@ -0,0 +1,984 @@ +{ + "p1": [ + { + "decode_position": 0, + "raw_tfstep_label": -1, + "l0_dh2_rel_drift_pct": 1.33, + "l0_dh2_hash_cpu": "38ad87cb", + "l0_dh2_hash_tt": "f88975f5", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 1, + "raw_tfstep_label": 0, + "l0_dh2_rel_drift_pct": 2.22, + "l0_dh2_hash_cpu": "98e63db6", + "l0_dh2_hash_tt": "02d1bb30", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 2, + "raw_tfstep_label": 1, + "l0_dh2_rel_drift_pct": 46.76, + "l0_dh2_hash_cpu": "4b86a5bf", + "l0_dh2_hash_tt": "88622861", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 3, + "raw_tfstep_label": 2, + "l0_dh2_rel_drift_pct": 2.82, + "l0_dh2_hash_cpu": "6dc61b80", + "l0_dh2_hash_tt": "541d151c", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 4, + "raw_tfstep_label": 3, + "l0_dh2_rel_drift_pct": 217.26, + "l0_dh2_hash_cpu": "282fc11f", + "l0_dh2_hash_tt": "00f06202", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 5, + "raw_tfstep_label": 4, + "l0_dh2_rel_drift_pct": 0.81, + "l0_dh2_hash_cpu": "e26a7973", + "l0_dh2_hash_tt": "1d187d1c", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 6, + "raw_tfstep_label": 5, + "l0_dh2_rel_drift_pct": 60.36, + "l0_dh2_hash_cpu": "a7d61449", + "l0_dh2_hash_tt": "bfb08d5b", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 7, + "raw_tfstep_label": 6, + "l0_dh2_rel_drift_pct": 0.8, + "l0_dh2_hash_cpu": "11beb0ef", + "l0_dh2_hash_tt": "0e8b81e4", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 8, + "raw_tfstep_label": 7, + "l0_dh2_rel_drift_pct": 46.29, + "l0_dh2_hash_cpu": "704c6a40", + "l0_dh2_hash_tt": "b42d0b7a", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 9, + "raw_tfstep_label": 8, + "l0_dh2_rel_drift_pct": 2.54, + "l0_dh2_hash_cpu": "27784426", + "l0_dh2_hash_tt": "9c45b744", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 10, + "raw_tfstep_label": 9, + "l0_dh2_rel_drift_pct": 9.32, + "l0_dh2_hash_cpu": "ad63b2b8", + "l0_dh2_hash_tt": "f21cf9dd", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 11, + "raw_tfstep_label": 10, + "l0_dh2_rel_drift_pct": 1.36, + "l0_dh2_hash_cpu": "7015725d", + "l0_dh2_hash_tt": "87f89a4d", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 12, + "raw_tfstep_label": 11, + "l0_dh2_rel_drift_pct": 1017.75, + "l0_dh2_hash_cpu": "235adb39", + "l0_dh2_hash_tt": "b07595e0", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 13, + "raw_tfstep_label": 12, + "l0_dh2_rel_drift_pct": 1.2, + "l0_dh2_hash_cpu": "2200b8e7", + "l0_dh2_hash_tt": "6893c4ec", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 14, + "raw_tfstep_label": 13, + "l0_dh2_rel_drift_pct": 12.29, + "l0_dh2_hash_cpu": "09607700", + "l0_dh2_hash_tt": "734be611", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 15, + "raw_tfstep_label": 14, + "l0_dh2_rel_drift_pct": 1.45, + "l0_dh2_hash_cpu": "557b7b69", + "l0_dh2_hash_tt": "acb9c914", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 16, + "raw_tfstep_label": 15, + "l0_dh2_rel_drift_pct": 21.73, + "l0_dh2_hash_cpu": "53448415", + "l0_dh2_hash_tt": "b6de2a8f", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 17, + "raw_tfstep_label": 16, + "l0_dh2_rel_drift_pct": 1.02, + "l0_dh2_hash_cpu": "6f57dd59", + "l0_dh2_hash_tt": "2881e7c7", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 18, + "raw_tfstep_label": 17, + "l0_dh2_rel_drift_pct": 978.46, + "l0_dh2_hash_cpu": "9a2d4b70", + "l0_dh2_hash_tt": "d4b8a0d7", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 19, + "raw_tfstep_label": 18, + "l0_dh2_rel_drift_pct": 2.2, + "l0_dh2_hash_cpu": "3ba4fc45", + "l0_dh2_hash_tt": "2d5ffbe3", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 20, + "raw_tfstep_label": 19, + "l0_dh2_rel_drift_pct": 488.6, + "l0_dh2_hash_cpu": "8a597aea", + "l0_dh2_hash_tt": "aa0a89a6", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 21, + "raw_tfstep_label": 20, + "l0_dh2_rel_drift_pct": 0.06, + "l0_dh2_hash_cpu": "49aaf45b", + "l0_dh2_hash_tt": "56e7e0da", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 22, + "raw_tfstep_label": 21, + "l0_dh2_rel_drift_pct": 41.6, + "l0_dh2_hash_cpu": "e9081bb9", + "l0_dh2_hash_tt": "2497cc0d", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 23, + "raw_tfstep_label": 22, + "l0_dh2_rel_drift_pct": 1.13, + "l0_dh2_hash_cpu": "e50d7fb3", + "l0_dh2_hash_tt": "b63727e2", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 24, + "raw_tfstep_label": 23, + "l0_dh2_rel_drift_pct": 81.68, + "l0_dh2_hash_cpu": "cea2903c", + "l0_dh2_hash_tt": "df568d20", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 25, + "raw_tfstep_label": 24, + "l0_dh2_rel_drift_pct": 2.49, + "l0_dh2_hash_cpu": "5b576faa", + "l0_dh2_hash_tt": "452d8ded", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + } + ], + "p5": [ + { + "decode_position": 0, + "raw_tfstep_label": -1, + "l0_dh2_rel_drift_pct": 0.73, + "l0_dh2_hash_cpu": "f2310d3a", + "l0_dh2_hash_tt": "a08df713", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 1, + "raw_tfstep_label": 0, + "l0_dh2_rel_drift_pct": 8.96, + "l0_dh2_hash_cpu": "9e25f5c5", + "l0_dh2_hash_tt": "a36e26f8", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 2, + "raw_tfstep_label": 1, + "l0_dh2_rel_drift_pct": 2.98, + "l0_dh2_hash_cpu": "150326d6", + "l0_dh2_hash_tt": "5c25f65c", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 3, + "raw_tfstep_label": 2, + "l0_dh2_rel_drift_pct": 486.62, + "l0_dh2_hash_cpu": "ed6a8b6d", + "l0_dh2_hash_tt": "6668bae4", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 4, + "raw_tfstep_label": 3, + "l0_dh2_rel_drift_pct": 2.88, + "l0_dh2_hash_cpu": "eafea96e", + "l0_dh2_hash_tt": "da60ab41", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 5, + "raw_tfstep_label": 4, + "l0_dh2_rel_drift_pct": 0.03, + "l0_dh2_hash_cpu": "afa70344", + "l0_dh2_hash_tt": "1473c619", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 6, + "raw_tfstep_label": 5, + "l0_dh2_rel_drift_pct": 0.38, + "l0_dh2_hash_cpu": "f76d7d4e", + "l0_dh2_hash_tt": "5a432ed4", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 7, + "raw_tfstep_label": 6, + "l0_dh2_rel_drift_pct": 9.23, + "l0_dh2_hash_cpu": "a956f61b", + "l0_dh2_hash_tt": "b99c4436", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 8, + "raw_tfstep_label": 7, + "l0_dh2_rel_drift_pct": 2.3, + "l0_dh2_hash_cpu": "ab670886", + "l0_dh2_hash_tt": "919f921e", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 9, + "raw_tfstep_label": 8, + "l0_dh2_rel_drift_pct": 18.31, + "l0_dh2_hash_cpu": "10947dfd", + "l0_dh2_hash_tt": "f0def415", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 10, + "raw_tfstep_label": 9, + "l0_dh2_rel_drift_pct": 2.18, + "l0_dh2_hash_cpu": "54a37c06", + "l0_dh2_hash_tt": "a1458151", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 11, + "raw_tfstep_label": 10, + "l0_dh2_rel_drift_pct": 5.07, + "l0_dh2_hash_cpu": "49a33ab0", + "l0_dh2_hash_tt": "647b98e9", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 12, + "raw_tfstep_label": 11, + "l0_dh2_rel_drift_pct": 1.09, + "l0_dh2_hash_cpu": "ecf81816", + "l0_dh2_hash_tt": "0d2d0cda", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 13, + "raw_tfstep_label": 12, + "l0_dh2_rel_drift_pct": 17.83, + "l0_dh2_hash_cpu": "47061c56", + "l0_dh2_hash_tt": "54d8a796", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 14, + "raw_tfstep_label": 13, + "l0_dh2_rel_drift_pct": 0.21, + "l0_dh2_hash_cpu": "e75868e3", + "l0_dh2_hash_tt": "a32b8461", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 15, + "raw_tfstep_label": 14, + "l0_dh2_rel_drift_pct": 32.22, + "l0_dh2_hash_cpu": "6128e322", + "l0_dh2_hash_tt": "797b0a87", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 16, + "raw_tfstep_label": 15, + "l0_dh2_rel_drift_pct": 3.11, + "l0_dh2_hash_cpu": "a25084a8", + "l0_dh2_hash_tt": "bf362d51", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 17, + "raw_tfstep_label": 16, + "l0_dh2_rel_drift_pct": 1.72, + "l0_dh2_hash_cpu": "73d8852c", + "l0_dh2_hash_tt": "637eb436", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 18, + "raw_tfstep_label": 17, + "l0_dh2_rel_drift_pct": 0.44, + "l0_dh2_hash_cpu": "5f30e403", + "l0_dh2_hash_tt": "9a049a82", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 19, + "raw_tfstep_label": 18, + "l0_dh2_rel_drift_pct": 14.88, + "l0_dh2_hash_cpu": "7f14e76d", + "l0_dh2_hash_tt": "4fd6443b", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 20, + "raw_tfstep_label": 19, + "l0_dh2_rel_drift_pct": 3.89, + "l0_dh2_hash_cpu": "dd50ecbe", + "l0_dh2_hash_tt": "a5453905", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 21, + "raw_tfstep_label": 20, + "l0_dh2_rel_drift_pct": 0.28, + "l0_dh2_hash_cpu": "615c762b", + "l0_dh2_hash_tt": "bb4af354", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 22, + "raw_tfstep_label": 21, + "l0_dh2_rel_drift_pct": 1.34, + "l0_dh2_hash_cpu": "9f999ab4", + "l0_dh2_hash_tt": "6935b5b4", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 23, + "raw_tfstep_label": 22, + "l0_dh2_rel_drift_pct": 5.41, + "l0_dh2_hash_cpu": "dd1712dd", + "l0_dh2_hash_tt": "9b4fbe0a", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + } + ], + "p4": [ + { + "decode_position": 0, + "raw_tfstep_label": -1, + "l0_dh2_rel_drift_pct": 0.75, + "l0_dh2_hash_cpu": "2e08db14", + "l0_dh2_hash_tt": "da93e43c", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 1, + "raw_tfstep_label": 0, + "l0_dh2_rel_drift_pct": 8.28, + "l0_dh2_hash_cpu": "2216ce4b", + "l0_dh2_hash_tt": "b2795751", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 2, + "raw_tfstep_label": 1, + "l0_dh2_rel_drift_pct": 6.43, + "l0_dh2_hash_cpu": "5186595f", + "l0_dh2_hash_tt": "b552129d", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 3, + "raw_tfstep_label": 2, + "l0_dh2_rel_drift_pct": 0.64, + "l0_dh2_hash_cpu": "f68b156b", + "l0_dh2_hash_tt": "88251083", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 4, + "raw_tfstep_label": 3, + "l0_dh2_rel_drift_pct": 5.81, + "l0_dh2_hash_cpu": "ba742061", + "l0_dh2_hash_tt": "ef6a9ca8", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 5, + "raw_tfstep_label": 4, + "l0_dh2_rel_drift_pct": 1.55, + "l0_dh2_hash_cpu": "16685359", + "l0_dh2_hash_tt": "4c7e409c", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 6, + "raw_tfstep_label": 5, + "l0_dh2_rel_drift_pct": 2.59, + "l0_dh2_hash_cpu": "7b1cee33", + "l0_dh2_hash_tt": "b8c2c251", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 7, + "raw_tfstep_label": 6, + "l0_dh2_rel_drift_pct": 2.02, + "l0_dh2_hash_cpu": "06fc7875", + "l0_dh2_hash_tt": "7dbda6d6", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 8, + "raw_tfstep_label": 7, + "l0_dh2_rel_drift_pct": 13.38, + "l0_dh2_hash_cpu": "eba3f8d2", + "l0_dh2_hash_tt": "0213d0c0", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 9, + "raw_tfstep_label": 8, + "l0_dh2_rel_drift_pct": 2.67, + "l0_dh2_hash_cpu": "54b8f545", + "l0_dh2_hash_tt": "e7023df7", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + }, + { + "decode_position": 10, + "raw_tfstep_label": 9, + "l0_dh2_rel_drift_pct": 3.11, + "l0_dh2_hash_cpu": "47b8364f", + "l0_dh2_hash_tt": "5adee953", + "expert_set_flips": 300, + "first_flip_layers": [ + 0, + 0, + 0, + 0, + 0, + 0 + ] + } + ] +} \ No newline at end of file diff --git a/docs/bench-evidence/kolibri1-tt-readjudication-20261012.md b/docs/bench-evidence/kolibri1-tt-readjudication-20261012.md new file mode 100644 index 0000000000..dfd4436dcc --- /dev/null +++ b/docs/bench-evidence/kolibri1-tt-readjudication-20261012.md @@ -0,0 +1,200 @@ +# kolibri1 TT P150: teacher-forced readjudication on the corrected instrument (2026-10-12) + +Issue: `ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN`. Branch: `row/tt-kolibri-readj` +(off `row/tt-b2ii-index` @ `2c4efce88`, which fixed the teacher-forced walk's +shifted chain — the walk now feeds `generated_ids[d-1]`, not +`generated_ids[step-1]`). + +## Why this run exists + +The prior hard-flip classification ("router-gain amplification of upstream +bf16 drift", recorded 2026-10-10 on the `:76`-era instrument) was computed +against a SHIFTED prefix: the walk fed the golden token one position early, so +every teacher-forced logit row answered a different question than the gate's +decode. All teacher-forced flip counts from that instrument era are INVALID as +per-step evidence. This run re-derives them on the corrected walk. +Determinism on this base is verified (3 identical chains per arm), so +cross-run comparison is valid. + +## Runs + +- Host: P150 (thor), `tt-smi -r 0` + clean `tt-metal-cache` / shm before each + arm; substrate = `/tmp/umdtrial-install` (carries the rms_norm fp32-accum + patch and the port fixes); model `/mnt/models/Aleph-Alpha/Kolibri-1`. +- Binary: `test_kolibri1_tt_b2ii --test-case='*device token gate*'`, + commit `2c4efce88` + the readjudication RouteDump probe (see below). +- `SKIPPED`: 0 in every leg; every device leg ran. +- Instrumentation: `VT_KOLIBRI1_TT_STAGE_DUMP=1` (existing per-stage bf16 + bit-hash probes, both arms, plus the CPU-vs-TT `SCRATCH dbg TF prompt` + teacher-forced leg) and a NEW scratch probe in the TT `RouteDump` + (`kolibri1_tt_forward.cpp`) printing the sorted top-8 BIASED router + selection scores per layer per token (`route_scores`), so the per-layer + top1-top2 / top6-top7 gaps are measurable against the incoming drift. + +## Step-label mapping (SCRATCH dbg TF walk vs the gate's decode steps) + +The `SCRATCH dbg TF prompt` legs print a raw `TFstep` counter that is NOT +the gate's decode position. Walk iteration `i` feeds `input_ids[i]` for +`i < input_len` and `generated_ids[i - input_len]` afterwards; its logits +sit at context position `i` and therefore predict +`generated_ids[i - input_len + 1]`. The print emitted `d = i - input_len` +BEFORE that +1, so: + + raw `TFstep N` == DECODE position N + 1 + +The last prompt token (`TFstep -1`) produces decode-position-0 logits, and +the first teacher-forced feed (`TFstep 0`) produces decode-position-1 +logits. The gate's own `HARD flip ... step d` lines use +`d = walk_step - input_len`, which IS the decode position (that forward +predicts `generated_ids[d]`) — no offset there. The test now prints both +(`TFstep N (decode position N+1)`; verified on device with a short +prompt-4 leg), and every step number below that comes from the SCRATCH +walk is given as a decode position. The committed parser +`kolibri1-readj/parse_tf_origin.py` derives `tf_origin.json` under the +corrected labels. + +## Adjudication table (corrected instrument, this run) + +| Arm | free divergences | tf flips | near-ties | HARD | worst tf gap | chain (matched/compared) | +|---|---|---|---|---|---|---| +| host-free ON | 170 | 69 | 29 | **40** | 10.617 nats | 3/11 (8 flips: 3 near-tie, 5 hard) | +| host-free OFF | 158 | 44 | 27 | **17** | 8.647 nats | 12/20 (8 flips: 6 near-tie, 2 hard) | + +Both arms reproduce the recorded #3453-era corrected-instrument numbers +exactly (ON 69/29/40/10.6, OFF 44/27/17/8.6). The residual hard-flip +population is REAL, not an instrument artifact (the zero-stop condition did +not fire). + +## Q1: where does the drift originate (corrected walk) + +From `SCRATCH dbg TF prompt` (CPU arm and TT arm in one process, +teacher-forced on the golden chain, per-stage bf16 bit hashes; prompts 1 +'der Mond…' 25 steps, 5 'Translation to German…' 23 steps, 4 'x1 = 3…' 10 +steps): + +- The first hash-bearing stage of every decode forward is `L0 dh2` (the L0 + MoE input), and it diverges at EVERY decode position from 0 on all three + prompts (decode positions 0–25, 0–23, 0–10 respectively, including + position 0 — the last-prompt-token forward the pre-fix logs labelled + `TFstep -1`; see `kolibri1-readj/tf_origin.json`, where every row's + `l0_dh2_hash_cpu != l0_dh2_hash_tt`) — before it, the L0 embed/attention + stages match the CPU arm. +- L0 `dh2` relative magnitude drift vs CPU: typically 0.03%–2.5% per step + (outlier steps to ~10x when the sum passes near zero), consistent with the + 1-bf16-ULP-class L0 input-norm origin the 2026-10-10 record established + element-level; the corrected walk does NOT move the origin. The drift + origin remains the L0 input norm / device attention chain at 1 bf16 ULP, + present from decode step 0. + +## Q2: does the router-selection-flip mechanism hold? YES. + +Per-layer CPU-vs-TT top-6-of-384 selected-expert overlap under the corrected +teacher-forced walk (1600 layer-steps per prompt): + +| Prompt | mean overlap (of 6) | full-agreement layer-steps | median router logit_sum rel diff | +|---|---|---|---| +| 'der Mond…' | 3.66 | 6.0% | 6.65% | +| 'Translation to German…' | 3.71 | 7.1% | 7.05% | +| 'x1 = 3…' | 3.05 | 6.4% | 9.65% | + +The selection disagreement is present AT EVERY LAYER including L0 (per-layer +mean overlap 1.4–5.7 of 6; e.g. the walk's first forward, prompt token 1, +at L0: CPU {47,284,19,228,359,141} vs TT {359,47,284,246,141,144}). The mechanism holds: a ~0.1–2.5% continuous input +drift through the unit-norm-router GEMM (K=2560) injects percent-level +absolute noise into O(1)-scale biased selection scores, and the top-6 +boundary flips 2–3 of 6 experts at nearly every layer-step. + +The gap structure pins WHERE the selection is fragile (route_scores probe, +OFF arm, teacher-forced pass at each hard-flip step; 50 layers): + +- top1–top2 gap: median across layers 0.12–0.48 (robust — rank-1 is almost + never the flip site); +- top6–top7 gap: median across layers 0.017–0.050, min 2.6e-5–1.8e-3; at + every hard-flip step, 26–41 of 50 layers (OFF arm) and 27–44 of 50 layers + (ON arm) hold a top6–top7 gap BELOW 0.05 — + the incoming per-layer noise (percent-level absolute on O(1) scores) + exceeds the rank-6/7 margin at most layers, so a boundary flip is + expected at essentially every layer, exactly as observed in the overlap + table. Top-1 selection is not the defect; the top-6-of-384 boundary is. + +## Q3: hard-flip distribution (band-2.5 standard input) + +Per-prompt / per-step-position distribution of the HARD teacher-forced flips: + +OFF arm (17 hard): 'der Mond…' steps 0,1 (opening burst); 'Australia' step 18 +(late isolated); 'Wissen…' steps 1,2,3 (burst), 14, 17 (late isolated); +'Translation…' steps 5,10,22 (spread); 'Ein gutes…' step 7; 'counting…' +steps 0–4 (contiguous burst). + +ON arm (40 hard): 'der Mond…' 0,1; 'Australia' 0,11,14,15,16,18,21,26 +(spread across the whole chain); 'Wissen…' 1,3,4,6,7,11,14,16,17,21 (spread); +'x1 = 3…' 2,4,5,9; 'Translation…' 4,5,10; 'Ein gutes…' +0,2,4,7,9,11,13,22,24 (spread); 'counting…' 0,1,2,3. + +First-divergent-layer for every hard flip: on the three instrumented +diverging prompts the corrected walk's first divergent stage is `L0 dh2` at +every decode position (0–25, 0–23, 0–10 respectively; corrected labels, see +the mapping section and `kolibri1-readj/tf_origin.json`) — there is no +late-switching localized defect layer; the drift is resident from step 0 and +the flip step is decided by the LM-head margin at that step, not by where the +drift begins. + +Concentration verdict: the flips SPREAD across prompts and decode positions +— opening bursts (decode positions 0–4) on some prompts and isolated +mid/late-chain flips (positions 14–26) on others, on 7 of 8 prompts, in +BOTH arms. What the evidence supports is that this spread is CONSISTENT +WITH the shared L0-origin drift amplification: the drift is resident at +every decode position from 0 (Q1), and the top-6-of-384 selection boundary +is sub-0.05-nat at most layers at every hard flip (Q2), so a boundary flip +is expected wherever the LM-head margin is thin, not at a localised site. +This does NOT prove that no single chain-wide change clears the residual: +a chain-wide f32 fix of the device attention/norm chain (lever 2 below) +addresses the shared origin itself, and the spread does not rule out that +such a fix clears most of it. What the spread DOES rule out is a +single-layer or single-position patch. Under the row's band-gate question +this is the STRUCTURAL class (drift x router gain), and the arm delta +(17 vs 40) shows the host-free shadow reconciliation remains the largest +single lever. + +## Recommended gate-standard input + +Keep the 2.5-nat / top-5 adjudication. Report the P150 band as STRUCTURAL +(drift x router gain): teacher-forced hard flips 17 (OFF) / 40 (ON) spread +over prompts and decode positions, consistent with a position-0-resident +1-ULP-class norm/attention-chain drift amplified by the router. The spread +is evidence against single-site patches, not against a chain-wide repair — +lever 2 targets the origin itself and remains live. The next levers, in +order: (1) host-free shadow reconciliation (halves the hard count), (2) +f32-enabling the device attention/norm chain (removes the origin), (3) any +future router-margin widening is a model change, not a backend repair — out +of scope for this row. + +## Raw evidence + +Committed, in `docs/bench-evidence/kolibri1-readj/` (a sibling directory of +this file): + +- `OFF.json`, `ON.json` — the parsed adjudication tables (summary, argmax + chain, and per-hard-flip route_scores gap records). +- `tf_origin.json` — the L0 drift-origin table under CORRECTED decode + position labels (see the mapping section; includes decode position 0). +- `tf-p{1,5,4}.log.gz` — the raw CPU-vs-TT teacher-forced stage-dump legs + (9.3 MB each uncompressed, 2.2 MB gzipped). +- The parsers that turn the logs into every table above: + `parse_adjudication.py` (`OFF`/`ON.json`), `parse_tf_origin.py` + (`tf_origin.json`), `parse_overlap.py` (the Q2 overlap table). Rerun + from the repo root: + + ```sh + python3 docs/bench-evidence/kolibri1-readj/parse_tf_origin.py + python3 docs/bench-evidence/kolibri1-readj/parse_overlap.py + python3 docs/bench-evidence/kolibri1-readj/parse_adjudication.py \ + OFF + ``` + +Not committed — the full-gate arm logs are too large: `arm-off.log` +95.7 MB, `arm-on3.log` 97.7 MB (the ON leg that reproduces the clean ON +counts exactly), `arm-on.log` 22.0 MB, `arm-on2.log` 3.4 MB. They remain +under `/tmp/readj/` on the measurement host and are regenerable with the +run recipe in `## Runs` plus the `parse_adjudication.py` invocations in +its docstring. diff --git a/src/vllm/model_executor/models/kolibri1_tt_forward.cpp b/src/vllm/model_executor/models/kolibri1_tt_forward.cpp index 9d1e9bf794..a0be0f1cfc 100644 --- a/src/vllm/model_executor/models/kolibri1_tt_forward.cpp +++ b/src/vllm/model_executor/models/kolibri1_tt_forward.cpp @@ -96,7 +96,7 @@ bool StageDumpOn() { void RouteDump(int64_t layer, const std::vector& logits, const Kolibri1HostRouting& route, int64_t t, int64_t e, - int64_t top_k); + int64_t top_k, const std::vector& bias); void StageDump(const char* tag, int64_t layer, DBuf& buf, Dev d) { if (!StageDumpOn()) return; @@ -150,9 +150,34 @@ float Kolibri1DbgToBf16(float v) { void RouteDump(int64_t layer, const std::vector& logits, const Kolibri1HostRouting& route, int64_t t, int64_t e, - int64_t top_k) { + int64_t top_k, const std::vector& bias) { if (!StageDumpOn()) return; for (int64_t i = 0; i < t; ++i) { + // READJUDICATION INSTRUMENT (ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN): + // the sorted top-8 BIASED selection scores (logits+bias, the exact key + // SigmoidLogitAddRouting selects on) so the per-layer router gaps + // (top1-top2, top6-top7) can be compared with the incoming drift. + { + std::vector sc(static_cast(e)); + for (int64_t jj = 0; jj < e; ++jj) + sc[static_cast(jj)] = + logits[static_cast(i * e + jj)] + + bias[static_cast(jj)]; + std::partial_sort(sc.begin(), sc.begin() + (e < 8 ? e : 8), sc.end(), + std::greater()); + std::string ss; + const int64_t nk = e < 8 ? e : 8; + for (int64_t kk = 0; kk < nk; ++kk) { + char b[32]; + std::snprintf(b, sizeof b, "%s%.6f", kk ? "," : "", + sc[static_cast(kk)]); + ss += b; + } + std::fprintf(stderr, + "[STAGE][TT] L%lld route_scores tok=%lld top8=[%s]\n", + static_cast(layer), static_cast(i), + ss.c_str()); + } double ls = 0; for (int64_t j = 0; j < e; ++j) ls += logits[static_cast(i * e + j)]; @@ -172,6 +197,31 @@ void RouteDump(int64_t layer, const std::vector& logits, "w=[%s]\n", static_cast(layer), static_cast(i), ls, ids.c_str(), ws.c_str()); + // READJUDICATION INSTRUMENT (ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN): + // the sorted top-8 BIASED selection scores (logits+bias, the exact key + // SigmoidLogitAddRouting selects on) so the per-layer router gaps + // (top1-top2, top6-top7) can be compared with the incoming drift. + { + std::vector sc(static_cast(e)); + for (int64_t j = 0; j < e; ++j) + sc[static_cast(j)] = + logits[static_cast(i * e + j)] + + bias[static_cast(j)]; + std::partial_sort(sc.begin(), sc.begin() + (e < 8 ? e : 8), sc.end(), + std::greater()); + std::string ss; + const int64_t nk = e < 8 ? e : 8; + for (int64_t k = 0; k < nk; ++k) { + char b[32]; + std::snprintf(b, sizeof b, "%s%.6f", k ? "," : "", + sc[static_cast(k)]); + ss += b; + } + std::fprintf(stderr, + "[STAGE][TT] L%lld route_scores tok=%lld top8=[%s]\n", + static_cast(layer), static_cast(i), + ss.c_str()); + } } if (g_kolibri1_dbg_route_layer == layer && g_kolibri1_dbg_route_logits.size() == logits.size()) { @@ -596,7 +646,7 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p, } } if (!kDeviceRouter) { - RouteDump(layer, logits, route, t, e, top_k); + RouteDump(layer, logits, route, t, e, top_k, bias); } // SLICE ii: the routed tier. Without a streaming context (the diff --git a/tests/vllm/models/test_kolibri1_tt_b2ii.cpp b/tests/vllm/models/test_kolibri1_tt_b2ii.cpp index cd183fcd5b..568c0f860e 100644 --- a/tests/vllm/models/test_kolibri1_tt_b2ii.cpp +++ b/tests/vllm/models/test_kolibri1_tt_b2ii.cpp @@ -1013,7 +1013,16 @@ TEST_CASE("SCRATCH dbg TF prompt") { int32_t at = static_cast( std::max_element(lt.begin(), lt.end()) - lt.begin()); const int d = i - static_cast(gp.input_ids.size()); - MESSAGE("TFstep " << d << " tok " << seq.back() << ": max_abs=" << ma + // LABEL MAPPING: this forward's logits sit at context position i and + // predict generated_ids[d + 1], i.e. DECODE position d + 1. The last + // prompt token (i == input_len - 1, d == -1) therefore produces + // decode-position-0 logits, and the first teacher-forced feed + // (generated_ids[0]) produces decode-position-1 logits. Print the + // decode position beside the raw step counter so the log lines line + // up with the gate's decode-step numbering (see + // docs/bench-evidence/kolibri1-readjudication-20261012.md). + MESSAGE("TFstep " << d << " (decode position " << (d + 1) + << ") tok " << seq.back() << ": max_abs=" << ma << " at " << amax << " argmax cpu=" << ac << " tt=" << at << " logit cpu=" << lc[static_cast(ac)] << " tt="