kolibri1 TT: re-attribute the host-free arm delta — the device add-norm kernel bias, not the shadow reconciliation

.agents/issues/BACKEND-TENSTORRENT/ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY.md

@@ -0,0 +1,46 @@
+ID: ISSUE-LOCAL-01M4YVEP000ENCRECVB0S6FDDY
+Title: kolibri1 TT host-free ON arm produces more teacher-forced hard flips than OFF (40 vs 17)
+Row: BACKEND-TENSTORRENT
+State: OPEN
+Kind: bug
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-15
+Updated: 2026-10-15
+Closed: -
+
+## Problem
+
+The #3454 readjudication attributes the kolibri1 P150 hard-flip count to device bf16 1-ULP drift x router gain, with the ARM DELTA (host-free ON 40 teacher-forced HARD flips vs OFF 17) left as the largest single lever: the host-free path's shadow handling (WeightViewShadows/Bfp8WeightShadows reconciliation) is suspected to introduce more drift than the eager path. The #3452 UAF fixes (memo eviction order + shadow sweeps at UnregisterHostBuffer/MarkScratchAcquired) landed after that measurement; this issue owns re-measuring the delta on the current base, bisecting which host-free-specific path carries the extra drift (rope/residual/PA force-host knobs, expert-weight shadow residency), and fixing it if it is a defect (stale shadow serve) or pricing it precisely if it is inherent precision difference.
+
+## Resolution
+
+- 2026-10-15 (row/tt-kolibri-shadow-recon): the delta REPRODUCES exactly on the
+  current base (ON 40 / OFF 17 TF HARD, gate 5/2). The shadow-reconciliation
+  hypothesis is FALSIFIED: both arms share the shadow maps and the #3452
+  sweeps, and the first diverging stage in the run is `L0 dhn` — the
+  input_layernorm — on BIT-IDENTICAL inputs (emb hash `dd76406a`, gam sum
+  858.502781 both arms): host f32 loop 9.846392 vs device 9.765238 (ratio
+  0.9918). `VT_TT_NORM_DEBUG` proves the device norm's inputs are bit-identical
+  (`dev_x`/`norm_f32` −20.613504, `wf32` 858.502781) while `dev_y` reads
+  9.777145: the deficit is in the tt-metal rms_norm kernel ON THE MODEL'S
+  ACTIVATION DISTRIBUTION — the committed micro (random inputs, same shape/eps,
+  patched `fp32_acc=true` substrate, verified installed) measures 0.9997 clean.
+  Bisect: `VT_TT_FORCE_HOST_ROPE` byte-identical flip set (ruled out);
+  `VT_TT_FORCE_HOST_PA` identical chain (ruled out); `VT_TT_FORCE_HOST_RESIDUAL`
+  reads 7 HARD — WORSE — because the host round-trips desync the device shadow
+  (the recorded 1e37 collapse class), so it is not a clean probe and the "host
+  residual is more accurate" reading of the pre-#3452 bisect is dead.
+  `VT_TT_RMSNORM_COMPOSED=1` re-confirmed substrate-blocked in-model
+  (`Invalid subtile broadcast type`, binary_ng_device_operation.cpp:224).
+  MECHANISM: host-free forces the device rms_norm at T=1 for capture; that
+  kernel carries a ~0.7% systematic input-dependent bias on kolibri1
+  activations; 50 layers x 2-3 norm sites compounds into multi-nat router-logit
+  shifts and the 40-vs-17 arm delta. Evidence:
+  docs/bench-evidence/kolibri1-readj/shadow-recon-bisect-20261015.md.
+  NEXT LEVER: substrate characterization — feed the in-model emb/gam row
+  through the micro to split input-dependence from program provenance
+  (program-cache/config under the model call pattern), then fix tt-metal or
+  force the clean config in-model. STAYS OPEN (the delta is not fixed; it is
+  re-attributed).

.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.md

@@ -195,6 +195,34 @@ B2b-i landed the dense-resident device forward with the routed-expert tier delib
   driver is the upstream drift class. CLASSIFICATION: selection flips
   DRIVEN by continuous bf16 device-arm precision drift through the
   attention/norm chain; per the row's adjudication rule this is an
-  upstream adjudication decision (f32-enabling the device attention/norm
+  upstream adjudication decision (  f32-enabling the device attention/norm
   chain or accepting the band), not a local patch. ISSUE STAYS OPEN on the
   device gates and the OOM debt.
+
+- 2026-10-12 (branch row/tt-kolibri-readj, off row/tt-b2ii-index
+  @ 2c4efce88): the teacher-forced adjudication RE-RUN on the CORRECTED
+  instrument — the prior classification was computed on a SHIFTED chain
+  (generated_ids[step-1] fed instead of generated_ids[d-1]), so every
+  prior per-step teacher-forced number is invalid as step evidence. The
+  corrected walk reproduces the #3453-era corrected counts exactly (ON 69
+  tf flips: 29 near-tie, 40 HARD, worst 10.617 nats; OFF 44: 27 near-tie,
+  17 HARD, worst 8.647) — the residual is REAL, not an instrument
+  artifact. Drift origin UNCHANGED: L0 dh2 (the L0 MoE input) is the first
+  divergent stage at EVERY decode step from step 0 on all three diverging
+  prompts (SCRATCH dbg TF prompt, CPU-vs-TT bit hashes), 0.03-2.5% rel per
+  step — the 1-bf16-ULP-class L0 input-norm origin stands. Router-flip
+  mechanism CONFIRMED: per-layer CPU-vs-TT top-6-of-384 selected-expert
+  overlap under the corrected walk averages 3.0-3.7 of 6 (full agreement
+  only 6-7% of layer-steps) at EVERY layer including L0, median router
+  logit_sum rel diff 6.7-9.7%; the route_scores probe shows top1-top2
+  margins are robust (median 0.13-0.48) while top6-top7 margins sit below
+  0.05 nats at 26-41 of 50 layers at every hard-flip step — the incoming
+  noise exceeds the rank-6/7 boundary margin at most layers. Distribution:
+  hard flips SPREAD, not concentrated — 7 of 8 prompts, opening bursts
+  (steps 0-4) on some prompts and isolated late flips (steps 14-26) on
+  others, both arms. Gate-standard input: the P150 band-2.5 residual is
+  STRUCTURAL (continuous step-0-resident drift x router gain onto a
+  sub-0.05-nat selection boundary), not a single-site repair; levers in
+  order: host-free shadow reconciliation (17 vs 40 hard shows it halves
+  the count), then f32 device attention/norm chain. Evidence:
+  docs/bench-evidence/kolibri1-tt-readjudication-20261012.md.

docs/bench-evidence/kolibri1-readj/OFF.json

@@ -0,0 +1,257 @@
+{
+ "arm": "OFF",
+ "summary": [
+  158,
+  44,
+  27,
+  17,
+  8.64696
+ ],
+ "chain": [
+  12,
+  20,
+  8,
+  6,
+  2
+ ],
+ "hard_flips": [
+  {
+   "prompt": "der Mond scheint hell \u00fcber den Bergen",
+   "step": 0,
+   "golden": 948,
+   "tfarg": 46,
+   "gap": 3.181908,
+   "intop5": "no",
+   "layer_top12_min": 0.0016739999999999533,
+   "layer_top12_med": 0.24162299999999998,
+   "layer_top67_min": 0.0002620000000002065,
+   "layer_top67_med": 0.028259000000000034,
+   "n_layers_gap67_below_0p05": 34,
+   "n_layers_gap12_below_0p05": 6
+  },
+  {
+   "prompt": "der Mond scheint hell \u00fcber den Bergen",
+   "step": 1,
+   "golden": 46252,
+   "tfarg": 3389,
+   "gap": 2.273598,
+   "intop5": "no",
+   "layer_top12_min": 0.0012659999999999894,
+   "layer_top12_med": 0.167883,
+   "layer_top67_min": 0.0007520000000000009,
+   "layer_top67_med": 0.020367000000000024,
+   "n_layers_gap67_below_0p05": 40,
+   "n_layers_gap12_below_0p05": 12
+  },
+  {
+   "prompt": "The capital of Australia is",
+   "step": 18,
+   "golden": 75361,
+   "tfarg": 543,
+   "gap": 3.553511,
+   "intop5": "yes",
+   "layer_top12_min": 0.016432000000000002,
+   "layer_top12_med": 0.484023,
+   "layer_top67_min": 0.0007150000000000212,
+   "layer_top67_med": 0.04172799999999999,
+   "n_layers_gap67_below_0p05": 28,
+   "n_layers_gap12_below_0p05": 5
+  },
+  {
+   "prompt": "Wissen ist Macht, aber Zeit ist Geld",
+   "step": 1,
+   "golden": 936,
+   "tfarg": 420,
+   "gap": 1.92732,
+   "intop5": "no",
+   "layer_top12_min": 0.003607000000000027,
+   "layer_top12_med": 0.21200000000000002,
+   "layer_top67_min": 0.00011499999999997623,
+   "layer_top67_med": 0.04392200000000002,
+   "n_layers_gap67_below_0p05": 34,
+   "n_layers_gap12_below_0p05": 9
+  },
+  {
+   "prompt": "Wissen ist Macht, aber Zeit ist Geld",
+   "step": 2,
+   "golden": 2973,
+   "tfarg": 936,
+   "gap": 2.048908,
+   "intop5": "no",
+   "layer_top12_min": 0.0017440000000000788,
+   "layer_top12_med": 0.316585,
+   "layer_top67_min": 0.0001020000000000465,
+   "layer_top67_med": 0.044141999999999904,
+   "n_layers_gap67_below_0p05": 30,
+   "n_layers_gap12_below_0p05": 10
+  },
+  {
+   "prompt": "Wissen ist Macht, aber Zeit ist Geld",
+   "step": 3,
+   "golden": 476,
+   "tfarg": 735,
+   "gap": 3.018167,
+   "intop5": "yes",
+   "layer_top12_min": 0.008242,
+   "layer_top12_med": 0.19427700000000003,
+   "layer_top67_min": 0.0009399999999999964,
+   "layer_top67_med": 0.02517999999999998,
+   "n_layers_gap67_below_0p05": 41,
+   "n_layers_gap12_below_0p05": 8
+  },
+  {
+   "prompt": "Wissen ist Macht, aber Zeit ist Geld",
+   "step": 14,
+   "golden": 18331,
+   "tfarg": 502,
+   "gap": 7.306021,
+   "intop5": "no",
+   "layer_top12_min": 0.0006349999999999412,
+   "layer_top12_med": 0.34341299999999997,
+   "layer_top67_min": 0.0001620000000001065,
+   "layer_top67_med": 0.034442000000000084,
+   "n_layers_gap67_below_0p05": 34,
+   "n_layers_gap12_below_0p05": 13
+  },
+  {
+   "prompt": "Wissen ist Macht, aber Zeit ist Geld",
+   "step": 17,
+   "golden": 936,
+   "tfarg": 286,
+   "gap": 3.616066,
+   "intop5": "yes",
+   "layer_top12_min": 0.002801999999999999,
+   "layer_top12_med": 0.19718500000000005,
+   "layer_top67_min": 0.0009519999999998419,
+   "layer_top67_med": 0.036826000000000025,
+   "n_layers_gap67_below_0p05": 33,
+   "n_layers_gap12_below_0p05": 7
+  },
+  {
+   "prompt": "Translation to German: the weather is beautiful today",
+   "step": 5,
+   "golden": 4484,
+   "tfarg": 286,
+   "gap": 3.819496,
+   "intop5": "yes",
+   "layer_top12_min": 0.0007070000000000132,
+   "layer_top12_med": 0.287526,
+   "layer_top67_min": 0.000280000000000058,
+   "layer_top67_med": 0.03786900000000004,
+   "n_layers_gap67_below_0p05": 33,
+   "n_layers_gap12_below_0p05": 7
+  },
+  {
+   "prompt": "Translation to German: the weather is beautiful today",
+   "step": 10,
+   "golden": 5445,
+   "tfarg": 2302,
+   "gap": 3.030571,
+   "intop5": "no",
+   "layer_top12_min": 0.003015999999999991,
+   "layer_top12_med": 0.19559200000000004,
+   "layer_top67_min": 5.100000000002325e-05,
+   "layer_top67_med": 0.027216000000000018,
+   "n_layers_gap67_below_0p05": 35,
+   "n_layers_gap12_below_0p05": 9
+  },
+  {
+   "prompt": "Translation to German: the weather is beautiful today",
+   "step": 22,
+   "golden": 5445,
+   "tfarg": 2302,
+   "gap": 1.899872,
+   "intop5": "no",
+   "layer_top12_min": 0.0015570000000000306,
+   "layer_top12_med": 0.29370300000000005,
+   "layer_top67_min": 0.001670999999999978,
+   "layer_top67_med": 0.042990000000000084,
+   "n_layers_gap67_below_0p05": 31,
+   "n_layers_gap12_below_0p05": 5
+  },
+  {
+   "prompt": "Ein gutes Buch liest man am besten",
+   "step": 7,
+   "golden": 51092,
+   "tfarg": 2459,
+   "gap": 3.285328,
+   "intop5": "yes",
+   "layer_top12_min": 0.004497999999999891,
+   "layer_top12_med": 0.278218,
+   "layer_top67_min": 0.0005789999999998852,
+   "layer_top67_med": 0.04965699999999995,
+   "n_layers_gap67_below_0p05": 26,
+   "n_layers_gap12_below_0p05": 6
+  },
+  {
+   "prompt": "counting: one two three four five six seven",
+   "step": 0,
+   "golden": 12843,
+   "tfarg": 14031,
+   "gap": 6.156881,
+   "intop5": "no",
+   "layer_top12_min": 0.008246999999999893,
+   "layer_top12_med": 0.170104,
+   "layer_top67_min": 0.0010459999999999914,
+   "layer_top67_med": 0.02381000000000011,
+   "n_layers_gap67_below_0p05": 39,
+   "n_layers_gap12_below_0p05": 13
+  },
+  {
+   "prompt": "counting: one two three four five six seven",
+   "step": 1,
+   "golden": 12843,
+   "tfarg": 14031,
+   "gap": 8.646961,
+   "intop5": "no",
+   "layer_top12_min": 0.002374999999999905,
+   "layer_top12_med": 0.24289000000000005,
+   "layer_top67_min": 0.0004789999999998962,
+   "layer_top67_med": 0.028886000000000023,
+   "n_layers_gap67_below_0p05": 38,
+   "n_layers_gap12_below_0p05": 12
+  },
+  {
+   "prompt": "counting: one two three four five six seven",
+   "step": 2,
+   "golden": 12843,
+   "tfarg": 14031,
+   "gap": 5.815979,
+   "intop5": "no",
+   "layer_top12_min": 0.004530999999999952,
+   "layer_top12_med": 0.128412,
+   "layer_top67_min": 0.00014899999999995472,
+   "layer_top67_med": 0.018912999999999958,
+   "n_layers_gap67_below_0p05": 41,
+   "n_layers_gap12_below_0p05": 10
+  },
+  {
+   "prompt": "counting: one two three four five six seven",
+   "step": 3,
+   "golden": 12843,
+   "tfarg": 14031,
+   "gap": 4.99181,
+   "intop5": "yes",
+   "layer_top12_min": 0.01208100000000012,
+   "layer_top12_med": 0.24573,
+   "layer_top67_min": 0.00014399999999997748,
+   "layer_top67_med": 0.030302999999999997,
+   "n_layers_gap67_below_0p05": 34,
+   "n_layers_gap12_below_0p05": 10
+  },
+  {
+   "prompt": "counting: one two three four five six seven",
+   "step": 4,
+   "golden": 12843,
+   "tfarg": 14031,
+   "gap": 3.630234,
+   "intop5": "yes",
+   "layer_top12_min": 0.000136000000000025,
+   "layer_top12_med": 0.20996,
+   "layer_top67_min": 0.0004939999999999944,
+   "layer_top67_med": 0.033832,
+   "n_layers_gap67_below_0p05": 32,
+   "n_layers_gap12_below_0p05": 10
+  }
+ ]
+}