kolibri1 TT: the routed-expert slot dequant read its fp8 source at twice the element stride

.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN.md

@@ -36,3 +36,122 @@ B2b-i landed the dense-resident device forward with the routed-expert tier delib
   token-gate leg is committed and runs on the first healthy card
   window. Issue stays OPEN on the device gates.
 -
+- 2026-10-10 (branch row/tt-kolibri-residue, off e510885d7 + the
+  teacher-forced instrument eb7cefb85): the prompt-dependent hard
+  divergence is ROOT-CAUSED to the tt-metal rms_norm kernel, evidence in
+  test_kolibri1_tt_b2ii.cpp "SCRATCH dbg rmsnorm micro" (red, kept). With
+  BIT-IDENTICAL bf16 inputs and gamma ([1,2560], random seed 7), the
+  device arm — ttnn::rms_norm over f32 tiles via the NormalizeDevF32Tile
+  pin-forward — measures sum ratio 0.98992 and max_abs 0.0625 (4 bf16
+  ULP) against the CPU row / host-double oracle; the CPU row matches
+  host-double to 1 ULP. Normed 3-4x per layer over 50 layers, that
+  systematic bias compounds into the multi-nat logit shifts the
+  teacher-forced instrument records (OFF arm 47 HARD, worst gap 5.17
+  nats; per-prompt 'der Mond...' steps 0-24, 'Translation to German...'
+  steps 0-22, 'x1 = 3...' steps 1-9). NOT a slot-pool/expert-stream
+  defect for these prompts: the slot-pool no-slot refusal STILL fires on
+  the fixed base (4 of 8 teacher-forced walks abort, e.what()=="1") but
+  the diverging prompts never abort — two defect classes share the row.
+  Two repairs were tried and refused by the substrate, both recorded in
+  src/vt/tenstorrent/tenstorrent_ops.cpp (RmsNormKernel): (1) a composed
+  f32 chain (x^2 -> mean -> +eps -> rsqrt -> scale), exact in the micro
+  (ratio 0.9998, 1 ULP) — its [rows,1] row scale cannot reach the model:
+  plain multiply broadcasts padded-tile garbage (gate collapse, 1e37
+  logits), BcastOpDim::W is numerically wrong (ratio 1.019), ::H is
+  refused, and repeat+same-shape multiply is refused by binary_ng
+  ("Invalid subtile broadcast type"); (2) serving the eager arm's
+  residual-free short-row norms from the host f32 loop — the extra host
+  round-trips desync the device shadow (gate 26/33 -> 0/8 with 1e37
+  garbage), both all-norms and hidden-width-only narrowings. Both
+  reverted; the landed tree keeps baseline routing (byte gate back at
+  26/33, 47 HARD — verified post-revert, /tmp/final_off.log) plus the
+  composed arm behind VT_TT_RMSNORM_COMPOSED=1 for A/B. The embed table
+  and the norm gamma stage BIT-EXACT on device (VT_KOLIBRI1_TT_STAGE_DUMP
+  emb/gam dumps, both arms) — staging is clean; the bias is in the
+  kernel math. defect class: tt-metal substrate (ttnn::rms_norm numeric
+  bias + binary_ng subtile broadcast refusal). ISSUE STAYS OPEN; the
+  fix needs either an upstream rms_norm with f32 statistics that
+  matches the oracle, or a broadcast primitive that survives the model
+  geometry, or the host-free shadow reconciliation this row already
+  owes.
+
+- 2026-10-10 (substrate rms reduction fix, this branch): the blocked
+  in-kernel repairs are superseded by a TT-METAL SUBSTRATE patch in the
+  trial tree `/tmp/tt-metal-umdtrial` (installed at
+  `/tmp/umdtrial-install`, rebuilt and reinstalled in
+  `/tmp/build-umdtrial2`). Mechanism: `ttnn::rms_norm`'s default compute
+  config hardwired `fp32_acc = false`
+  (`ttnn/cpp/ttnn/operations/normalization/rmsnorm/rmsnorm.cpp`), so the
+  layernorm program factory ran Float16_b circular buffers and
+  `float32_reduction = fp32_dest_acc_en && !legacy_reduction` was false
+  (`layernorm_op_multi_core.cpp`) EVEN when the vllm.cpp f32-shadow arm
+  fed FLOAT32 tiles — the sum of squares accumulated in bf16, which is
+  the whole 0.9899 bias. The patch sets `fp32_acc = true` and
+  `approx_mode = false` for the rms_norm default config (f32 CBs, f32
+  reduce, accurate SFPU rsqrt) and widens the reduce scaler CB to
+  Float32 under an f32 reduction. Micro (`SCRATCH dbg rmsnorm micro`,
+  same bit-identical bf16 [1,2560] seed-7 inputs): ratio 0.98992 ->
+  0.99972 vs the CPU row, max_abs 4 ULP -> 1 ULP, and vs the
+  host-double-bf16 denominator TT 0.99915 vs CPU 0.99942 — the residual
+  is the bf16 output-store quantization floor, which the CPU oracle row
+  itself cannot beat. Gate verdicts (both arms re-run, clean
+  reset+cache per leg): ON 26/33, 7 flips (0 near-tie, 7 hard),
+  instrument 40 tf flips (5 near-tie, 35 HARD) worst 3.19 nats; OFF
+  26/33, 7 flips (1 near-tie, 6 hard — baseline was 7 hard), instrument
+  116 tf flips (11 near-tie, 105 HARD) worst 7.59 nats. The ARGMAX
+  chain did not move materially: the remaining flips are dominated by
+  the two open op-level drifts (kGdnDecode state reduction,
+  kMatmulBTQuantGrouped Q4_K envelope), both re-confirmed failing in
+  isolation; the >=141 target stays out of reach. Suite
+  `test_tenstorrent_backend`: the only cases failing in isolation are
+  the two known ones (kGdnDecode, kMatmulBTQuantGrouped); the W4
+  EnsureDevice2D counter and the matmul region class split fail only in
+  full-suite order state and pass isolated. NOTE: the OFF-arm
+  teacher-forced instrument reads 105 HARD / worst 7.59 nats in this
+  run against the 47 / 5.17 recorded on 5277bdc72 — deterministic
+  across a clean reset; the instrument counts over the full 33-position
+  harness, so part of the delta is coverage, and the norm arm change
+  plausibly moved teacher-forced near-ties; the argmax chain (the
+  gate's contract) is unchanged. THE TRIAL TT-METAL TREE NOW CARRIES
+  THIS REDUCTION-PRECISION PATCH ON TOP OF THE PORT FIXES — any future
+  measurement or gate on this stack must use the rebuilt
+  `/tmp/umdtrial-install` (rebuild: `cmake --build /tmp/build-umdtrial2
+  -j 4 && cmake --install . --prefix /tmp/umdtrial-install`).
+
+- 2026-10-10 (branch row/tt-kolibri-flips2, host-free worktree): the
+  remaining-gate-divergence localization RE-DERIVED on the fixed substrate
+  (rmsnorm fp32-accumulation patch in /tmp/umdtrial-install). The prior
+  attribution to kGdnDecode / kMatmulBTQuantGrouped is FALSIFIED for kolibri1
+  at census level: the kolibri TT forward dispatches Embedding, MatmulBT,
+  RmsNorm/FusedChain, MoeSiluMul, MoeCombine, RopeNeox (sliding layers),
+  the KV write + PagedAttention, and Add — kolibri has NO GDN op and its
+  experts are fp8-block dequanted to bf16 on HOST (no kMatmulBTQuant, no
+  kMatmulBTQuantGrouped arm anywhere in the forward). The first diverging
+  stage on the fixed substrate was the routed-expert MoE block at layer 0:
+  the memoized slot dequant of the DOWN projection dequanted the WRONG
+  source bytes — `byte_base += n * k * 2` doubled for BOTH the fp8 source
+  (one byte per element) and the bf16 destination, so up read down's slot
+  region and down read past the slot end (NaN/Inf weights, 1e38 expert
+  outputs, every downstream stage corrupt). Red-first evidence:
+  `downdequant` ref-vs-memo 1,177,065/1,310,720 elements mismatched
+  (NaN memo sums) and the one-hot down_w probe returned inf/NaN at k>=256
+  against a clean scale grid; slotcmp proved the slot bytes themselves
+  byte-exact. FIX: separate `packed_base` (n*k) and `bf16_base` (n*k*2)
+  offsets in kolibri1_tt_forward.cpp. After: dequant mismatch 0/1,310,720,
+  L0 moe matches the CPU arm exactly, first diverging stage moves to L1 moe
+  at ~3% relative (near-tie class). Numbers (fixed substrate, this branch):
+  teacher-forced 'der Mond...' worst gap 11.76 nats -> 10.05 nats with the
+  argmax chain recovering CPU agreement at 5 of 13 steps (was 0);
+  device token gate 4 HARD flips ON (steps to flip moved much later:
+  'der Mond' step 7 gap -5.92; 'capital of Australia' step 5 -4.56;
+  'Wissen ist Macht' step 8 -13.35; 'x1 = 3...' step 22 -3.06) and 3 HARD
+  flips OFF before the run ABORTS on device DRAM exhaustion
+  (TT_FATAL Out of Memory, 4.27 GiB bank space full — the memoized
+  expert-dequant device staging budget; walks now survive much longer than
+  the pre-fix early flips, so the gate reaches the exhaustion
+  deterministically at the same point on both arms). THE OOM IS OPEN DEBT:
+  the full 33-prompt tally cannot complete on this substrate until the
+  memoized-expert device residency is capped or freed. The streaming
+  slot-pool `what()=="1"` throw did NOT fire on the fixed substrate in any
+  instrument or gate run (0 occurrences). The remaining flips after the fix
+  are the residual bf16 device-arm drift class, not the byte bug.

docs/bench-evidence/kolibri1-tt-b2ii-20261009.md

@@ -179,3 +179,50 @@ Owed after this session: root-cause the host-free device handoff
 paths host materialization bypasses); the two op-level drifts above;
 the decode bench anchor 109726 (chain 101807/109726), gated behind the
 token gate per the addendum.
+
+## 7. Substrate rms reduction-precision patch (2026-10-10, row/tt-kolibri-residue)
+
+The rms_norm bias root-caused in section 6's red micro was fixed at the
+TT-METAL SUBSTRATE level, not in vllm.cpp. The trial tt-metal tree
+`/tmp/tt-metal-umdtrial` now carries a reduction-precision patch on top
+of the port fixes, and `/tmp/umdtrial-install` was rebuilt from it
+(build dir `/tmp/build-umdtrial2`; rebuild recipe:
+`cmake --build /tmp/build-umdtrial2 -j 4 && cmake --install . --prefix
+/tmp/umdtrial-install`). ANY future measurement or gate on this stack
+must use this rebuilt install.
+
+Substrate patch (3 edits in the trial tree):
+
+- `ttnn/cpp/ttnn/operations/normalization/rmsnorm/rmsnorm.cpp`
+  (`rmsnorm_default_compute_config`): `fp32_acc = true`,
+  `approx_mode = false`. Mechanism: the old default (`fp32_acc=false`)
+  made the layernorm program factory run Float16_b CBs with
+  `float32_reduction=false` (`layernorm_op_multi_core.cpp:499`) even
+  for FLOAT32 input tiles — the kolibri1 norms' sum-of-squares
+  accumulated in bf16, the whole -3.4% bias.
+- `ttnn/cpp/ttnn/operations/normalization/layernorm/device/
+  layernorm_op_multi_core.cpp`: the reduce scaler CB is Float32 when
+  `fp32_dest_acc_en` (the 1/W mean factor no longer rides a bf16 tile).
+
+Verdicts (same env/reset/cache recipe as section 6, clean reset per leg):
+
+- Micro (`SCRATCH dbg rmsnorm micro`, bit-identical bf16 [1,2560]
+  seed 7): sum ratio 0.98992 -> 0.99972 vs the CPU row, max_abs
+  0.0625 -> 0.015625 (4 ULP -> 1 ULP). Against the host-double sum
+  rounded per element to bf16, TT 0.99915 vs CPU 0.99942: the residual
+  is the bf16 output-store quantization floor, not a kernel bias.
+- B2b-ii gate, host-free ON: ARGMAX CHAIN 26/33, flips 7 (7 hard);
+  instrument 40 tf flips (5 near-tie, 35 HARD), worst 3.19 nats
+  (baseline 47 HARD, worst 5.17).
+- B2b-ii gate, `VT_TT_HOST_FREE_DECODE=0`: ARGMAX CHAIN 26/33, flips 7
+  (1 near-tie, 6 hard — baseline 7 hard); instrument 116 tf flips
+  (105 HARD), worst 7.59 nats, deterministic across a clean reset.
+- Verdict: the systematic norm bias is gone but the argmax chain did
+  not move: the remaining flips belong to the two open op-level drifts
+  (kGdnDecode, kMatmulBTQuantGrouped), both re-confirmed failing in
+  isolation on this stack. The >=141 target stays open.
+- `test_tenstorrent_backend` on this stack: the only cases failing in
+  isolation are the two known ones; the W4 EnsureDevice2D
+  `uploads_bulk_bf16` counter and the matmul region class split fail
+  only in full-suite order state and pass isolated (same intermittent
+  class as the section-6 RAC flake).

scripts/env-doc-allowlist.txt

@@ -124,6 +124,7 @@ VT_KOLIBRI1_PROFILE
 VT_KOLIBRI1_TT_B2I_PROGRESS
 VT_KOLIBRI1_TT_B2BI_PROGRESS
 VT_KOLIBRI1_TT_B2II_STREAM
+VT_KOLIBRI1_TT_STAGE_DUMP
 VT_KV_ALLOC_LOG
 VT_ATTN_SELECT_LOG
 VT_LAGUNA_DECODE_GRAPH
@@ -213,6 +214,7 @@ VT_TT_NO_IDS_WARM
 VT_TT_NO_IDX_WARM
 VT_TT_NO_ROPE_REFRESH
 VT_TT_RECAPTURE_EVERY
+VT_TT_RMSNORM_COMPOSED
 VT_TT_SDPA_TEST
 VT_TT_SHARD_Q
 VT_TT_TRACE_REGION_MB

src/vllm/model_executor/models/kolibri1_forward.cpp

@@ -464,6 +464,7 @@ ForwardLogits ForwardKolibri1Forward(
     Tensor tab = ResidentWeight(d, weights.embed_tokens, {vocab, h});
     vt::Embedding(d.q, hidden_buf.t(), tab, ids.t());
   }
+  StageDump("emb", -2, hidden_buf, d);
   Tensor hidden = hidden_buf.t();
   DBuf res(d, DType::kBF16, {t, h});
   res.Zero(d);
@@ -493,6 +494,30 @@ ForwardLogits ForwardKolibri1Forward(
     // input_layernorm + residual (the vLLM fused add-norm contract).
     DBuf dhn(d, DType::kBF16, {t, h});
     Tensor w_in = ResidentWeight(d, lw.input_layernorm, {h});
+    if (StageDumpOn() && l == 0) {
+      // SCRATCH: the gamma bytes as the device arm would see them (bf16).
+      double s = 0;
+      auto val = [](uint16_t bits) {
+        uint32_t u = static_cast<uint32_t>(bits) << 16;
+        float f;
+        std::memcpy(&f, &u, 4);
+        return f;
+      };
+      std::vector<uint16_t> bf(static_cast<size_t>(h));
+      if (w_in.dtype == vt::DType::kBF16) {
+        const auto* p16 = static_cast<const uint16_t*>(w_in.data);
+        for (int64_t i = 0; i < h; ++i) bf[static_cast<size_t>(i)] = p16[i];
+      } else {
+        const auto* pf = static_cast<const float*>(w_in.data);
+        for (int64_t i = 0; i < h; ++i) {
+          float v = pf[i];
+          std::memcpy(&bf[static_cast<size_t>(i)], &v, 2);  // low bits (trunc)
+        }
+      }
+      for (int64_t i = 0; i < h; ++i) s += val(bf[static_cast<size_t>(i)]);
+      std::fprintf(stderr, "[STAGE] gam sum=%.6f first=[%.6f %.6f %.6f %.6f]\n",
+                   s, val(bf[0]), val(bf[1]), val(bf[2]), val(bf[3]));
+    }
     Tensor dhn_t = dhn.t();
     Tensor res_t = res.t();
     if (dense_attn::FusedChainAdoptEnabled()) {

src/vllm/model_executor/models/kolibri1_tt_forward.cpp

@@ -488,6 +488,14 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p,
   // Re-record the epoch AFTER the fills so the next layer's check fires
   // only on a NEW content change (swaps it did not cause itself).
   st.epoch.Record(st.pool.policy);
+  if (StageDumpOn()) {
+    std::fprintf(stderr,
+                 "[STAGE] pool L%lld staged=%lld verified=%lld fills=%lld\n",
+                 static_cast<long long>(layer),
+                 static_cast<long long>(st.staged_bytes),
+                 static_cast<long long>(st.readback_verified_bytes),
+                 static_cast<long long>(st.slot_fills));
+  }
 
   // The routed experts: the CPU row's gather/ExpertMlp/scatter over the
   // slot dequants, then the weighted combine with the always-added shared
@@ -500,6 +508,7 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p,
             const_cast<int32_t*>(route.ids.data()));
   DBuf shared = ExpertMlp(d, dhn, t, p.shared_expert_intermediate_size,
                           sh_gate_w, sh_up_w, sh_down_w);
+  StageDump("moe_shared", layer, shared, d);
   DBuf expert_out(d, DType::kBF16, {t, top_k, h});
   expert_out.Zero(d);
   for (int64_t ex = 0; ex < e; ++ex) {
@@ -559,7 +568,14 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p,
         be.Free(ptr);
       });
       vt::Tensor* dsts[3] = {&dq.gate, &dq.up, &dq.down};
-      int64_t byte_base = 0;
+      // Two offsets over the carved allocation: the packed fp8 SOURCE
+      // advances ONE byte per element, the bf16 destination TWO. The old
+      // single byte_base doubled for both and walked the source out of the
+      // slot: up dequanted down's region and down dequanted past the slot
+      // end — NaN/Inf weights, the 1e38 routed-expert outputs, and the hard
+      // flips downstream (the local record this commit cites).
+      int64_t packed_base = 0;
+      int64_t bf16_base = 0;
       for (int pi = 0; pi < 3; ++pi) {
         const Kolibri1Projection& pr =
             pi == 0 ? ewx.gate_proj : (pi == 1 ? ewx.up_proj : ewx.down_proj);
@@ -568,16 +584,17 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p,
         VT_CHECK(static_cast<int64_t>(pr.fp8_block.packed.bytes.size()) ==
                      n * k,
                  "kolibri1-tt B2b-ii: routed projection byte math diverged");
-        auto* dst = static_cast<uint16_t*>(bp) + byte_base / 2;
-        const auto* src = rb.data() + byte_base;
+        auto* dst = static_cast<uint16_t*>(bp) + bf16_base / 2;
+        const auto* src = rb.data() + packed_base;
         const auto* sc =
             reinterpret_cast<const float*>(pr.fp8_block.scale.bytes.data());
         host_parallel::ForOutputRows(n, k, [&](int64_t n0, int64_t n1) {
           kolibri1_fp8::DequantRowsBf16(src, sc, scale_cols, n0, n1, k,
                                         pr.fp8_block.block_n,
                                         pr.fp8_block.block_k, dst);
         });
-        byte_base += n * k * 2;
+        packed_base += n * k;
+        bf16_base += n * k * 2;
         *dsts[pi] = MakeTensor(dst, DType::kBF16, d.q.device,
                                std::vector<int64_t>{n, k});
       }
@@ -593,8 +610,10 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p,
         d.b.Copy(d.q, dp + s * rb,
                  sp + static_cast<size_t>(token_rows[s]) * rb, rb);
     }
+    StageDump("moe_gath", layer, gathered, d);
     DBuf o = ExpertMlp(d, gathered.t(), ne, inter, it->second.gate,
                        it->second.up, it->second.down);
+    StageDump("moe_ex", layer, o, d);
     for (int64_t i = 0; i < ne; ++i) {
       d.b.Copy(d.q,
                static_cast<char*>(expert_out.ptr()) +
@@ -607,9 +626,16 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p,
     }
   }
   DBuf out(d, DType::kBF16, {t, h});
+  if (StageDumpOn()) {
+    std::vector<float> wsum(route.weights.begin(), route.weights.end());
+    double s = 0;
+    for (float v : wsum) s += v;
+    std::fprintf(stderr, "[STAGE] route sum=%.6f n=%zu\n", s, wsum.size());
+  }
   Tensor shared_t = shared.t();
   vt::MoeCombine(d.q, out.t(), expert_out.t(), dtw.t(), &shared_t,
                  /*routed_scale=*/1.0f);
+  StageDump("moe_combined", layer, out, d);
   return out;
 }
 
@@ -751,6 +777,7 @@ ForwardLogits ForwardKolibri1TTResidentForward(
     Tensor tab = ResidentWeight(d, weights.embed_tokens, {vocab, h});
     vt::Embedding(d.q, hidden_held->t(), tab, ids.t());
   }
+  StageDump("emb", -2, *hidden_held, d);
   Tensor hidden = hidden_held->t();
   DBuf res(d, DType::kBF16, {t, h});
   res.Zero(d);
@@ -783,6 +810,23 @@ ForwardLogits ForwardKolibri1TTResidentForward(
     // CPU row's exact conditional, so the op sequence matches its default.
     DBuf dhn(d, DType::kBF16, {t, h});
     Tensor w_in = ResidentWeight(d, lw.input_layernorm, {h});
+    if (StageDumpOn() && l == 0) {
+      std::vector<uint8_t> tmp(static_cast<size_t>(h) * 2);
+      d.b.Copy(d.q, tmp.data(),
+               static_cast<const uint8_t*>(w_in.data), tmp.size());
+      d.b.Synchronize(d.q);
+      const auto* bf = reinterpret_cast<const uint16_t*>(tmp.data());
+      double s = 0;
+      auto val = [](uint16_t bits) {
+        uint32_t u = static_cast<uint32_t>(bits) << 16;
+        float f;
+        std::memcpy(&f, &u, 4);
+        return f;
+      };
+      for (int64_t i = 0; i < h; ++i) s += val(bf[i]);
+      std::fprintf(stderr, "[STAGE] gam sum=%.6f first=[%.6f %.6f %.6f %.6f]\n",
+                   s, val(bf[0]), val(bf[1]), val(bf[2]), val(bf[3]));
+    }
     Tensor dhn_t = dhn.t();
     Tensor res_t = res.t();
     if (dense_attn::FusedChainAdoptEnabled()) {