tenstorrent: f32-out matmul must not land on the bf16 grid; the kolibri residue is chain drift through a high-gain router

[lu-zero]

Two results from the L1-moe localization (the ~3% relative divergence that remained after #3443's dequant fix).

A real contract defect, fixed: TT kMatmul/kMatmulBT produced a bf16-upcast tensor on commit for f32-out callers — all 384 kolibri router logits sat on the bf16 grid with bf16 partial sums. F32-out now requests dtype FLOAT32 with fp32_dest_acc_en (bf16grid 0/384 post-fix, instrumentation committed behind VT_KOLIBRI1_TT_STAGE_DUMP).

The residue classified — STOP class: with the router GEMM now correct (both arms match host-reference logits computed from their own inputs: CPU 1.4e-5, TT 0.15), the remaining hard flips are continuous precision drift driving discrete selection flips. The L0 input norm differs by 1 bf16 ULP class (substrate-adjudicated drift); by the L0 moe input that carries ~0.4% rms outliers; and the kolibri router (norm-weighted, K=2560, logits O(5)) amplifies that into O(1–5) absolute logit noise, flipping the near-tie top-6-of-384 selection at every layer. The hard flips (4 ON / 3 OFF, same positions) are this amplifier, not an op bug — the router gate weights are byte-identical across arms and both GEMMs are verifiably correct.

The endgame is therefore an adjudication decision, not a local patch: either the device attention/norm chain goes f32 (a perf and substrate-capability question), or the device gate's standard moves to the near-tie instrument's terms (band + zero-hard under teacher forcing, with a re-captured device golden), exactly mirroring the CPU #3438 discussion. The instrument to verify either path is merged (#3442).

Issue: ISSUE-LOCAL-01M4ER0E9HHM95YZYJB7T5FECN (full evidence chain; probes committed behind VT_KOLIBRI1_TT_STAGE_DUMP).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]

[localai-org-maint-bot]

@mudler @richiejp One source-level blocker in the added diagnostic path at 1bf1e5c:

In kolibri1_tt_forward.cpp::MoeBlock, the StageDumpOn() && l3dbg(layer) block indexes g_kolibri1_dbg_route_logits[k] unconditionally while iterating over href. That global starts empty and is populated only by the CPU forward path's RouteDump. Running TT alone with VT_KOLIBRI1_TT_STAGE_DUMP=1 therefore reads out of bounds; a prior CPU run with a shorter token batch also leaves it too small. The size check in TT RouteDump runs later and does not protect this earlier loop.

Please separate the same-input host-reference comparison from the optional CPU comparison, and require matching reference dimensions before reading the latter. The reference also needs to identify the same layer/prefix; a process-global last CPU layer cannot establish layer-by-layer parity by itself. A diagnostic test should cover TT-only execution and mismatched reference sizes.

This is a focused source finding, not acceptance of the stacked TT changes. I could not run the hardware/model gates on this host, so I have not pushed an unvalidated alteration to this branch.