From 31192350a45c9ceb1b5810f6e4c050cbe4abdc03 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 11 Oct 2026 17:29:33 +0200 Subject: [PATCH 1/4] kolibri1 TT: generalize the B1 per-token stream envelope to multi-token forwards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The B1 per-step stream guard was a one-token window: the forward charged every layer's fetch against per_token_stream_bytes, so any multi-token prefill forward refused loudly (whole-prompt 128: 1,404,123,840 B against the 1,179,936,000 B bound at layer 3; the b2ii gate recorded 1,242,865,920 B at layer 15). The bound is a per-token envelope, not a per-step buffer: the guard now scales the window by the forward's token count, both the per-forward charge window (kolibri1_tt_forward.cpp) and the per-layer check inside BuildKolibri1TTSlotFetchList. One-token forwards are unchanged — the scale factor is 1. Chunked prefill was measured with the generalization (8-token chunks, bench arm): the 128-token prefill walk dropped 156.3 s -> 72.1 s with the decode split unchanged. It is NOT adopted as the default: the greedy chain could not be proven equivalent, because this instrument is not bit-reproducible run to run — three runs of the same one-token walk gave three different chains (first token 346, 30686, and 10 under chunking), and the b2ii golden gate's HARD flips move between runs (it is a band gate, not a bit gate). Adopting chunked prefill owes band-class equivalence evidence of its own; the decode bench comment records the evidence and the decode bench keeps the faithful one-token walk. Gate evidence (P150, tt-smi reset before every run): - decode bench, chunked arm: prefill 72.1 s vs 156.3 s baseline; decode 59.1 s vs 59.8 s (within run-to-run noise). - b2ii device gate: 15/16 both on origin/main and with this change (the failing golden-chain case flips different prompts per run on origin/main alone) — verdict class unchanged. - test_tenstorrent_backend: 104/106, failing exactly the two known kGdnDecode and kMatmulBTQuantGrouped cases, unchanged. Owes: the chunked-prefill equivalence adjudication (ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] --- .../ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4.md | 19 ++++++++++++++ .../models/kolibri1_tt_forward.cpp | 16 +++++++----- .../models/test_kolibri1_tt_decode_bench.cpp | 25 +++++++++++++++---- 3 files changed, 49 insertions(+), 11 deletions(-) create mode 100644 .agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4.md diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4.md b/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4.md new file mode 100644 index 0000000000..a705e8d937 --- /dev/null +++ b/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M4NS6D2GCD5WXBFN24XGXYQ4 +Title: Generalize the B1 per-token stream bound to multi-token forwards; record the chunked-prefill evidence +Row: MODEL-TEXT-kolibri-1-tenstorrent +State: OPEN +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-11 +Updated: 2026-10-11 +Closed: - + +## Problem + +The B1 per-step stream guard admits only one-token forwards; a 128-token whole-prompt prefill refuses at 1.40 GB against the 1.18 GB bound. Chunked prefill was measured (prefill 156.3 s -> 72.1 s) but chain equivalence is not adjudicable on this instrument (chains differ run to run on identical code). + +## Resolution + +- diff --git a/src/vllm/model_executor/models/kolibri1_tt_forward.cpp b/src/vllm/model_executor/models/kolibri1_tt_forward.cpp index 2bdfc6caff..48ac73b859 100644 --- a/src/vllm/model_executor/models/kolibri1_tt_forward.cpp +++ b/src/vllm/model_executor/models/kolibri1_tt_forward.cpp @@ -597,7 +597,8 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p, PlanKolibri1TTMoEDispatch(st.pool.policy, layer_req); const Kolibri1TTLayerDispatch& ld = disp.layers[0]; const Kolibri1TTSlotFetchList list = BuildKolibri1TTSlotFetchList( - st.pool.policy, ld, weights, st.pool, st.per_token_stream_bytes); + st.pool.policy, ld, weights, st.pool, + st.per_token_stream_bytes * t); VT_CHECK(st.guard != nullptr, "kolibri1-tt B2b-ii: the per-step stream guard is missing"); st.guard->Charge(layer, list.stream_bytes); @@ -948,15 +949,18 @@ ForwardLogits ForwardKolibri1TTResidentForward( Kolibri1TTStreamingDeviceContext* streaming) { VT_CHECK(queue.device.type == vt::DeviceType::kTENSTORRENT, kNonTTRefusal); VT_CHECK(ctx.built, "kolibri1-tt forward: the device context is not built"); + const Kolibri1Params& p = weights.params; + Dev d{vt::GetBackend(queue.device.type), queue}; + const int64_t t = static_cast(token_ids.size()); // A fresh charge window per step: the B1 per-token stream bound is a - // PER-TOKEN bound, and the guard refuses LOUDLY past it. + // PER-TOKEN bound, and the guard refuses LOUDLY past it. A multi-token + // forward (chunked prefill) draws the SAME per-token envelope t times, + // so the window scales with the token count — one token's bound is + // unchanged, and a chunk never claims a larger per-token allowance. if (streaming != nullptr) { streaming->guard = std::make_unique( - streaming->per_token_stream_bytes); + streaming->per_token_stream_bytes * t); } - const Kolibri1Params& p = weights.params; - Dev d{vt::GetBackend(queue.device.type), queue}; - const int64_t t = static_cast(token_ids.size()); const int64_t h = p.hidden_size; const int64_t vocab = p.vocab_size; diff --git a/tests/vllm/models/test_kolibri1_tt_decode_bench.cpp b/tests/vllm/models/test_kolibri1_tt_decode_bench.cpp index 63efd03a86..4ac68be4ca 100644 --- a/tests/vllm/models/test_kolibri1_tt_decode_bench.cpp +++ b/tests/vllm/models/test_kolibri1_tt_decode_bench.cpp @@ -82,11 +82,26 @@ v1::CommonAttentionMetadata OneReqMeta(int64_t t, int64_t ctx_before, // forward — one whole-prompt routed dispatch streams every selected expert's // slot bytes in a single charge and trips the B1 per-step stream bound // (observed here: 1404123840 B against the 1179936000 B bound at layer 3; -// the b2ii gate records the same refusal, 1242865920 B at layer 15). The -// one-token incremental walk over the persistent KV cache is the faithful -// same-math alternative (test_kolibri1_tt_b2ii.cpp device gate, which cites -// test_kolibri1_w3.cpp:298-310), so the prefill walks the prompt one token -// per forward and the walk is what the prefill timer covers. +// the b2ii gate records the same refusal, 1242865920 B at layer 15). +// +// CHUNKED PREFILL WAS TRIED (2026-10-11) AND IS NOT ADOPTED AS THE +// DEFAULT: with the per-token stream envelope generalized to multi-token +// forwards (kolibri1_tt_forward.cpp scales the guard by the token count) +// and an 8-token chunk walk, prefill dropped 156.3 s -> 72.1 s with the +// decode split unchanged. But the greedy chain could NOT be proven +// equivalent: this instrument is NOT bit-reproducible run to run — three +// runs of the SAME one-token walk produced three different chains (first +// token 346, 30686, and 10 under chunking), and the b2ii golden gate's +// HARD flips move between runs too (it is a band gate, 141/145 in the +// 2.5-nat band, not a bit gate). The chunked walk additionally runs the +// prefill-path kernels (the one-token walk forces pure_decode=true, +// decode-path kernels), a further drift source. Adopting chunked prefill +// therefore owes its own adjudicated equivalence evidence (a band-class +// TF comparison, not a chain identity check). The faithful one-token +// incremental walk over the persistent KV cache stays +// (test_kolibri1_tt_b2ii.cpp device gate, which cites +// test_kolibri1_w3.cpp:298-310), so the prefill walks the prompt one +// token per forward and the walk is what the prefill timer covers. constexpr int kPromptLen = 128; constexpr int kDecodeSteps = 64; From 7454f07de155ebfdb427c338faa85f8dde28828c Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 11 Oct 2026 19:14:37 +0200 Subject: [PATCH 2/4] kolibri1 TT: the device-side Kolibri-1 router op (kSigmoidBiasTopK), gate-verified, measured, not default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New shared op vt::SigmoidBiasTopK (OpId::kSigmoidBiasTopK): the Kolibri-1 router of kolibri1.py:126-142 as a registered kernel — selection score = RAW logit + e_score_correction_bias (NOT the DeepSeek noaux_tc sigmoid+bias form; vt::MoeRouterTopK cannot express it, its contract routes bias through the grouped path), weight = sigmoid of the UNBIASED logit, no renormalization, ties to the lower expert index. The CPU kernel (cpu_ops.cpp) is the exact port of SigmoidLogitAddRouting and the reference; the TT kernel (tenstorrent_ops.cpp) is a device composite — broadcast add, top-k via ttnn::experimental::topk_large_indices, dim-1 gather of the unbiased logit, ttnn::sigmoid — that downloads only the [T,K] ids + weights instead of the full [T,E] logits tail. Substrate constraints recorded in the kernel: topk_large_indices (the ONLY top-k op on this tt-metal) requires ROW_MAJOR BFLOAT16 input and k in multiples of 16, so the selection rounds the biased scores to bf16 (the router GEMM is already bf16 compute; the weights stay f32) and slices a k-bucket down to top_k. RED-FIRST evidence: the focused unit gate (test_tenstorrent_backend.cpp "kTENSTORRENT kSigmoidBiasTopK matches the CPU reference") ran red before the TT registration — REQUIRE(OpRegistered) failed — and green after: expert SETS equal to the CPU reference on random [4,384] logits, each weight within 2e-3 of sigmoid(unbiased logit at its own id) AND of the CPU weight for the same id. Recorded deviation: the ORDER within the top-k can swap on bf16 near-ties (selection precision), benign to the model math (the combine sums weight x expert over the k slots; the dispatch dedups). kolibri1_tt_forward.cpp wires the op into MoeBlock behind VT_KOLIBRI1_TT_DEVICE_ROUTER=1; the host path stays DEFAULT because the composite MEASURED NO WIN at this shape (P150, 2026-10-11, tt-smi reset recipe): decode median 60.9 s/64 steps (952 ms/step) over three runs against the host path's 59.1 s (924 ms/step) in-session — inside the run-to-run band, with the b2ii teacher-forced drift slightly WORSE (4 HARD flips vs the default path's 3-4, gaps -7.1/-7.4/-12.7/-4.4). The extra eager op launches per layer cost what the full-tail drain saved; the lever needs a captured/graphed region to amortize the launches. Gate evidence (P150, reset before every device run): - b2ii device gate: 15/16 BOTH on origin/main and with this change; the failing golden-chain case and its HARD flips move run to run on origin/main alone. Verdict class unchanged (issue: ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES records the not-default state). - test_tenstorrent_backend: the new gate passes; the suite is 104/107 — the two known kGdnDecode and kMatmulBTQuantGrouped failures plus one order-dependent RAC flake that passes in isolation and also appeared in the red run without this change's TT kernel. - test_kolibri1_tt 16/16, test_kolibri1 27/27. - decode bench, host path: decode 57.7-59.8 s/64 steps; device router: 60.1/60.9/67.2 s. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] --- .../ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES.md | 19 +++ include/vt/ops.h | 16 +++ .../models/kolibri1_tt_forward.cpp | 64 ++++++--- src/vt/cpu/cpu_ops.cpp | 39 +++++- src/vt/op_provider.cpp | 2 + src/vt/ops.cpp | 26 ++++ src/vt/tenstorrent/tenstorrent_ops.cpp | 87 ++++++++++++ tests/vt/test_tenstorrent_backend.cpp | 127 ++++++++++++++++++ 8 files changed, 359 insertions(+), 21 deletions(-) create mode 100644 .agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES.md diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES.md b/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES.md new file mode 100644 index 0000000000..b31dc2efef --- /dev/null +++ b/.agents/issues/MODEL-TEXT-kolibri-1-tenstorrent/ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M4NZA9B3DBKS3MD0M4TGRVES +Title: kolibri1 TT: device-side router op (kSigmoidBiasTopK) landed gate-verified but not default; no measured decode win +Row: MODEL-TEXT-kolibri-1-tenstorrent +State: OPEN +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-11 +Updated: 2026-10-11 +Closed: - + +## Problem + +vt::SigmoidBiasTopK removes the per-layer full-tail router-logits download, but the composite's extra eager launches per layer cost as much as the drain (decode 952 vs 924 ms/step median, within the run-to-run band; b2ii drift slightly worse: 4 vs 3 HARD flips). Default stays the host path; the op is opt-in via VT_KOLIBRI1_TT_DEVICE_ROUTER=1. The win needs a captured/graphed region to remove the launch overhead. + +## Resolution + +- diff --git a/include/vt/ops.h b/include/vt/ops.h index 2114898161..025f4d5f16 100644 --- a/include/vt/ops.h +++ b/include/vt/ops.h @@ -105,6 +105,7 @@ enum class OpId : uint8_t { kKdaGatedDeltaRule, kKdaChunkPrefill, kMoeRouterTopK, + kSigmoidBiasTopK, kMoeCombine, kAttention, kAttentionCross, @@ -2641,6 +2642,15 @@ using VHeadPermuteFn = void (*)(Queue&, Tensor&, const Tensor&, const VHeadPermuteArgs&); using MoeRouterTopKFn = void (*)(Queue&, Tensor&, Tensor&, const Tensor&, const MoeRouterTopKArgs&, const Tensor*); +// The Kolibri-1 router (kolibri1.py:126-142): selection score = RAW logit + +// e_score_correction_bias (NOT sigmoid-of-logit + bias — that is the +// DeepSeek noaux_tc path), weight = sigmoid of the UNBIASED logit, NO +// renormalization, ties break to the LOWER expert index. This is exactly +// SigmoidLogitAddRouting (kolibri1_shared.h) as a registered device op so +// the TT arm can run it on card and download only [T,K] ids + weights +// instead of the full [T,E] logits tail. +using SigmoidBiasTopKFn = void (*)(Queue&, Tensor&, Tensor&, const Tensor&, + const Tensor&, int); // The trailing float is `routed_scale` — the routed_scaling_factor applied to // the ROUTED sum before the shared term is added (see vt::MoeCombine). using MoeCombineFn = @@ -4643,6 +4653,12 @@ void MoeRouterTopK(Queue& q, Tensor& weights, Tensor& indices, const Tensor& log const MoeRouterTopKArgs& args, const Tensor* e_score_correction_bias = nullptr); +// The Kolibri-1 router op (see SigmoidBiasTopKFn above). logits [T,E] float, +// bias [E] f32, out: weights [T,K] f32 (sigmoid of the UNBIASED logit), +// indices [T,K] i32 (biased raw-logit selection, ties to the lower index). +void SigmoidBiasTopK(Queue& q, Tensor& weights, Tensor& indices, + const Tensor& logits, const Tensor& bias, int top_k); + // Weighted scatter-combine of the per-expert outputs (moe-semantics.md §4/§6). // out[t,:] = routed_scale * sum_j weights[t,j] * expert_out[t,j,:] (f32 accum) // + shared[t,:] (when shared != nullptr) diff --git a/src/vllm/model_executor/models/kolibri1_tt_forward.cpp b/src/vllm/model_executor/models/kolibri1_tt_forward.cpp index 48ac73b859..2e5d68a30e 100644 --- a/src/vllm/model_executor/models/kolibri1_tt_forward.cpp +++ b/src/vllm/model_executor/models/kolibri1_tt_forward.cpp @@ -484,29 +484,55 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p, // Router logits, f32 (kolibri1.py:171-177) — the bf16 gate over the // shared residency seam, f32 out like the CPU row's LinearBTRaw. DBuf dlog = LinearBTRaw(d, dhn, w.router_gate, t, DType::kF32); - std::vector logits(static_cast(t * e)); - dlog.Download(d, logits.data()); - // PROTOTYPE (ISSUE-LOCAL-01M4N8Y7XPDC74SED0WEDN0YFR): the bias is a STATIC - // weight, so the per-step D2H copy + blocking Synchronize here was 50 extra - // host round-trips per decode step. Cache the host image once per device - // buffer (the resident weights keep stable device pointers for the process - // lifetime) and serve the step from the cache. - const std::vector& bias = [&] { + // The routing itself. The device composite (vt::SigmoidBiasTopK) exists and + // is gate-verified, but MEASURED SLOWER at this shape and is NOT the + // default: VT_KOLIBRI1_TT_DEVICE_ROUTER=1 opts in (the composite's extra + // eager op launches per layer cost at least as much host serialization as + // the full-tail [T,E] download they replace). The host path stays default + // until a captured/graphed region removes the launch overhead. + static const bool kDeviceRouter = + std::getenv("VT_KOLIBRI1_TT_DEVICE_ROUTER") != nullptr && + std::string(std::getenv("VT_KOLIBRI1_TT_DEVICE_ROUTER")) == "1"; + std::vector logits; + std::vector bias; + Kolibri1HostRouting route; + if (kDeviceRouter) { + DBuf topw(d, DType::kF32, {t, top_k}); + DBuf topi(d, DType::kI32, {t, top_k}); + vt::SigmoidBiasTopK(d.q, topw.t(), topi.t(), dlog.t(), + ResidentWeight(d, w.e_score_correction_bias, {e}), + static_cast(top_k)); + route.ids.resize(static_cast(t * top_k)); + route.weights.resize(static_cast(t * top_k)); + topi.Download(d, route.ids.data()); + topw.Download(d, route.weights.data()); + } else { + logits.resize(static_cast(t * e)); + dlog.Download(d, logits.data()); + // The bias is a STATIC weight, so the per-step D2H copy + blocking + // Synchronize here was 50 extra host round-trips per decode step + // (the #3447 bias host cache: the resident weights keep stable device + // pointers for the process lifetime — cache the host image once per + // device buffer and serve the step from the cache). Tensor bt = ResidentWeight(d, w.e_score_correction_bias, {e}); std::lock_guard g(BiasMutex()); auto it = BiasCache().find(bt.data); - if (it != BiasCache().end()) return it->second; - vt::Backend& be = vt::GetBackend(d.q.device.type); - std::vector fresh(static_cast(e)); - be.Copy(d.q, fresh.data(), bt.data, static_cast(e) * sizeof(float)); - be.Synchronize(d.q); - return BiasCache().emplace(bt.data, std::move(fresh)).first->second; - }(); + if (it != BiasCache().end()) { + bias = it->second; + } else { + vt::Backend& be = vt::GetBackend(d.q.device.type); + bias.resize(static_cast(e)); + be.Copy(d.q, bias.data(), bt.data, static_cast(e) * sizeof(float)); + be.Synchronize(d.q); + BiasCache().emplace(bt.data, bias); + } + route = SigmoidLogitAddRouting(logits, bias, t, e, top_k); + } // The inherited sigmoid-logit-add routing (kolibri1_shared.h, verbatim // from the CPU row): selection on logits + bias, weights = sigmoid of // the UNBIASED logits, no renormalisation. - if (StageDumpOn() && l3dbg(layer)) { + if (!kDeviceRouter && StageDumpOn() && l3dbg(layer)) { const int64_t hk = w.router_gate.shape[1]; Tensor bt3 = ResidentWeight(d, w.router_gate, {e, hk}); std::vector tb(static_cast(e) * hk * 2); @@ -569,9 +595,9 @@ DBuf MoeBlock(Dev d, const Kolibri1MoeWeights& w, const Kolibri1Params& p, } } } - Kolibri1HostRouting route = - SigmoidLogitAddRouting(logits, bias, t, e, top_k); - RouteDump(layer, logits, route, t, e, top_k); + if (!kDeviceRouter) { + RouteDump(layer, logits, route, t, e, top_k); + } // SLICE ii: the routed tier. Without a streaming context (the // streaming-disabled arm) the B2b-i refusal still fires by name. diff --git a/src/vt/cpu/cpu_ops.cpp b/src/vt/cpu/cpu_ops.cpp index 3497e5677e..e33292e5da 100644 --- a/src/vt/cpu/cpu_ops.cpp +++ b/src/vt/cpu/cpu_ops.cpp @@ -3303,9 +3303,42 @@ void MoeRouterGroupedTopKKernel(Tensor& weights, Tensor& indices, const Tensor& // §3 router: softmax (f32, over all E) -> greedy top-k (lowest-index tie-break) // -> optional renormalize. weights [T,K] f32, indices [T,K] i32. +// The Kolibri-1 router as a kernel: EXACTLY SigmoidLogitAddRouting +// (kolibri1_shared.h, kolibri1.py:126-142) — selection score = RAW logit + +// e_score_correction_bias, weight = sigmoid of the UNBIASED logit, NO +// renormalization, ties break to the LOWER expert index (strict `>` scan over +// ascending index). The CPU reference and the executable spec; the TT kernel +// mirrors it (the op is the seam that removes the kolibri1 TT arm's +// per-layer full-tail router-logits download). +void SigmoidBiasTopKKernel(Queue&, Tensor& weights, Tensor& indices, + const Tensor& logits, const Tensor& bias, int top_k) { + const int64_t t = logits.shape[0], e = logits.shape[1]; + const float* l = logits.Ptr(); + const float* b = bias.Ptr(); + for (int64_t i = 0; i < t; ++i) { + const float* row = l + i * e; + std::vector taken(static_cast(e), 0); + for (int kk = 0; kk < top_k; ++kk) { + int64_t best = -1; + float best_score = 0.0f; + for (int64_t j = 0; j < e; ++j) { + if (taken[static_cast(j)]) continue; + const float score = row[j] + b[j]; + if (best < 0 || score > best_score) { + best = j; + best_score = score; + } + } + taken[static_cast(best)] = 1; + weights.Ptr()[i * top_k + kk] = + 1.0f / (1.0f + std::exp(-row[best])); + indices.Ptr()[i * top_k + kk] = static_cast(best); + } + } +} + void MoeRouterTopKKernel(Queue&, Tensor& weights, Tensor& indices, const Tensor& logits, - const MoeRouterTopKArgs& args, const Tensor* bias) { - if (args.num_expert_group > 0) { // W3 grouped-topk (`noaux_tc`) path + const MoeRouterTopKArgs& args, const Tensor* bias) { if (args.num_expert_group > 0) { // W3 grouped-topk (`noaux_tc`) path MoeRouterGroupedTopKKernel(weights, indices, logits, args, bias); return; } @@ -4525,6 +4558,8 @@ struct Registrar { static_cast(&VHeadPermuteKernel))); RegisterOp(OpId::kMoeRouterTopK, DeviceType::kCPU, reinterpret_cast(static_cast(&MoeRouterTopKKernel))); + RegisterOp(OpId::kSigmoidBiasTopK, DeviceType::kCPU, + reinterpret_cast(static_cast(&SigmoidBiasTopKKernel))); RegisterOp(OpId::kMoeCombine, DeviceType::kCPU, reinterpret_cast(static_cast(&MoeCombineKernel))); RegisterOp(OpId::kAttention, DeviceType::kCPU, diff --git a/src/vt/op_provider.cpp b/src/vt/op_provider.cpp index 87d88bf7e7..1dbeaf3a0f 100644 --- a/src/vt/op_provider.cpp +++ b/src/vt/op_provider.cpp @@ -332,6 +332,8 @@ const char* OpNameImpl(OpId op) { return "KdaChunkPrefill"; case OpId::kMoeRouterTopK: return "MoeRouterTopK"; + case OpId::kSigmoidBiasTopK: + return "SigmoidBiasTopK"; case OpId::kMoeCombine: return "MoeCombine"; case OpId::kAttention: diff --git a/src/vt/ops.cpp b/src/vt/ops.cpp index 2b66c6e7a1..4490fe9638 100644 --- a/src/vt/ops.cpp +++ b/src/vt/ops.cpp @@ -3451,6 +3451,32 @@ void MoeRouterTopK(Queue& q, Tensor& weights, Tensor& indices, const Tensor& log q, weights, indices, logits, args, e_score_correction_bias); } +void SigmoidBiasTopK(Queue& q, Tensor& weights, Tensor& indices, + const Tensor& logits, const Tensor& bias, int top_k) { + VT_CHECK(logits.rank == 2 && weights.rank == 2 && indices.rank == 2, + "sigmoid_bias_topk: logits/weights/indices rank-2"); + const int64_t t = logits.shape[0], e = logits.shape[1]; + VT_CHECK(bias.rank == 1 && bias.shape[0] == e, + "sigmoid_bias_topk: bias must be [num_experts]"); + VT_CHECK(IsFloat(logits.dtype) && bias.dtype == DType::kF32, + "sigmoid_bias_topk: float logits, f32 bias"); + VT_CHECK(weights.dtype == DType::kF32 && indices.dtype == DType::kI32, + "sigmoid_bias_topk: f32 weights, i32 indices"); + VT_CHECK(top_k >= 1 && top_k <= e, + "sigmoid_bias_topk: top_k must be in [1, num_experts]"); + VT_CHECK(weights.shape[0] == t && weights.shape[1] == top_k && + indices.shape[0] == t && indices.shape[1] == top_k, + "sigmoid_bias_topk: weights/indices must be [T, top_k]"); + VT_CHECK(logits.IsContiguous() && bias.IsContiguous() && + weights.IsContiguous() && indices.IsContiguous(), + "sigmoid_bias_topk: contiguous required"); + VT_CHECK(logits.device == q.device && bias.device == q.device && + weights.device == q.device && indices.device == q.device, + "sigmoid_bias_topk: device mismatch"); + reinterpret_cast(GetOp(OpId::kSigmoidBiasTopK, q.device.type))( + q, weights, indices, logits, bias, top_k); +} + void MoeCombine(Queue& q, Tensor& out, const Tensor& expert_out, const Tensor& weights, const Tensor* shared, float routed_scale) { VT_CHECK(expert_out.rank == 3 && weights.rank == 2 && out.rank == 2, diff --git a/src/vt/tenstorrent/tenstorrent_ops.cpp b/src/vt/tenstorrent/tenstorrent_ops.cpp index 3ec5e0be55..c8d9bdc29b 100644 --- a/src/vt/tenstorrent/tenstorrent_ops.cpp +++ b/src/vt/tenstorrent/tenstorrent_ops.cpp @@ -20,6 +20,7 @@ #include #include +#include namespace vt::tenstorrent { namespace { @@ -1693,6 +1694,89 @@ void GreedyArgmaxKernel(Queue&, Tensor& token_ids, const Tensor& logits) { CommitHost(token_ids); } +// ---- kSigmoidBiasTopK: the Kolibri-1 router ON DEVICE ------------------------ +// Composite of device ops mirroring SigmoidLogitAddRouting +// (kolibri1_shared.h / kolibri1.py:126-142; the CPU kernel in cpu_ops.cpp is +// the reference): selection score = RAW logit + bias (broadcast add), top-k +// selection via ttnn::experimental::topk_large_indices, the UNBIASED logit +// gathered at the selected ids (ttnn::gather), weight = ttnn::sigmoid. Only +// the [T,K] ids + weights cross back to the host — the full [T,E] logits +// tail NEVER downloads (the per-layer full-tail drain was the top +// serialization in the kolibri1 decode step). +// +// TIE DEVIATION (recorded): the host rule breaks an exact score tie to the +// LOWER expert index (strict `>` ascending scan). topk_large_indices packs +// (value, index) words for its TopK LLK, which also prefers the lower index +// on an exact tie, but that is the substrate's behavior, not a contract we +// own; the unit test pins the constructed-tie case against the CPU kernel. +void SigmoidBiasTopKKernel(Queue&, Tensor& weights, Tensor& indices, + const Tensor& logits, const Tensor& bias, int top_k) { + TT_OP_TRACE("SigmoidBiasTopK"); + VT_CHECK(logits.rank == 2 && weights.rank == 2 && indices.rank == 2, + "tenstorrent kSigmoidBiasTopK: rank-2 logits/weights/indices"); + VT_CHECK(logits.dtype == DType::kF32 && bias.dtype == DType::kF32 && + weights.dtype == DType::kF32 && indices.dtype == DType::kI32, + "tenstorrent kSigmoidBiasTopK: f32 logits/bias/weights, i32 indices"); + VT_CHECK(logits.IsContiguous() && bias.IsContiguous() && + weights.IsContiguous() && indices.IsContiguous(), + "tenstorrent kSigmoidBiasTopK: contiguous required"); + const uint32_t t = static_cast(logits.shape[0]); + const uint32_t e = static_cast(logits.shape[1]); + const uint32_t k = static_cast(top_k); + VT_CHECK(bias.rank == 1 && bias.shape[0] == e, + "tenstorrent kSigmoidBiasTopK: bias must be [E]"); + VT_CHECK(weights.shape[0] == t && weights.shape[1] == k && + indices.shape[0] == t && indices.shape[1] == k, + "tenstorrent kSigmoidBiasTopK: weights/indices must be [T, K]"); + + MeshDevice& device = SharedMeshDevice(); + ttnn::Tensor dev_logits = EnsureDevice2D(logits, device); // [T,E] f32 + EnsureHost(bias); + ttnn::Tensor dev_biasb = BroadcastOperandDevice(bias, t, e, device); + ttnn::Tensor scores = ttnn::add(dev_logits, dev_biasb); + // SUBSTRATE CONSTRAINT (recorded): topk_large_indices — the ONLY top-k op + // on this tt-metal — requires a ROW_MAJOR BFLOAT16 input and k in + // multiples of 16. The selection therefore rounds the biased scores to + // bf16 (the router GEMM itself is already bf16 compute; the weights stay + // f32 below) and requests the k-slot bucket, then keeps the leading + // `top_k` of the descending order. + ttnn::Tensor scores_bf16 = ttnn::typecast( + ttnn::to_layout(scores, ttnn::Layout::ROW_MAJOR), + ttnn::DataType::BFLOAT16); + uint32_t k_bucket = ((k + 15u) / 16u) * 16u; + ttnn::Tensor ids_top = ttnn::experimental::topk_large_indices( + scores_bf16, k_bucket); + ttnn::Tensor ids = + k_bucket == k + ? ids_top + : ttnn::slice(ids_top, + ttsl::SmallVector{0, 0}, + ttsl::SmallVector{static_cast(t), + static_cast(k)}, + ttsl::SmallVector{1, 1}); + // The WEIGHT reads the UNBIASED logit at each selected id (dim=1 gather), + // in f32 — the selection precision stays bf16, the weight does not. + ttnn::Tensor logits_rm = ttnn::to_layout(dev_logits, ttnn::Layout::ROW_MAJOR); + ttnn::Tensor sel = ttnn::gather(logits_rm, /*dim=*/1, ids, + /*sparse_grad=*/false, std::nullopt); + ttnn::Tensor w = ttnn::sigmoid(ttnn::to_layout(sel, ttnn::Layout::ROW_MAJOR)); + + std::vector host_ids = ids.to_vector(); + std::vector host_w = w.to_vector(); + VT_CHECK(host_ids.size() == static_cast(t) * k && + host_w.size() == static_cast(t) * k, + "tenstorrent kSigmoidBiasTopK: device returned wrong element count"); + std::memcpy(weights.data, host_w.data(), host_w.size() * sizeof(float)); + int32_t* out_ids = indices.Ptr(); + for (size_t i = 0; i < host_ids.size(); ++i) { + VT_CHECK(host_ids[i] < e, + "tenstorrent kSigmoidBiasTopK: expert id out of range"); + out_ids[i] = static_cast(host_ids[i]); + } + CommitHost(weights); + CommitHost(indices); +} + // ---- GDN prefill op set (BACKEND-TENSTORRENT-GDN W1) ------------------------- // kL2Norm / kRmsNormGated / kCausalConv1dFwd / kGdnPrefill: the op chain the // Qwen3.5-family GDN layer issues in prefill. The CPU f32 arm (cpu_ops.cpp @@ -1738,6 +1822,9 @@ struct Registrar { RegisterOp(OpId::kMoeCombine, DeviceType::kTENSTORRENT, reinterpret_cast( static_cast(&MoeCombineKernel))); + RegisterOp(OpId::kSigmoidBiasTopK, DeviceType::kTENSTORRENT, + reinterpret_cast( + static_cast(&SigmoidBiasTopKKernel))); RegisterOp(OpId::kCastBf16, DeviceType::kTENSTORRENT, reinterpret_cast(static_cast(&CastBf16Kernel))); RegisterOp(OpId::kCastF32, DeviceType::kTENSTORRENT, diff --git a/tests/vt/test_tenstorrent_backend.cpp b/tests/vt/test_tenstorrent_backend.cpp index 669d2b5fc4..eae32f09c7 100644 --- a/tests/vt/test_tenstorrent_backend.cpp +++ b/tests/vt/test_tenstorrent_backend.cpp @@ -12362,3 +12362,130 @@ TEST_CASE("kTENSTORRENT wave-3 fused Q4_K decode launch is bit-exact to the chai // Exact-f32 decode arm, single chunk. run_shape(1, 130, 2560, rng); } + +// ---- kSigmoidBiasTopK: the Kolibri-1 router on device ------------------------ +// RED-FIRST: this gate ran red before the TT kernel existed (GetOp threw "no +// kernel for SigmoidBiasTopK on kTENSTORRENT") and goes green only when the +// device composite matches the CPU reference kernel (cpu_ops.cpp +// SigmoidBiasTopKKernel, the exact port of SigmoidLogitAddRouting / +// kolibri1.py:126-142): selection score = RAW logit + bias, weight = sigmoid +// of the UNBIASED logit, no renormalization, ties to the LOWER expert index. +TEST_CASE("kTENSTORRENT kSigmoidBiasTopK matches the CPU reference " + "(selection on biased raw logits, sigmoid-of-unbiased weights)") { + if (!TenstorrentPresent()) { + MESSAGE("SKIPPED: no Tenstorrent device on this box"); + return; + } + REQUIRE(vt::OpRegistered(vt::OpId::kSigmoidBiasTopK, + DeviceType::kTENSTORRENT)); + + constexpr int64_t T = 4, E = 384, K = 6; + Backend& backend = vt::GetBackend(DeviceType::kTENSTORRENT); + Queue q = backend.CreateQueue(); + + std::mt19937 rng(20261011u); + std::uniform_real_distribution dist(-8.0f, 8.0f); + std::vector host_logits(T * E), host_bias(E); + for (auto& v : host_logits) v = dist(rng); + for (auto& v : host_bias) v = dist(rng) * 0.25f; + + auto* cpu_kernel = reinterpret_cast( + vt::GetOp(vt::OpId::kSigmoidBiasTopK, DeviceType::kCPU)); + Tensor cpu_logits = Tensor::Contiguous( + const_cast(host_logits.data()), vt::DType::kF32, + Device{DeviceType::kCPU, 0}, {T, E}); + Tensor cpu_bias = Tensor::Contiguous( + const_cast(host_bias.data()), vt::DType::kF32, + Device{DeviceType::kCPU, 0}, {E}); + std::vector cpu_w(T * K); + std::vector cpu_ids(T * K); + Tensor cpu_wt = Tensor::Contiguous(cpu_w.data(), vt::DType::kF32, + Device{DeviceType::kCPU, 0}, {T, K}); + Tensor cpu_idt = Tensor::Contiguous(cpu_ids.data(), vt::DType::kI32, + Device{DeviceType::kCPU, 0}, {T, K}); + Queue cpu_q{Device{DeviceType::kCPU, 0}, nullptr, 0}; + cpu_kernel(cpu_q, cpu_wt, cpu_idt, cpu_logits, cpu_bias, K); + + void* mem_logits = backend.Alloc(host_logits.size() * sizeof(float)); + void* mem_bias = backend.Alloc(host_bias.size() * sizeof(float)); + void* mem_w = backend.Alloc(T * K * sizeof(float)); + void* mem_ids = backend.Alloc(T * K * sizeof(int32_t)); + backend.Copy(q, mem_logits, host_logits.data(), + host_logits.size() * sizeof(float)); + backend.Copy(q, mem_bias, host_bias.data(), host_bias.size() * sizeof(float)); + Tensor dev_logits = Tensor::Contiguous( + mem_logits, vt::DType::kF32, Device{DeviceType::kTENSTORRENT, 0}, {T, E}); + Tensor dev_bias = Tensor::Contiguous( + mem_bias, vt::DType::kF32, Device{DeviceType::kTENSTORRENT, 0}, {E}); + Tensor dev_w = Tensor::Contiguous( + mem_w, vt::DType::kF32, Device{DeviceType::kTENSTORRENT, 0}, {T, K}); + Tensor dev_ids = Tensor::Contiguous( + mem_ids, vt::DType::kI32, Device{DeviceType::kTENSTORRENT, 0}, {T, K}); + vt::SigmoidBiasTopK(q, dev_w, dev_ids, dev_logits, dev_bias, K); + + std::vector tt_w(T * K); + std::vector tt_ids(T * K); + backend.Copy(q, tt_w.data(), mem_w, tt_w.size() * sizeof(float)); + backend.Copy(q, tt_ids.data(), mem_ids, tt_ids.size() * sizeof(int32_t)); + backend.Synchronize(q); + backend.Free(mem_logits); + backend.Free(mem_bias); + backend.Free(mem_w); + backend.Free(mem_ids); + + for (int64_t i = 0; i < T; ++i) { + // The selection is a SET: the device top-k must pick the SAME experts as + // the CPU reference. The ORDER may differ: the substrate selects on + // bf16-rounded scores, so two experts whose f32 scores differ slightly + // can tie in bf16 and swap positions. Order within the top-k is benign + // to the model math — the combine sums weight x expert over the k slots + // and the dispatch dedups — and the deviation is recorded in the TT + // kernel comment. + std::vector cpu_set(cpu_ids.begin() + static_cast(i * K), + cpu_ids.begin() + static_cast((i + 1) * K)); + std::vector tt_set(tt_ids.begin() + static_cast(i * K), + tt_ids.begin() + static_cast((i + 1) * K)); + std::sort(cpu_set.begin(), cpu_set.end()); + std::sort(tt_set.begin(), tt_set.end()); + REQUIRE(cpu_set.size() == K); + CHECK(tt_set == cpu_set); + for (int j = 0; j < K; ++j) { + const int32_t id = tt_ids[static_cast(i * K + j)]; + // Each returned weight is sigmoid of the UNBIASED logit AT ITS OWN id + // (the biased score selects, the unbiased logit weights), within the + // elementwise envelope (device sigmoid vs host std::exp). + const float expect = + 1.0f / (1.0f + std::exp(-host_logits[static_cast(i * E + id)])); + CHECK(std::fabs(tt_w[static_cast(i * K + j)] - expect) < 2e-3f); + // And it must equal the CPU reference weight for the SAME id. + const auto pos = std::find(cpu_ids.begin() + static_cast(i * K), + cpu_ids.begin() + static_cast((i + 1) * K), + id); + REQUIRE(pos != cpu_ids.begin() + static_cast((i + 1) * K)); + const auto cpos = static_cast(pos - cpu_ids.begin()); + CHECK(std::fabs(tt_w[static_cast(i * K + j)] - + cpu_w[static_cast(cpos)]) < 2e-3f); + } + } + + // The tie CONTRACT is on the CPU reference: two experts with an exactly + // equal biased score select the LOWER index first (host convention, pinned + // so a TT-arm tie-order deviation is at least a RECORDED deviation). + { + std::vector tl(1 * 4), tb(4, 0.0f); + tl[0] = 1.0f; tl[1] = 3.0f; tl[2] = 3.0f; tl[3] = 0.0f; // tie at 1,2 + Tensor cl = Tensor::Contiguous(tl.data(), vt::DType::kF32, + Device{DeviceType::kCPU, 0}, {1, 4}); + Tensor cb = Tensor::Contiguous(tb.data(), vt::DType::kF32, + Device{DeviceType::kCPU, 0}, {4}); + std::vector cw(1 * 2); + std::vector ci(1 * 2); + Tensor cwt = Tensor::Contiguous(cw.data(), vt::DType::kF32, + Device{DeviceType::kCPU, 0}, {1, 2}); + Tensor cit = Tensor::Contiguous(ci.data(), vt::DType::kI32, + Device{DeviceType::kCPU, 0}, {1, 2}); + cpu_kernel(cpu_q, cwt, cit, cl, cb, 2); + CHECK(ci[0] == 1); + CHECK(ci[1] == 2); + } +} From c11fdebd8948850c904a92faa527292a5a125455 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 11 Oct 2026 20:19:35 +0200 Subject: [PATCH 3/4] record: re-anchor KERNEL-ATTN-DFLASH-BLOCK after the sigmoid-bias-topk CPU arm shifted cpu_ops.cpp The new op's CPU reference arm added ~33 lines ahead of DFlashBlockAttentionKernel; the anchor ratchet correctly flagged the stale line. Re-pointed 3619 -> 3652; check-agent-record green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] --- .agents/kernel-matrix.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md index b5bbc0ba0f..ce45b4d477 100644 --- a/.agents/kernel-matrix.md +++ b/.agents/kernel-matrix.md @@ -147,7 +147,7 @@ host/sched. Detail: state `KERNEL-FA2-GQA-SWAP-FLIP`. | `KERNEL-ATTN-FLASHINFER-TRTLLM` | FlashInfer paged attention, XQA, and TensorRT-LLM generation kernels | availability/dispatch `vllm/utils/flashinfer.py:206-342,373-511`; SM90 decode and SM100+ prefill/decode gates | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | | `KERNEL-ATTN-TRITON-FLEX-HPC` | Triton, FlexAttention, TurboQuant, and HPC attention families | registry `registry.py:34-120`; Triton contract `triton_attn.py:250-383`; HPC gate `hpc_attn.py:235-247` | - | - | [inventory](specs/kernel-family-inventory.md) | `INVENTORIED` | - | | `KERNEL-ATTN-MLA-SPARSE` | MLA and sparse attention: CUTLASS, FlashMLA, FlashInfer, FA, Triton, MSA **W6: the MLA attention BLOCK + WEIGHT ABSORPTION — the layer that COMPOSES W3+W4+W5** — [mla_attention.h](../include/vllm/model_executor/models/mla_attention.h) + [mla_attention.cpp](../src/vllm/model_executor/layers/attention/mla_attention.cpp) <- `mla.py:119-181` (`MultiHeadLatentAttentionWrapper.forward`) over `mla_attention.py:553-620` (the cache-update-BEFORE-attention order), `:624-874` (`forward_impl`: the dispatch + the absorbed decode) and `:2344-2425` (`forward_mha`); `AbsorbKvBProjBf16` <- `:875-962 process_weights_after_loading` (split `:892-900`, permutes `:959-962`); `MakeMlaUpProjectFn` <- `:2141-2170` (the `kv_b_proj` callback W5 left open); `BuildDeepseekRopeCosSinCache` <- `deepseek_scaling_rope.py:76-118` over `rotary_embedding/common.py:34-70`; `MlaAttentionScale` <- `deepseek_v2.py:995,1067-1075` (the mscale^2 correction, kept SEPARATE from the rope's own rotation mscale). **Absorption is a LOAD-TIME weight transform plus TWO batched GEMMs, not a fused kernel**, so it needed only two new general primitives: **`vt::BatchedMatmul`** <- `torch.bmm` at `mla_attention.py:789` (q-side W_UK fold) and `:1034` (`_v_up_proj`), CUDA impl = cuBLASLt STRIDED-BATCHED [cuda_matmul.cu](../src/vt/cuda/cuda_matmul.cu) (the cuBLASLt form of the cuBLAS `gemmStridedBatchedEx` torch.bmm resolves to; the only upstream alternatives are ROCm-only aiter fp8/fp4 bmm branches) + CPU ref [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp), stride-driven because BOTH call sites pass `.transpose(0,1)` views; and **`vt::ConcatMlaNopeRope`** <- `ConcatMLAQKernel` (`csrc/libtorch_stable/concat_mla_q.cuh`) + wrapper `cache_kernels.cu:1555-1600`, GENERALIZED to arbitrary nope/rope widths and a head-BROADCAST rope operand so one op also serves `_concat_k_nope_k_pe` (`:2063-2092`) — CUDA [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu), CPU [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp). Two ADDITIVE relaxations of existing ops, integer-identical for contiguous tensors: `vt::RopeFromCache` stride-driven on q/k (DeepSeek rotates the TRAILING 64-dim slice and its `k_pe` is a column block of the fused kv_a projection) and `vt::MatmulBT` accepting a row-strided ACTIVATION (`kv_b_proj` applied to a 512-column slice of the 576-wide workspace, `:2160`) | CUDA priority `vllm/platforms/cuda.py:84-176` (`_get_backend_priorities`, both branches); MLA classes `vllm/v1/attention/backends/mla/*.py`; MLA prefill selector `mla/prefill/selector.py:47-76`; capability filter `vllm/v1/attention/backend.py:307-360`; CUTLASS build `CMakeLists.txt:1037-1061` **W6** [test_mla_attention_block.cpp](../tests/vllm/model_executor/layers/attention/test_mla_attention_block.cpp) **10/10 cases / 2,372,644 assertions** and [test_ops_mla_absorb.cpp](../tests/vt/test_ops_mla_absorb.cpp) **9/9 / 1,644,807 assertions** on dgx sm_121 — ports of `tests/kernels/test_concat_mla_q.py` (BOTH arms incl. the NON-CONTIGUOUS transposed-nope case, compared bit-exactly since a concat is a pure copy), the MLA-geometry sweep of `tests/v1/attention/test_mla_backends.py`, and the two-pass-oracle discipline of `tests/kernels/attention/test_mla_decode_cpu.py`. **THE ABSORBED-vs-UNABSORBED EQUIVALENCE IS PROVEN NUMERICALLY, THREE WAYS:** an INDEPENDENT double-precision block oracle computing the attention BOTH ways agrees to **< 1e-11** (the identity itself, at both query branches); our absorbed decode reproduces the UNABSORBED oracle to **< 2e-4** (f32); and the SAME batch driven once through the ABSORBED MQA decode kernel and once through the UNABSORBED materialized-MHA prefill path agrees to **< 3e-4** (CPU f32) / **< 4e-2** (CUDA bf16) — two code paths sharing nothing but the weights. Real geometry throughout (V2-Lite 512/128/64/128/16-head, plus V3's 7168 / 128-head / `q_lora_rank=1536` for the lora branch, which has NO e2e coverage and says so). Decode-only / prefill-only-no-context / chunked-prefill-with-context / MIXED (decode packed FIRST) all gated; NaN-poisoned outputs; run-to-run BIT-exact; CUDA cases proven to EXECUTE (124,941 + 290,835 assertions when run alone). memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** (the last requires `--num-cuda-barriers 65536`: the default table OVERFLOWS on a binary driving this many kernel families and the tool then emits a bogus `unspecified launch failure`). Clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). No speed number — W9 owns tuning | **SELECTION (W2) + the DECODE KERNEL (W4) + the PREFILL PATH and CHUNKED-CONTEXT LOOP (W5). What is still absent is the MLA attention BLOCK and MODEL (W6/W7).** Priority TABLE [cuda_attn_priority.h:49](../include/vllm/platforms/cuda_attn_priority.h#L49) (both branches, one row per upstream arch arm) + lookup [cuda_attn_priority.h:86](../include/vllm/platforms/cuda_attn_priority.h#L86); `is_mla()`/`is_sparse()` filter [registry.cpp:63](../src/vllm/v1/attention/registry.cpp#L63); `TritonMLABackend` NAME + 3-D `get_kv_cache_shape` [backend.h:581](../include/vllm/v1/attention/backend.h#L581), [backend.cpp:83](../src/vllm/v1/attention/backend.cpp#L83), registration [backend.cpp:108](../src/vllm/v1/attention/backend.cpp#L108) — **W4: `vt::MlaDecodeAttention`** — `OpId::kMlaDecodeAttention` + args/validation [ops.h](../include/vt/ops.h), [ops.cpp](../src/vt/ops.cpp); CPU single-pass REFERENCE [cpu_mla_attn.cpp](../src/vt/cpu/cpu_mla_attn.cpp) (numerics from `csrc/cpu/mla_decode.cpp`); CUDA two-stage split-KV [cuda_mla_attn.cu](../src/vt/cuda/cuda_mla_attn.cu) — `MlaDecodeStage1` <- `_fwd_grouped_kernel_stage1` (`triton_decode_attention.py:278-458`, IS_MLA `v = tl.trans(k)` branch `:424-431`), `MlaDecodeStage2` <- `_fwd_kernel_stage2` (`:575-639`), `ComputeNumKvSplits` <- `_compute_num_kv_splits` (`triton_mla.py:40-47`), split workspace via the house grow-only per-stream scratch (upstream's `_reserve_attn_logits_workspace` `:57-78`). Deterministic by construction: fixed ASCENDING split merge, NO atomicAdd. `TritonMLABackend::get_impl_cls()` now returns a real `TritonMLAImpl` [backend.h](../include/vllm/v1/attention/backend.h), [backend.cpp](../src/vllm/v1/attention/backend.cpp); PREFILL remains W5 and `TritonMLAImpl::forward` refuses a prefill-shaped batch by name. **W5: `vt::MlaPrefillAttention` + `vt::GatherMlaCache` + `vt::MergeAttnStates` + the chunked-context driver** — `vt::MlaPrefillAttention` [cuda_mla_prefill.cu](../src/vt/cuda/cuda_mla_prefill.cu) / CPU ref [cpu_mla_prefill.cpp](../src/vt/cpu/cpu_mla_prefill.cpp) <- `mla/prefill/flash_attn.py:153-248` `FlashAttnPrefillBackend` (the ONLY MLA prefill backend reachable on sm_121 per `mla/prefill/selector.py:66-76`, and it HARD-RAISES with no fallback at `:191-194`), running over the vendored FA-2 through the NEW launcher entry `LaunchMlaPrefillFA2Bf16` [cuda_flash_attn_fa2.cu](../src/vt/cuda/cuda_flash_attn_fa2.cu) plus two new explicit instantiations of the UNCHANGED generic template (`flash_fwd_split_hdim192_bf16{,_causal}_sm80.cu`). V is ZERO-PADDED 128->192 and the output sliced back, exactly as upstream's `requires_v_padding` path does (`flash_attn.py:88-99,164-168,196-197`) — which is WHY the asymmetric QK 192 / V 128 pair needs no asymmetric kernel. `vt::GatherMlaCache` <- `csrc/libtorch_stable/cache_kernels.cu:992-1064`; `vt::MergeAttnStates` <- `csrc/libtorch_stable/attention/merge_attn_states.cu:18-192` (BOTH `-inf` edge cases ported verbatim). The workspace-bounded loop is [mla_chunked_context.h](../include/vllm/model_executor/layers/attention/mla_chunked_context.h) <- `mla_attention.py:1422-1451,1667-1745,2094-2199,2344-2425`. **The paged launcher `LaunchPrefillFA2Bf16` that every non-MLA prefill calls is textually UNTOUCHED** (211 insertions / 0 deletions in that TU; 2 new vendored files) | [test_attn_backend_registry.cpp:146](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L146) (GB10 MLA list), [:203](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L203) (`use_mla=true` -> `TRITON_MLA`, matching the W0 oracle observation), [:230](../tests/vllm/v1/attention/test_attn_backend_registry.cpp#L230) (the DSA seam, proven both directions with a stand-in sparse backend) — ports of `test_attention_backends_selection.py` (MLA cases), `test_mla_prefill_selector.py`, `test_mla_prefill_registry.py`; **W4** [test_ops_mla_attn.cpp](../tests/vt/test_ops_mla_attn.cpp) — port of `tests/kernels/attention/test_mla_decode_cpu.py` (`ref_mla` as a TWO-PASS oracle, its bs=4/mean_seq_len=256/h_q=16/d=576/dv=512/block=16 parametrization, BOTH varlen arms, and its NaN-padding out-of-bounds detector) plus the `test_mla_backends.py` shape sweep: ragged, multi-block, single-block/single-token, EVERY num_kv_splits in {1,2,3,4,5,8,16,17,64,300,512} (incl. splits > seq_len, the empty-split path both stages must skip), 128-head DeepSeek-V3 geometry, head counts 1/3/17 that do not fill a BLOCK_H tile, a 288/256 block-32 non-V2-Lite geometry, bf16 + f32, and run-to-run BIT-exactness over 5 runs. Gated on dgx/sm_121: 11/11 cases, 2,303,193 assertions; `compute-sanitizer` memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors**; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 6/6, Qwen3-dense 16/16, OPT 6/6). NO speed number yet — decode perf is W9. **W5** [test_ops_mla_prefill.cpp](../tests/vt/test_ops_mla_prefill.cpp) **4/4 cases / 2,377,052 assertions** and [test_ops_mla_chunked_context.cpp](../tests/vt/test_ops_mla_chunked_context.cpp) **5/5 / 306,037 assertions** on dgx sm_121 — ports of `tests/v1/attention/test_mla_backends.py` and `tests/v1/attention/test_mla_prefill_quant_output.py` (its fp8 arms NOT ported: they need device-capability family 100, unreachable on sm_121 — recorded, not dropped). REAL V2-Lite prefill geometry (QK 192 / V 128 / latent 576, block 16, mscale^2 scale) against an INDEPENDENT double-precision TWO-PASS oracle, plus — for the chunked loop — a SINGLE-SHOT whole-sequence oracle that never chunks: exact / +1 / -1 chunk boundaries, a request with NO context, a chunk in which a request contributes ZERO keys, ragged multi-chunk, 128-head V3, single-token queries, ADVERSARIAL reverse-interleaved block tables, NaN-poisoned outputs, run-to-run BIT-exact over 5 runs. memcheck **0 errors**, racecheck **0 hazards**, synccheck **0 errors** on both binaries; clean CUDA build 0 warn/0 err; regression set UNCHANGED (27B 235/235, 35B 315/315, Coder 138/138, Qwen3-dense 664/664, OPT 36/36). Prefill perf is W9 | [MLA campaign spike](specs/mla-deepseek-campaign.md) | `PARTIAL` | `CLAIM-MLA-DEEPSEEK` | -| `KERNEL-ATTN-DFLASH-BLOCK` | **DFlash in-block attention — the project's FIRST non-causal / bidirectional attention primitive** (SPEC-DFLASH D2, DF-DRAFT-MODEL). Per-request uniform (1+k) query block attends within its own block: FULL-attention layers BIDIRECTIONAL (`causal=false`, no mask), SWA layers causal-within-window. f32 online softmax, GQA broadcast. A SEPARATE `vt::` op from the causal `kAttention`/`kPagedAttention` so every other model stays byte-identical | `vllm/model_executor/models/qwen3_dflash.py:86-146` (`_resolve_layer_attention`: full layers default non-causal, SWA causal) + `:149-263` (`DFlashQwen3Attention`); flashinfer non-causal path (vllm#48167 Blackwell non-causal attn, in-pin) | `OpId::kDFlashBlockAttention` + `DFlashBlockAttentionArgs` + decl [ops.h:1713](../include/vt/ops.h#L1713) + wrapper/validation [ops.cpp:2069](../src/vt/ops.cpp#L2069); CPU REFERENCE `DFlashBlockAttentionKernel` [cpu_ops.cpp:3619](../src/vt/cpu/cpu_ops.cpp#L3619) (three-pass block-local softmax, the authoritative impl); CUDA `DFlashBlockAttentionKernelCuda` [cuda_ops.cu:3031](../src/vt/cuda/cuda_ops.cu#L3031) mirroring the causal `AttentionKernel` block-reduction recurrence with per-block bounds + the bidirectional/window mask; the draft model that consumes it [qwen3_dflash.cpp:52](../src/vllm/model_executor/models/qwen3_dflash.cpp#L52) | **CPU GATE GREEN** [test_ops_dflash_block_attn.cpp:79](../tests/vt/test_ops_dflash_block_attn.cpp#L79) 5 cases / 12 assertions — hand-checked non-causal (query 0 sees the future key), the RED causal-vs-non-causal separation (the mask is load-bearing), per-request cu_seqlens block isolation, SWA window bound, GQA; model forward [test_qwen3_dflash_forward.cpp:116](../tests/vllm/models/test_qwen3_dflash_forward.cpp#L116) 5 cases / 95 assertions (RED full-layer-causal-flip); existing causal `test_ops_attention` 9/9·23 UNCHANGED. **GPU GATE GREEN on dgx (2026-07-26, GB10 sm_121a):** CUDA `-Werror=all-warnings` build clean (kernel compiles as-written, no change); CUDA==CPU parity [test_ops_dflash_block_attn CUDA case](../tests/vt/test_ops_dflash_block_attn.cpp#L153) 198412/198412 within the 1e-4 f32-softmax envelope over all 5 corners; `compute-sanitizer --tool memcheck` 0 errors; consumed by the draft-forward parity gate ([test_qwen3_dflash_draft_parity](../tests/parity/test_qwen3_dflash_draft_parity.cpp), fc rel-L2 0.46% / hidden ≤1.3% vs the real vLLM draft). **DONE 2026-07-27 with the DFlash block (`CLAIM-DFLASH-D14`):** the D2 non-causal in-block primitive is the CPU/materialized reference the D12+ paged/warp kernels are gated against; closure [ledger](parity-ledger.md#L722). | [DFlash spec](specs/dflash-spec-decode.md) §1.3/§6 D2 | `DONE` | `489a7544` | +| `KERNEL-ATTN-DFLASH-BLOCK` | **DFlash in-block attention — the project's FIRST non-causal / bidirectional attention primitive** (SPEC-DFLASH D2, DF-DRAFT-MODEL). Per-request uniform (1+k) query block attends within its own block: FULL-attention layers BIDIRECTIONAL (`causal=false`, no mask), SWA layers causal-within-window. f32 online softmax, GQA broadcast. A SEPARATE `vt::` op from the causal `kAttention`/`kPagedAttention` so every other model stays byte-identical | `vllm/model_executor/models/qwen3_dflash.py:86-146` (`_resolve_layer_attention`: full layers default non-causal, SWA causal) + `:149-263` (`DFlashQwen3Attention`); flashinfer non-causal path (vllm#48167 Blackwell non-causal attn, in-pin) | `OpId::kDFlashBlockAttention` + `DFlashBlockAttentionArgs` + decl [ops.h:1713](../include/vt/ops.h#L1713) + wrapper/validation [ops.cpp:2069](../src/vt/ops.cpp#L2069); CPU REFERENCE `DFlashBlockAttentionKernel` [cpu_ops.cpp:3652](../src/vt/cpu/cpu_ops.cpp#L3652) (three-pass block-local softmax, the authoritative impl); CUDA `DFlashBlockAttentionKernelCuda` [cuda_ops.cu:3031](../src/vt/cuda/cuda_ops.cu#L3031) mirroring the causal `AttentionKernel` block-reduction recurrence with per-block bounds + the bidirectional/window mask; the draft model that consumes it [qwen3_dflash.cpp:52](../src/vllm/model_executor/models/qwen3_dflash.cpp#L52) | **CPU GATE GREEN** [test_ops_dflash_block_attn.cpp:79](../tests/vt/test_ops_dflash_block_attn.cpp#L79) 5 cases / 12 assertions — hand-checked non-causal (query 0 sees the future key), the RED causal-vs-non-causal separation (the mask is load-bearing), per-request cu_seqlens block isolation, SWA window bound, GQA; model forward [test_qwen3_dflash_forward.cpp:116](../tests/vllm/models/test_qwen3_dflash_forward.cpp#L116) 5 cases / 95 assertions (RED full-layer-causal-flip); existing causal `test_ops_attention` 9/9·23 UNCHANGED. **GPU GATE GREEN on dgx (2026-07-26, GB10 sm_121a):** CUDA `-Werror=all-warnings` build clean (kernel compiles as-written, no change); CUDA==CPU parity [test_ops_dflash_block_attn CUDA case](../tests/vt/test_ops_dflash_block_attn.cpp#L153) 198412/198412 within the 1e-4 f32-softmax envelope over all 5 corners; `compute-sanitizer --tool memcheck` 0 errors; consumed by the draft-forward parity gate ([test_qwen3_dflash_draft_parity](../tests/parity/test_qwen3_dflash_draft_parity.cpp), fc rel-L2 0.46% / hidden ≤1.3% vs the real vLLM draft). **DONE 2026-07-27 with the DFlash block (`CLAIM-DFLASH-D14`):** the D2 non-causal in-block primitive is the CPU/materialized reference the D12+ paged/warp kernels are gated against; closure [ledger](parity-ledger.md#L722). | [DFlash spec](specs/dflash-spec-decode.md) §1.3/§6 D2 | `DONE` | `489a7544` | | `KERNEL-ATTN-DFLASH-PAGED-BLOCK` | **DFlash PAGED in-block attention — the CAPTURE-SAFE form of `KERNEL-ATTN-DFLASH-BLOCK`** (SPEC-DFLASH D12 Part B, the CUDA-graph draft-attention primitive). The (1+k) block queries attend over `[PAGED context ; their own (1+k) block]`: the growing context enters as DATA (paged K/V cache `[pages,block_size,Hkv,D]` + per-request `seq_lens` + `block_table`, mirroring `PagedAttentionKernel`) instead of a variable-size materialized combined buffer, so the launch grid is STATIC over the fixed `Nq=(1+k)*num_reqs` rows and EVERY metadata input is a persistent DEVICE tensor read in place — NO `cudaMallocAsync`/`cudaMemcpyAsync` of a function-local host `cu_seqlens` (the [[cudagraph-capture-bakes-stack-addresses]] UAF class the eager `LaunchDFlashBlockAttention` had). Same f32 online softmax + D2 in-block mask over the COMBINED index; bit-identical to `DFlashBlockAttention` over the materialized `[context;block]` buffer | vLLM full CG `dflash/cudagraph.py` + `speculator.py:411-458` + `precompute_and_store_context_kv` (`qwen3_dflash.py:548-619`) @ `555967922`; paged read mirrors our `PagedAttentionKernel` [cuda_paged_attn.cu:184](../src/vt/cuda/cuda_paged_attn.cu#L184) | `OpId::kDFlashPagedBlockAttention` + `DFlashPagedBlockAttentionArgs` + decl [ops.h](../include/vt/ops.h) + wrapper/validation [ops.cpp](../src/vt/ops.cpp); CPU REFERENCE `DFlashPagedBlockAttentionKernel` [cpu_ops.cpp](../src/vt/cpu/cpu_ops.cpp); CUDA `DFlashPagedBlockAttentionKernelCuda` [cuda_ops.cu](../src/vt/cuda/cuda_ops.cu#L3324) (static grid, persistent device metadata) + D14 WARP variant [DFlashPagedBlockAttentionWarpKernel](../src/vt/cuda/cuda_ops.cu#L1433) | **GPU GATE GREEN on dgx (2026-07-27, GB10 sm_121a):** CUDA `-Werror` clean (0 warnings); [test_ops_dflash_paged_block_attn.cpp](../tests/vt/test_ops_dflash_paged_block_attn.cpp#L79) cross-checks CPU-paged == materialized `DFlashBlockAttention` across 6 corners (non-causal, causal-SWA, block isolation, GQA, multi-page, zero-context) + CUDA==CPU (f32+bf16) = **795648/795648 assertions**; `compute-sanitizer --tool memcheck` **0 errors**. **D13 (2026-07-27, `CLAIM-DFLASH-D13`): WIRED INTO PRODUCTION** — the single-request DFlash draft forward (`ForwardPagedBody`, `qwen3_dflash.cpp`) now runs the (1+k) block through this kernel reading a fixed-capacity paged `DflashDeviceKVStore`, and the whole draft step is captured into a per-request CUDA graph + replayed (the growing context enters only via the in-place `seq_lens`). Capture-correctness PROVEN: `test_qwen27_dflash_spec_decode` 27/27 with the graph BIT-IDENTICAL to eager (same tokens + acceptance 19/39/29/25); c1 throughput NEAR-PARITY with vLLM-DFlash-ON (ours 0.978×, ~2% below the tight 3-rep band; gap closed 0.917×→0.978× via the paged read, the CG is perf-neutral) — the kernel is landed + wired + gated; STAYS `ACTIVE` with the engine feature (the ~2% ≥vLLM residual is per-step compute for an nsys). **D14 (2026-07-27, `CLAIM-DFLASH-D14`): the residual WAS this kernel → WARP-scoped variant added → SPEED GATE MET, `DONE`.** An nsys (`--cuda-graph-trace=node`) attributed the D13 ~2% residual to THIS kernel: `DFlashPagedBlockAttentionKernel` = 242.9 ms = 1.8% of the graphed step's GPU time, median ~460 us/call (grid `(nq=17,hq=32)` × kBlock=256 threads looping SERIALLY over C~500-640 keys with a 256-wide shared-mem tree reduction + 2 `__syncthreads` PER key — the latency/sync storm the ViT tower fixed with `AttentionDenseFast`), vs vLLM's fused flash draft-attn ~0.15%. Added `DFlashPagedBlockAttentionWarpKernel` ([cuda_ops.cu](../src/vt/cuda/cuda_ops.cu)): ONE WARP per (block-query,head), `__shfl_xor` butterfly head_dim reduction, register accumulator, NO `__syncthreads`; SAME paged/block combined-index read + causal/SWA mask + GQA (copied verbatim from the block kernel), mirroring the shipped `AttentionWarpKernel`. Default ON; `VT_DFLASH_ATTN_BLOCK=1` keeps the bit-identical D12/D13 block kernel. Draft attn **242.9 → 77.9 ms (3.1×)**; our-ON c1 **28.60 → 29.32 tok/s**; FINAL 3-rep A/B our-ON 29.32 ≥ vLLM-ON 29.240 (non-overlapping bands, 1.003×) ⇒ **≥vLLM MET**. Not bit-identical to the block kernel but same f32-online-softmax math within envelope; CUDA==CPU `test_ops_dflash_paged_block_attn` **795648/795648** (f32 1e-4/bf16 3e-2) + **compute-sanitizer 0**; e2e 27/27 graph==eager, acceptance 19/39/29/25 unchanged (1629 accepted identical warp-vs-block); SACRED 235/235 + MTP 9/9 inert; `-Werror` clean; closure [ledger](parity-ledger.md#L738) | [DFlash spec](specs/dflash-spec-decode.md) §0 D12/D13/D14 | `DONE` | `489a7544` | | `KERNEL-ATTN-DSA-SPARSE-INDEX` | **DeepSeek-V4 DSA "Lightning Indexer" sparse-attention SELECTION — the project's FIRST sparse candidate-selection primitive** (DeepSeek-V4-Flash W3). Two ops: (1) the weighted-MQA INDEXER LOGIT `logit[t,s] = Σ_h w[t,h]·ReLU(q[t,h]·k[s])` over the causal candidate window (the per-head **ReLU** is load-bearing — it is what makes the indexer a learned sparse SELECTOR, not a plain attention score), where `w[t,h] = weights_proj[t,h]·index_head_dim^-0.5·index_n_heads^-0.5`; and (2) the per-row **causal top-k** that keeps the `index_topk=512` highest-logit keys (short-context: every candidate, ascending; else top-k with -1 padding). Distinct from every dense/paged/MLA family, which score ALL keys — this one PICKS a sparse key subset the downstream MLA then attends over. W3 also lands the two 512-wide-MLA OUTPUT seams V2/V3 lack (per-head attention-**sink** softmax + **grouped output-LoRA** `wo_a` bmm→`wo_b`) as portable host references beside it | MQA logit `vllm/v1/attention/ops/triton_fp8_mqa_logits.py:120-156` (dot→×kv_scale→ReLU→×weights→Σheads); weight fold `vllm/model_executor/layers/sparse_attn_indexer.py:203-207`; top-k `sparse_attn_indexer.py:488-497` + short-context `vllm/models/deepseek_v4/attention.py:70-86,:813-831`; sinks `deepseek_v4/nvidia/flashinfer_sparse.py:777,:896`; grouped output-LoRA `deepseek_v4/nvidia/ops/o_proj.py:58-73` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_dsa.cpp](../src/vllm/model_executor/models/deepseek_v4_dsa.cpp) + [deepseek_v4_dsa.h](../include/vllm/model_executor/models/deepseek_v4_dsa.h): `DsaIndexerWeightFold` / `DsaIndexerLogits` / `DsaTopkSelect` / `SoftmaxWithSink` / `GroupedOutputLora` | **CPU UNIT GATE GREEN (2026-07-28, `-Wall -Werror -Wextra` 0-warn):** [test_deepseek_v4_dsa.cpp](../tests/vllm/models/test_deepseek_v4_dsa.cpp) **13/13 cases · 38 assertions** — hand-derived literal cases (the ReLU clip, the weight fold, short-context all-select, full top-k, tie→smaller-index, causal-window offset, sink probability mass, sink numerical stability, grouped-LoRA) + from-first-principles double-precision references on randomized shapes (indexer logits + grouped output-LoRA rel-L2 < 1e-6). Full-model gate is multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W3 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W3` | | `KERNEL-ATTN-DSA-COMPRESSOR` | **DeepSeek-V4 DSA COMPRESSOR + fp8_ds_mla KV-cache state — the second half of the sparse-attention stack** (DeepSeek-V4-Flash W4). Where `KERNEL-ATTN-DSA-SPARSE-INDEX` SELECTS keys, this POOLS + QUANTIZES them into the compressed latent the MLA reads and defines how it is cached across steps. Three ops: **(1)** the softmax-weighted window POOL — at a compress boundary the compressor gathers `(1+overlap)·compress_ratio` KV-state rows and computes, PER head-dim column, `softmax(score, dim=0)·kv` (each channel pools the window with its OWN weights — the load-bearing nuance), then RMSNorm; **(2)** the fused save-time APE add `score_state = score + ape[position % compress_ratio]`; **(3)** the **fp8_ds_mla** KV-cache STATE layout — the 512-wide latent split into a 448-wide NoPE part quantized to FP8 e4m3 with per-64 **UE8M0** power-of-two block scales (exponent `= ceil(log2(absmax/448))`, byte `= exp+127`) and a 64-wide RoPE part stored bf16, at a **576-byte** token stride with a padded **7+1** scale region — plus the dequant READ (`nope = e4m3·2^(byte-127)`, `rope = bf16`) | pool+RMSNorm `vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py:198-218`; save-time APE `common/ops/save_partial_states.py:92-101`; fp8_ds_mla store `fused_compress_quant_cache.py:220-297`; layout `deepseek_v4/compressor.py:307-309`; dequant READ cross-checked to SGLang `v0.5.15` `dsv4/dequant_k_cache.py:12-18,:122-136` @ `555967922` | Portable host reference (device kernel is a W7 residual) [deepseek_v4_compressor.cpp](../src/vllm/model_executor/models/deepseek_v4_compressor.cpp) + [deepseek_v4_compressor.h](../include/vllm/model_executor/models/deepseek_v4_compressor.h): `CompressorSaveScoreApe` / `CompressorPoolNorm` / `MakeFp8DsMlaLayout` / `Fp8DsMlaEncodeToken` / `Fp8DsMlaDecodeToken` | **CPU UNIT GATE GREEN (2026-07-29, Debug full-library build, 0-warn on the new TUs):** [test_deepseek_v4_compressor.cpp](../tests/vllm/models/test_deepseek_v4_compressor.cpp) **12/12 cases · 164 assertions** — hand-derived literal cases (APE modulo wrap; per-column softmax pool proven load-bearing via the column-ratio-survives-RMSNorm case; window masking; V4 layout 448/64/576/7+1; all-ones→UE8M0 byte 119 exact round-trip; value-3→byte 120; bf16 rope verbatim) + from-first-principles double-precision references (pool+norm rel-L2 < 1e-6; independent UE8M0 scale-byte recompute; encode→decode round-trip < 0.05 fp8 granularity). RED-first PROVEN: perturbing the scale bias `+127→+126` fails 4 cases / 135 assertions; revert restores 12/12. Honest gate form: hand-case + structural review vs vLLM+SGLang `file:line` (fixed-config 167B not constructible at a tiny shape ⇒ NOT a dumped-oracle rel-L2). Full-model gate multi-Spark-blocked (156.7 GiB); MHC (W5) + sqrtsoftplus/hash MoE (W6) + the fused device kernel + forward integration (W7) are named residuals | [deepseek-v4-flash spike](specs/deepseek-v4-flash.md) §W4 | `SPIKE` | `CLAIM-DEEPSEEK-V4-W4` | From c51e85ee30dc972ddf62dae3dfc9f261518cfc1e Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Sun, 11 Oct 2026 22:00:53 +0200 Subject: [PATCH 4/4] docs: classify VT_KOLIBRI1_TT_DEVICE_ROUTER kernel-internal The device-router opt-in from the kSigmoidBiasTopK commit is an engineering A/B switch, same class as the other TT diagnostic knobs on the allowlist. Pre-push checks (env-doc + agent-record) are now run before pushing. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] --- scripts/env-doc-allowlist.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/env-doc-allowlist.txt b/scripts/env-doc-allowlist.txt index b1e2e2fe88..4ab5c26ab2 100644 --- a/scripts/env-doc-allowlist.txt +++ b/scripts/env-doc-allowlist.txt @@ -124,6 +124,7 @@ VT_KOLIBRI1_PROFILE VT_KOLIBRI1_TT_B2I_PROGRESS VT_KOLIBRI1_TT_B2BI_PROGRESS VT_KOLIBRI1_TT_B2II_STREAM +VT_KOLIBRI1_TT_DEVICE_ROUTER VT_KOLIBRI1_TT_FETCH_PROFILE VT_KOLIBRI1_TT_FILL_VERIFY_ONCE VT_KOLIBRI1_TT_MEMO_FROM_SHADOW