Repository navigation
kolibri1 TT: the production-shape decode throughput bench - #3446
Merged
Merged
Conversation
tests/vllm/models/test_kolibri1_tt_decode_bench.cpp is the TT-device mirror of the CPU test_kolibri1_decode_bench production shape: load the real fp8 checkpoint once through the registry, walk the 128-token prompt one forward per step over the persistent KV cache (the engine cannot whole-prompt prefill — the routed dispatch exceeds the B1 per-step stream bound, the b2ii gate records the same refusal), then 64 free-running greedy decode steps timed end to end. Wall/prefill/decode and the full greedy chain are PRINTED, never asserted: this TU measures, the numerics adjudication stays with the operator. The per-step minimum crosses the device: vt::GreedyArgmax runs on-device and only the 8-byte token id comes back (VT_KOLIBRI1_TT_DECODE_BENCH_LOGITS_COPY=1 restores a full f32 logits row for diagnostics). The bench surfaced two engine defects, both fixed here and both filed: ISSUE-LOCAL-01M4MHZM17MAAJJF1Z071YA7R0 — a miss fetch's LRU eviction could evict an expert the SAME step's route held as a resident (recency was stamped only at fill time), tripping the miss-handle refusal on a fully warm table (observed: layer 3, experts 374 then 258). The fetch loop now refreshes same-step residents before staging the misses. ISSUE-LOCAL-01M4MJG2CKSM1FK53PZ44QFN1J — the slot-dequant memo grew without bound (7.86 MB per (layer, expert) entry) and exhausted device DRAM deterministically at prefill step ~75. Capped at 512 entries; a dropped entry re-dequants bit-identically from the readback-verified slot bytes. Gates: bench completes in both arms (host-free on: decode 65.1 s, 0.98 tok/s; host-free off: 64.3 s, 1.00 tok/s); b2ii 13/13, b2bi 10/10, tt 16/16. Registered in tests/CMakeLists.txt beside its TT siblings. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The P150 had no production-shaped throughput measurement: the B2b-ii gate is a correctness harness (its diagnostic arm runs a CPU forward + dumps per step) and the sweep produced no rate-bearing numbers because every leg failed correctness first. This adds the TT sibling of the CPU decode bench: load the real checkpoint once via ModelRegistry, walk the 128-token prompt, then 64 free-running greedy steps timed from before the first decode forward to after the last sync. Wall / prefill / decode / tok/s printed; the full greedy chain printed; NO anchor assertions (the numerics adjudication is the gate's question, this is the measurement instrument).
Loud SKIP without a TT device or VT_KOLIBRI1_TT_B2II_MODEL. Per-step host crossing is the 8-byte token id — greedy argmax runs on-device (VT_KOLIBRI1_TT_DECODE_BENCH_LOGITS_COPY=1 restores a full f32 row for diagnostics).
Two engine defects found and fixed in-flow while building it (issues filed, closed with evidence): a miss-fetch LRU could evict an expert the SAME step's route held resident (fix: refresh same-step residents' recency before staging misses), and the slot-dequant memo grew unbounded (7.86 MB per (layer,expert) entry, deterministic OOM at prefill step ~75; fix: capped at 512 entries, correctness-free eviction from readback-verified slot bytes).
First measured numbers (indicative — the B2b-ii gate stands at 26/33 pending the adjudication decision, so these are not publishable figures): host-free ON 1.04 tok/s decode, host-free OFF 0.98; prefill walks 128 tokens in ~140-150 s under the B1 per-step stream bound (the whole-prompt prefill refuses: 1.40 GB vs the 1.18 GB bound — the bound raise is recorded as the next lever).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]