Repository navigation
kolibri1 TT: three eager-path levers — decode step 1077-1254 ms to 610 ms (-47%) - #3447
Merged
Merged
Conversation
…totype BACKEND-TENSTORRENT / ISSUE-LOCAL-01M4N8Y7XPDC74SED0WEDN0YFR. The kolibri1 P150 decode step (~1077-1254 ms, 50 layers) never captures: docs/gap-kolibri1-tt-captured-decode.md records the precise gap against the qwen3.5 driver (no decode-graph scaffold, no Warm* hooks, per-layer router Download, host-driven streaming expert fetch with per-miss readback pivot, device RAC/PA arms declined on block_size 16 vs the %32 TILE guard). Verdict: captured decode is NOT feasible for this model without a device-resident MoE redesign (all experts resident, a vt::MoeRouterTopK TT port, grouped expert GEMM) — the geometry decline itself is a config fix (cache block_size 32), not a hard tt-metal constraint. Prototype: the per-layer e_score_correction_bias D2H copy + blocking Synchronize (50 per step, constant bytes) is now cached once per device buffer in kolibri1_tt_forward.cpp. Measured with test_kolibri1_tt_decode_bench (cherry-picked from 4dab09abd, noted as a manual-equivalent copy in the bench header): decode 1152 ms/step vs 1077/1254 baseline across two control runs — inside the +-10% run-to-run noise band, no win; greedy chains are unreproducible run-to-run in every arm (the row's known 7-flip residue). Finding: the bias syncs wait only on the router GEMM tail; the real eager levers are the expert-fetch readback/verify cost, the per-layer full-tail router download, and the block_size-32 + WarmRacIdx un-decline. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/tt-kolibri-capture
branch
from
October 11, 2026 14:07
39fa8da to
fdd0f7f
Compare
VT_KOLIBRI1_TT_FETCH_PROFILE (the expert-fetch profiler), VT_KOLIBRI1_TT_MEMO_FROM_SHADOW (the memo-from-shadow lever) and VT_KOLIBRI1_TT_FILL_VERIFY_ONCE (the honest no-op experiment) are engineering switches from the eager-lever campaign: allowlist classification, the same one the other TT diagnostic knobs hold. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
force-pushed
the
row/tt-kolibri-capture
branch
3 times, most recently
from
October 11, 2026 14:49
3331674 to
09c82e3
Compare
check-site's bijection requires every top-level docs/*.md to have a nav.yaml entry; the gap analysis is a published page. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The captured-decode investigation (docs/gap-kolibri1-tt-captured-decode.md) concluded full trace capture is NOT feasible under ~3 days for kolibri1 — the MoE routing is host-side and data-dependent (per-layer full-tail router-logits download + streaming expert fetch with per-miss D2H readback/verify/CPU dequant), and ttnn capture is fatal across those round-trips. The missing Warm*/graph scaffold is ordinary porting but worth ~zero while the routing serializes every layer on the host. In its place, this lands the re-ranked EAGER levers, measured with 3-run medians against the bench's measured +-10% noise:
A router-bias host-cache prototype from the investigation is kept (inert to numerics) but recorded as NOT a perf result: removing 50 per-layer blocking syncs moved nothing — those wait on the tiny router GEMM tail, not the queue; the serialization lives in the full-tail router-logits download and the expert-fetch round-trips.
Also recorded honestly: the greedy chains are NOT reproducible run-to-run in any arm (consistent with the B2b-ii record's 7-flip residue, adjudication pending #3438-style decision). Indicative throughput after the levers: ~1.6 tok/s decode (from ~0.93), prefill walk 149-194 -> 105-119 s.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]