Skip to content

kolibri1 TT: three eager-path levers — decode step 1077-1254 ms to 610 ms (-47%) - #3447

Merged
lu-zero merged 3 commits into
localai-org:mainfrom
lu-zero:row/tt-kolibri-capture
Oct 11, 2026
Merged

lu-zero merged 3 commits into
localai-org:mainfrom
lu-zero:row/tt-kolibri-capture

Conversation

@lu-zero

@lu-zero lu-zero commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

The captured-decode investigation (docs/gap-kolibri1-tt-captured-decode.md) concluded full trace capture is NOT feasible under ~3 days for kolibri1 — the MoE routing is host-side and data-dependent (per-layer full-tail router-logits download + streaming expert fetch with per-miss D2H readback/verify/CPU dequant), and ttnn capture is fatal across those round-trips. The missing Warm*/graph scaffold is ordinary porting but worth ~zero while the routing serializes every layer on the host. In its place, this lands the re-ranked EAGER levers, measured with 3-run medians against the bench's measured +-10% noise:

  1. Decode bench cache allocation at block_size 32 (bench only; the correctness gate keeps 16): the %32 TILE guards now pass and decode reaches median 953 ms/step (-17%, all three runs below the best baseline). The win is host-path geometry — the device attention arms still decline (the raw host NHD cache carries no rank-4 device KV shadow), recorded honestly.
  2. The streaming expert path's SECOND per-miss D2H readback (memo dequant) measured 60.5 s cumulative over a gate run — ~315 ms of every ~940 ms step. Fix: memo dequant reads the slot shadow's already-verified host image (bytes identical by construction; readbacks -> 0). Decode: median 610 ms/step (-36% vs lever-1 base, -47% vs baseline).
  3. A verified-bitmap fill-verify skip: honest no-op (0 skips fired — every fill is first-touch after eviction), kept env-gated off the critical path.
  4. Lever 2's full port scope (device-side routing via a vt::MoeRouterTopK TT port, the prerequisite for trace capture later) is documented in the gap doc rather than ported here.

A router-bias host-cache prototype from the investigation is kept (inert to numerics) but recorded as NOT a perf result: removing 50 per-layer blocking syncs moved nothing — those wait on the tiny router GEMM tail, not the queue; the serialization lives in the full-tail router-logits download and the expert-fetch round-trips.

Also recorded honestly: the greedy chains are NOT reproducible run-to-run in any arm (consistent with the B2b-ii record's 7-flip residue, adjudication pending #3438-style decision). Indicative throughput after the levers: ~1.6 tok/s decode (from ~0.93), prefill walk 149-194 -> 105-119 s.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]

…totype

BACKEND-TENSTORRENT / ISSUE-LOCAL-01M4N8Y7XPDC74SED0WEDN0YFR. The
kolibri1 P150 decode step (~1077-1254 ms, 50 layers) never captures:
docs/gap-kolibri1-tt-captured-decode.md records the precise gap against
the qwen3.5 driver (no decode-graph scaffold, no Warm* hooks, per-layer
router Download, host-driven streaming expert fetch with per-miss
readback pivot, device RAC/PA arms declined on block_size 16 vs the
%32 TILE guard). Verdict: captured decode is NOT feasible for this
model without a device-resident MoE redesign (all experts resident, a
vt::MoeRouterTopK TT port, grouped expert GEMM) — the geometry decline
itself is a config fix (cache block_size 32), not a hard tt-metal
constraint.

Prototype: the per-layer e_score_correction_bias D2H copy + blocking
Synchronize (50 per step, constant bytes) is now cached once per device
buffer in kolibri1_tt_forward.cpp. Measured with
test_kolibri1_tt_decode_bench (cherry-picked from 4dab09abd, noted as a
manual-equivalent copy in the bench header): decode 1152 ms/step vs
1077/1254 baseline across two control runs — inside the +-10%
run-to-run noise band, no win; greedy chains are unreproducible
run-to-run in every arm (the row's known 7-flip residue). Finding: the
bias syncs wait only on the router GEMM tail; the real eager levers
are the expert-fetch readback/verify cost, the per-layer full-tail
router download, and the block_size-32 + WarmRacIdx un-decline.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
@lu-zero
lu-zero force-pushed the row/tt-kolibri-capture branch from 39fa8da to fdd0f7f Compare October 11, 2026 14:07
VT_KOLIBRI1_TT_FETCH_PROFILE (the expert-fetch profiler), VT_KOLIBRI1_TT_MEMO_FROM_SHADOW (the memo-from-shadow lever) and VT_KOLIBRI1_TT_FILL_VERIFY_ONCE (the honest no-op experiment) are engineering switches from the eager-lever campaign: allowlist classification, the same one the other TT diagnostic knobs hold.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
@lu-zero
lu-zero force-pushed the row/tt-kolibri-capture branch 3 times, most recently from 3331674 to 09c82e3 Compare October 11, 2026 14:49
check-site's bijection requires every top-level docs/*.md to have a nav.yaml entry; the gap analysis is a published page.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]
@lu-zero
lu-zero merged commit e2c9501 into localai-org:main Oct 11, 2026
22 of 29 checks passed
@lu-zero
lu-zero deleted the row/tt-kolibri-capture branch October 12, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant