qwen36 tier: re-plan the VRAM residents from the prompt's own prefill routing (QT_PREFILL_REPLAN=1)

[kreuzzelg]

The warmstart fills VRAM from the heat file, i.e. from what earlier prompts routed to, and the LFRU tick then corrects one expert per sixteen tokens and device. For a prompt on a new topic that is thousands of tokens of catching up. Measured on the 35B at 37 % residency: a heat file from six other prompts gives a fresh prompt 55-63 % decode hits, a cold start 38 %, a heat file from the same prompt 89-91 % -- the heat warmstart is an oracle for a repeated prompt and a stranger's guess for a new one.

The prefill knows better, because its routing is this prompt's routing. Offline (14 route traces, four held-out prompts), a static set of B experts per layer chosen from the prompt's own prefill counts covers its decode routing far better than the heat file at the same budget: at B = 96 of 256 (the 8 GB card's share) 0.741 against 0.593, oracle 0.890; at B = 32, 0.421 against 0.217. Mixing in heat does not help. This is the prefill-as-probe idea from #1078 without the Atlas: the prompt's own counts are finer than any cluster, and there is nothing pre-fitted to leak the evaluation set (the concern raised there).

What the PR does

Tier. qt_replan(layer, counts, cap): given one layer's routing counts over the prompt rows, residents the prompt never routed to are swapped, budget-neutral, for the prompt's most-routed non-residents of that layer -- the same victim-first swap the LFRU tick uses, in strict count order, only while the newcomer's count beats the victim's. As many swaps as the upload queue takes start at once; the rest are pending and drain on later calls and, four per token, on the decode ticks. Nothing blocks. qt_evicted_take reports evicted experts, and qt_stats_mark lets qt_stats print the hit rate counted from the first decode token (decode VRAM hit rate, the number that moves).

Engine. QT_PREFILL_REPLAN=1 (opt-in): after each prefill layer's routing loop moe() hands that layer's counts to qt_replan, so the swaps of layer L upload while layers L+1.. still compute. QT_PREFILL_REPLAN_MAX caps swaps per layer (default 24, see below). At the start of every step the engine rebuilds the RAM int8 copies of the experts the tier evicted, in parallel, before the layers run: on the int4 container the warmstart freed them, and left to the miss path 2,000 victims cost ~7 ms/token over the following 300 tokens (slot_ensure_int8 inside the decode loop). Routing is untouched: placement never changes which experts run.

Backend. coli_cuda_tensor_overwrite: every expert of a model has the same geometry, so a swap writes the newcomer into the victim's device buffers instead of cudaFree + cudaMalloc, each of which synchronises the device under the async expert groups. Optional in the DLL loader (RESOLVE_OPT); a backend without the entry point, or a failure half-way, falls back to free-then-upload. The fake backend implements it and can refuse it, so the fallback is tested.

Measured (RTX 3070 alone, per-row int4 35B, 200-250-token prompts, 300 decoded tokens, heat file from six other prompts, cap 24 = 960 swaps per prompt)

prompt	trunk	decode VRAM hit rate	MoE ms/token	total ms/token	TTFT
chat	CPU	54.8 -> 72.8 %	18.8 -> 15.2	67.7 -> 68.5	3.9 -> 5.0 s
code	CPU	59.9 -> 75.1 %	18.0 -> 14.5	69.5 -> 66.8	5.1 -> 5.4 s
Chinese	CPU	62.7 -> 74.8 %	16.8 -> 14.6	68.8 -> 65.8	4.0 -> 5.0 s
reasoning	CPU	63.1 -> 75.3 %	17.4 -> 15.9	70.4 -> 71.6	4.4 -> 5.5 s
chat	VRAM (all five)	41.8 -> 61.7 %	22.9 -> 18.7	49.2 -> 45.2	6.4 -> 6.6 s
code	VRAM	45.1 -> 64.8 %	22.3 -> 17.5	49.0 -> 44.1	8.1 -> 8.2 s
Chinese	VRAM	49.6 -> 60.3 %	20.4 -> 20.4	46.3 -> 47.8	6.2 -> 7.0 s
reasoning	VRAM	44.5 -> 66.7 %	22.4 -> 17.6	48.7 -> 44.4	7.0 -> 7.5 s

Hit rate +12 to +22 points, as the offline table predicted; MoE phase -2 to -5 ms/token; with the trunk in VRAM that is 8-10 % of the token on three prompts of four (the fourth sits in the ±2 ms run-to-run noise). With the trunk on the CPU the saving disappears in the 50 ms the dense trunk costs there.

The price, honestly. The swaps upload during the prefill, and each one is six synchronous host-to-device copies (three matrices, three scale vectors), about 1 ms on this box. Uncapped (~1,500 swaps) that was +1.3-1.8 s of TTFT for the same hit rate; capped at 24 per layer (960 swaps) it is +0.1-1.1 s, paid back by the decode after 100-250 tokens. The in-place overwrite did not change that number -- the cost was never the allocation, it is the copies serialising with the prefill's own expert groups. Offline at B = 96: cap 12 per layer (480 swaps) reaches 0.709 of the uncapped 0.741, cap 24 reaches 0.741, a minimum-count threshold buys nothing -- hence the default. The follow-up that would take the TTFT price away is a pinned, asynchronous staging path on a non-blocking stream, or one contiguous device block per expert; both are backend work I would rather do as their own PR.

Two things I measured on the way and did not ship: draining the swaps on the decode ticks instead of the prefill (48 per token) made decode slower by 7 ms/token while they drained, because each swap was then a cudaFree/cudaMalloc under the async groups -- that is why the tick drains four and why the overwrite exists; and without the evicted-expert rebuild the hit-rate gain was eaten whole by slot_ensure_int8 in the miss path.

Verified

• tests/test_qwen36_tier_replan.c (fake backend, in make check): counts order the plan and only a newcomer that beats its victim is swapped; a cap holds; a layer plan leaves other layers alone while the whole-table form trades across layers; swaps are budget-neutral and in place (the newcomer takes the victim's tensor pointers, no new upload); evicted experts are reported once, with their layer; with overwrite refused the swap still lands by free-then-upload; qt_stats_mark snapshots.
• The existing tier tests (withdraw, fill_wait, invariants, int8_engine, ...) unchanged and green; CPU-only build (make qwen36) carries the inline stubs; colibri (the DLL loader) builds.
• Real container, QT_PREFILL_REPLAN=0 against =1: the tokens are not byte-identical (the eight pairs above diverge between byte 8 and byte 767 of the text). That is the non-identity any change of residency has today -- a different heat file, an LFRU swap -- because a resident expert (GPU kernel) and a CPU miss (int8 slot) do not sum in the same order and a greedy route flips; measured in qwen36: offer the rest of the dense trunk to the VRAM placer (dnout, attnproj, shexp) #1657's thread, two runs of the same configuration with a frozen heat file are byte-identical, two configurations are not. Routing itself is untouched: the re-plan only decides where an expert is computed.

Logs, traces (2.6 GB) and the offline scripts are in my workspace; happy to attach any of them.

Thanks to Claude Code for the help along the way.