Repository navigation
Conversation
|
Interesting numbers. Could you run a matched A/B on current main, and say how long the other slots stall while a read owns the cards? |
007dbfd to
0c94eaf
Compare
|
Ran the matched A/B on current main. Short version: while a read owns the cards the other slots stall for the Setup: 4x RTX 5060 Ti, 60 GB RAM,
Two things I'd read off that, since they change how the trade looks:
How the arms were matched, since that was the blocker last time: both are the same tree, differing by exactly this Also, I re-ported the diff onto current main ( |
0c94eaf to
7ffab28
Compare
…dows wait Re-ported onto the tree that moved serve/ to src/models/qwen4exp/serve/: the two hunks are unchanged, only their file and class names moved. Upstream has since taken the third hunk of the original commit (the loan limit now tests `beside` instead of `batch_on()`), so what remains here is exactly the two conditions: - read_part: a split read is always exclusive (`!runtime_.stages.empty()` in the early-out, dropping `&& !batch_on()`) - lend: `beside` gains `runtime_.stages.empty()`, so an exclusive read sizes its loan and ring depth as it does on an idle engine A read beside decoding slots measured 163-174 tok/s against 3900 alone: the read's borrowed experts and the windows' hot sets evict each other at every boundary (1.38M pool serves, 117 GB of file re-reads for two 86k prompts).
7ffab28 to
7bd3a8f
Compare
What changed
On a layer split a prompt read is exclusive: the decode windows wait for it instead of running beside it. Two
lines make that true, and both are needed:
src/models/qwen4exp/serve/prompt_read.cpp—read_part's early-out becomes... || !runtime_.stages.empty() || ...src/models/qwen4exp/serve/prompt.cpp—besidegainsruntime_.stages.empty()Re-ported onto current
main:serve/moved tosrc/models/qwen4exp/serve/andRequestis nowQfnRequest.The conditions are unchanged. Upstream has meanwhile taken the third hunk of the original commit (the loan limit
now tests
besiderather thanbatch_on()), so what remains here is exactly the two.The condition it relaxes
Upstream is not unaware of exclusivity; it has a partial version.
read_partreads:that is — on a split, take the exclusive path only when no batch slots are active. With
--parallel Nand theprompt cache on,
batch_on()is true, so a split read still runs beside the decoding windows. This change dropsthe
&& !batch_on().The second line is not cosmetic.
besideselects the staging-ring depth and whether the loan is capped tokLendCapDecoding. Leaving it true on a split keeps the decode-sized loan for a read that is now exclusive, whichgives back most of what the first line buys.
Why
A read beside decoding slots is catastrophically slow on a split, because the read's borrowed expert slots and the
decode windows' hot sets evict each other at every chunk boundary.
Results — a matched A/B
4x RTX 5060 Ti (sm_120), 60 GB RAM, official
Qwen3.8-FN-NVFP4-DQ8pack,--parallel 2, int8 KV, forced split8,20,34,--prefill 12288,--resident-budget-gib 26, ctx 98304 (49152/slot). A 32,212-token prompt readarrives while a decode is running in the other slot; every streamed delta of both requests is timestamped, so the
stall is measured rather than inferred. Order alternated (base→change, change→base), two rounds per arm; ranges
are across rounds.
--spec 0(drafter off):--spec 8(drafter on), one round each:The other slots stall for the read's own duration — 10.0 s with the drafter off, 11.0 s with it on, once per
read. That buys the read 118 s → 10.5–11.5 s (≈11x).
Two things worth reading off the table:
while the read crawled. The difference is spread out vs one block, not free vs stalled. A decode that
only needs a few tokens barely notices either way (median gap 16–21 ms).
0.28–0.30 s in both arms, because prompts under
--short-readtake the decode-window path rather thanread_part. "The windows wait" should be read as "the prompt path owns the cards", not "everything waits".How the arms were matched
Both arms are the same tree; they differ by exactly this commit. The tree is current
mainplus the two flagsfrom #212 (
--layer-split,--layer-split-forced), which are needed because the price model picks the expert tieron this box (expert tier 24.4 ms vs layer split 28.3 ms per window) and there is otherwise no way to ask for a
split at all. Those flags only select the architecture — the split is forced identically in both arms and neither
touches the read path — so they cannot bias the comparison.
Note that
--parallel 2is the minimum configuration with another slot to stall: the fp16 default forces--parallel 1, where this change has nothing to act on. That is likely why the earlier numbers here could not bereproduced as a matched A/B.
Tests
None yet — the observable is a timing property, so a test wants the serve harness plus a concurrent-decode
scenario. The A/B harness used above is reproducible and is offered if it is wanted as a starting point.
Quality check
Not run: no Q8_0 reference and no
.kldbase on this machine. This changes which read path runs on a split, soboth gates are owed: greedy-identity (the single-GPU path should be bit-neutral) and PPL/KLD (the split path, since
chunking moves K/V rounding).
Development note
Engine, kernels and tests were developed with DeepSeek Flash 4.1 and GLM 5.3 Flash, and dogfooded against
Qwen3.8-Flash-Next NVFP4 served by this engine — with a human driving every decision and measurement, no
autonomous agent loops.