Skip to content

serve: a big prompt read on a layer split owns the cards - decode windows wait - #213

Open
gtrak wants to merge 1 commit into
jesdga95:mainfrom
gtrak:pr-big-read-exclusive
Open

gtrak wants to merge 1 commit into
jesdga95:mainfrom
gtrak:pr-big-read-exclusive

Conversation

@gtrak

@gtrak gtrak commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What changed

On a layer split a prompt read is exclusive: the decode windows wait for it instead of running beside it. Two
lines make that true, and both are needed:

  1. src/models/qwen4exp/serve/prompt_read.cpp — read_part's early-out becomes ... || !runtime_.stages.empty() || ...
  2. src/models/qwen4exp/serve/prompt.cpp — beside gains runtime_.stages.empty()

Re-ported onto current main: serve/ moved to src/models/qwen4exp/serve/ and Request is now QfnRequest.
The conditions are unchanged. Upstream has meanwhile taken the third hunk of the original commit (the loan limit
now tests beside rather than batch_on()), so what remains here is exactly the two.

The condition it relaxes

Upstream is not unaware of exclusivity; it has a partial version. read_part reads:

if (runtime_.o.batch <= 0 || runtime_.piped || (!runtime_.stages.empty() && !runtime_.batch_on()) ||
    (b0 - a0 <= runtime_.kReadAlone && resumed_chunk <= 0))

that is — on a split, take the exclusive path only when no batch slots are active. With --parallel N and the
prompt cache on, batch_on() is true, so a split read still runs beside the decoding windows. This change drops
the && !batch_on().

The second line is not cosmetic. beside selects the staging-ring depth and whether the loan is capped to
kLendCapDecoding. Leaving it true on a split keeps the decode-sized loan for a read that is now exclusive, which
gives back most of what the first line buys.

Why

A read beside decoding slots is catastrophically slow on a split, because the read's borrowed expert slots and the
decode windows' hot sets evict each other at every chunk boundary.

Results — a matched A/B

4x RTX 5060 Ti (sm_120), 60 GB RAM, official Qwen3.8-FN-NVFP4-DQ8 pack, --parallel 2, int8 KV, forced split
8,20,34, --prefill 12288, --resident-budget-gib 26, ctx 98304 (49152/slot). A 32,212-token prompt read
arrives while a decode is running in the other slot; every streamed delta of both requests is timestamped, so the
stall is measured rather than inferred. Order alternated (base→change, change→base), two rounds per arm; ranges
are across rounds.

--spec 0 (drafter off):

base (read beside) change (read exclusive)
read alone 9.82–9.87 s · 3,270–3,298 t/s 9.88–9.89 s · 3,277–3,278 t/s
read with a decode in the other slot 118.5–118.7 s · 272 t/s 10.51 s · 3,145 t/s
decoder's longest gap 1.03 s 10.01–10.08 s
decoder, total time in >1 s gaps 1.03 s 10.01–10.08 s
decoder, median gap 21 ms 16 ms

--spec 8 (drafter on), one round each:

base change
read alone 11.02 s · 2,942 t/s 11.00 s · 2,947 t/s
read with a decode in the other slot 118.31 s · 273 t/s 11.46 s · 2,891 t/s
decoder's longest gap 1.13 s 10.95 s
decoder, total in >1 s gaps 8.39 s 10.95 s

The other slots stall for the read's own duration — 10.0 s with the drafter off, 11.0 s with it on, once per
read.
That buys the read 118 s → 10.5–11.5 s (≈11x).

Two things worth reading off the table:

  • "Beside" is not free either. With the drafter on, the beside decoder accumulated 8.4 s of gaps over 1 s
    while the read crawled. The difference is spread out vs one block, not free vs stalled. A decode that
    only needs a few tokens barely notices either way (median gap 16–21 ms).
  • The stall is a property of prompt-path reads. A new short request arriving mid-read is served in
    0.28–0.30 s in both arms, because prompts under --short-read take the decode-window path rather than
    read_part. "The windows wait" should be read as "the prompt path owns the cards", not "everything waits".

How the arms were matched

Both arms are the same tree; they differ by exactly this commit. The tree is current main plus the two flags
from #212 (--layer-split, --layer-split-forced), which are needed because the price model picks the expert tier
on this box (expert tier 24.4 ms vs layer split 28.3 ms per window) and there is otherwise no way to ask for a
split at all. Those flags only select the architecture — the split is forced identically in both arms and neither
touches the read path — so they cannot bias the comparison.

Note that --parallel 2 is the minimum configuration with another slot to stall: the fp16 default forces
--parallel 1, where this change has nothing to act on. That is likely why the earlier numbers here could not be
reproduced as a matched A/B.

Tests

None yet — the observable is a timing property, so a test wants the serve harness plus a concurrent-decode
scenario. The A/B harness used above is reproducible and is offered if it is wanted as a starting point.

Quality check

Not run: no Q8_0 reference and no .kld base on this machine. This changes which read path runs on a split, so
both gates are owed: greedy-identity (the single-GPU path should be bit-neutral) and PPL/KLD (the split path, since
chunking moves K/V rounding).

Development note

Engine, kernels and tests were developed with DeepSeek Flash 4.1 and GLM 5.3 Flash, and dogfooded against
Qwen3.8-Flash-Next NVFP4 served by this engine — with a human driving every decision and measurement, no
autonomous agent loops.

@gtrak
gtrak marked this pull request as ready for review November 2, 2026 17:50
@jesdga95

jesdga95 commented Nov 8, 2026

Copy link
Copy Markdown
Owner

Interesting numbers. Could you run a matched A/B on current main, and say how long the other slots stall while a read owns the cards?

@gtrak
gtrak force-pushed the pr-big-read-exclusive branch from 007dbfd to 0c94eaf Compare November 10, 2026 16:07
@gtrak

gtrak commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Ran the matched A/B on current main. Short version: while a read owns the cards the other slots stall for the
read's whole duration — 10.0 s with the drafter off, 11.0 s with it on, for a 32k read — and that buys the read
118 s → 10.5–11.5 s.

Setup: 4x RTX 5060 Ti, 60 GB RAM, Qwen3.8-FN-NVFP4-DQ8, --parallel 2, int8 KV, split 8,20,34, --prefill 12288, resident budget 26 GiB, ctx 98304 (49152/slot). A 32,212-token prompt read arrives while a decode runs in
the other slot; every streamed delta of both requests is timestamped, so the stall is measured, not inferred.
Order alternated (base→change, change→base), two rounds per arm.

--spec 0:

base (read beside) this PR (read exclusive)
read alone 9.82–9.87 s · 3,270–3,298 t/s 9.88–9.89 s · 3,277–3,278 t/s
read with a decode in the other slot 118.5–118.7 s · 272 t/s 10.51 s · 3,145 t/s
decoder's longest gap 1.03 s 10.01–10.08 s
decoder, total in >1 s gaps 1.03 s 10.01–10.08 s
decoder, median gap 21 ms 16 ms

--spec 8, one round each: read alone 11.02 s (2,942 t/s) vs 11.00 s (2,947); read beside a decode 118.31 s
(273 t/s) vs 11.46 s (2,891 t/s); decoder's longest gap 1.13 s vs 10.95 s, and decoder time in >1 s gaps
8.39 s vs 10.95 s.

Two things I'd read off that, since they change how the trade looks:

  • Beside is not free either. With the drafter on, the beside decoder lost 8.4 s to gaps over 1 s while the read
    crawled. So it's spread-out pain vs one block, not free vs stalled — and a decode that only needs a few tokens
    barely notices either way (median gap 16–21 ms).
  • The stall belongs to prompt-path reads. A new short request arriving mid-read is served in 0.28–0.30 s in
    both arms, because prompts under --short-read take the decode-window path, not read_part. So it's "the prompt
    path owns the cards", not "everything waits".

How the arms were matched, since that was the blocker last time: both are the same tree, differing by exactly this
commit. The tree is current main plus #212's --layer-split / --layer-split-forced, which are needed because the
price model picks the expert tier on this box (24.4 vs 28.3 ms per window) and there is otherwise no way to ask for
a split at all — those flags only select the architecture, the split is forced identically in both arms, and
neither touches the read path. --parallel 2 is also the minimum with another slot to stall: the fp16 default
forces --parallel 1, where this change has nothing to act on. That's likely why the old numbers couldn't be
reproduced as a matched A/B.

Also, I re-ported the diff onto current main (serve/ → src/models/qwen4exp/serve/, Request → QfnRequest);
the two conditions are unchanged, and upstream has already taken the third hunk of the original commit.

…dows wait

Re-ported onto the tree that moved serve/ to src/models/qwen4exp/serve/: the two
hunks are unchanged, only their file and class names moved. Upstream has since
taken the third hunk of the original commit (the loan limit now tests `beside`
instead of `batch_on()`), so what remains here is exactly the two conditions:

- read_part: a split read is always exclusive (`!runtime_.stages.empty()` in the
  early-out, dropping `&& !batch_on()`)
- lend: `beside` gains `runtime_.stages.empty()`, so an exclusive read sizes its
  loan and ring depth as it does on an idle engine

A read beside decoding slots measured 163-174 tok/s against 3900 alone: the
read's borrowed experts and the windows' hot sets evict each other at every
boundary (1.38M pool serves, 117 GB of file re-reads for two 86k prompts).
@gtrak
gtrak force-pushed the pr-big-read-exclusive branch from 7ffab28 to 7bd3a8f Compare November 19, 2026 19:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants