Price a prefill call's 204 ms fixed cost

[lvyufeng]

Summary

serving_throughput_scaling.md establishes that TTFT in the ladder grows
407-422 ms per request of wave width, and names the cause — batch_prefill runs
its requests one at a time. It never prices what a call costs, so the merge that
would fix it is quoted only as the ~1.06x in the engine's own comment.

This measures the decomposition. A prefill call costs 204 ms before it costs
anything per token, and 577-599 us per token after that. At the ladder's
322-token prompts the fixed term is 52% of the call, so merging L16's wave into
one 5162-row forward is worth 1.9-2.0x on TTFT and the same on prefill
TFLOP/s (40.5 -> ~80 on four cards), not 1.06x.

Implementation details

Three manipulations, all at one slot with --request-rate inf so nothing queues
and TTFT is the server's own prefill:

1. Call count at constant tokens. Hold the prompt at 3703 (or 1242) tokens
   and cut --prefill-token-budget to force 1/2/4/8 calls. Each extra call is
   164-245 ms; the joint fit is TTFT = 204.2 ms x n_calls + 577.4 us x tokens + 27.7.
2. Length at one call. Nine points from 67 to 3703 tokens with the budget
   above the prompt, so every point is exactly one call: intercept 205.7 ms
   (190.7 excluding the 115-token point), slope 599.1 us, residuals within
   ±81 ms. The real token counts are the engine's prompt_tokens=, not the
   harness's nominal length.
3. The page's own pf128 row, re-read. It adds two calls to each of 16
   requests for 6195.4 ms of mean TTFT. Under the bimodal structure the page
   already established — one short first request, fifteen that wait for the whole
   wave — a per-call cost k moves the mean by k x (32 x 15/16 + 2 x 1/16) = 30.1k, so 6195.4 ms is 205.7 ms a call.

The negative control rules out the collective: POCKET_ASCEND_IPC_ALLREDUCE_SKIP=1
makes all 129 all-reduces a pass a no-op and moves the server's prefill for a
1242-token prompt from 881.2 to 885.2 ms. The 276.8 ms the profiler attributes to
tp_all_reduce is latency the ranks absorb, not serialized work.

Two consequences recorded:

• The merged-forward estimate is corrected in place. qwen_engine.cpp:5091
  now says the ~1.06x is specific to 2048-token prompts and points at the ~2x
  the ladder's short prompts are worth. The old number was the right arithmetic
  for the saturation sweep's question, so it is scoped rather than deleted.
• The obstacle is unchanged and still real. A merged forward needs a
  segmented recurrent state in the 48 linear-attention layers and a block-diagonal
  mask in the 16 full-attention layers. Without the mask a 5162-row forward
  spends 16x on the dense attention matrix what sixteen 322-row forwards spend in
  total.

scripts/run_serving_sweep.sh gains a prefill subcommand that reproduces both
sweeps; the four ladder rows the decomposition is checked against are ladder's
L1/L4/L16/L32.

Testing status

• mkdocs build --strict passes; the two new intra-page anchors resolve to
  generated ids.
• bash -n scripts/run_serving_sweep.sh passes, and prefill was dry-run with a
  stubbed point to confirm the 16 tags and their per-point --prefill-token-budget.
• All figures are read from the --json-out records and the engine's own
  prompt_tokens=, never from the console table.
• Re-run on master at 4fc72a1; the only cpp_engine/ diff across that range is
  the bench's +20 phase-profile lines, and the re-run reproduces the page's own
  L1 intercept to 1% (511.7 ms against 517 ms).
• Not measured: TP1/TP2, which do not load this 52 GB bf16 checkpoint on 32 GB
  cards, so the whole record is TP4.

🤖 Generated with Claude Code