Phase 6: final benchmark vs. vLLM and SGLang

CHANGELOG.md

@@ -12,6 +12,54 @@ deletes its own evidence is not a retraction.
 No phase shipped yet (`v0.1.0` is still the scaffold version; a tag lands
 once a phase's exit criteria are actually met).
 
+- **Phase 6 (final benchmark vs. vLLM and SGLang) complete**, 2026-09-20,
+  branch `phase-6-final-benchmark` (PR pending)
+  (`docs/plans/2026-09-19-phase-6-final-benchmark-plan.md`). Ran the
+  at-scale correctness gate Phase 5b flagged as owed: naive, persistent
+  and int8 all agree with the stock reference at 1,036 positions
+  (97.1-97.3% top-1 agreement, zero large-gap disagreements against a
+  pre-registered 1.0-logit threshold), superseding Phase 5a's 29-position
+  claim for these configs — `make check` green throughout (262 tests,
+  after a whole-branch review found and fixed 20 real issues in this
+  session's own new code, none affecting the measured GPU results;
+  `docs/STATUS.md`, "Task 15 step 5").
+  **The kernel race is a real, disclosed loss**: vLLM 0.29.0 beats
+  dispatch's kernels at every one of the 28 measured (precision, token
+  count, distribution) combinations, bf16 and int8 alike, on the identical
+  GPU, model, and `torch==2.13.0`/`triton==3.7.1` build every contestant
+  shared — the gap runs 2.4-4.1x slower at 1 token, narrowing to
+  ~1.1-1.3x slower at 128 tokens and up, never closing. dispatch does beat
+  SGLang specifically at 128 and 512 tokens, both precisions — a real
+  result against one contestant, not the other. vLLM and SGLang both
+  served the real model correctly across
+  concurrency 1/4/16/64 (vLLM ~6-9% ahead of SGLang throughout, e.g.
+  1269.2 vs. 1180.4 output tok/s at concurrency 64); dispatch has no
+  served, batched engine of its own to place in that same ranked table
+  (design doc §6 amended), so its concurrency-1 numbers (naive: 21.04
+  tok/s, ~1.8x stock's 11.87, consistent with Phase 1) are reported
+  alongside it, labeled non-comparable. Full tuning of vLLM's and
+  SGLang's own MoE tuners was not run: a live check found each takes
+  ~18 minutes per token count and saves its config only after every
+  requested count finishes (confirmed by starting a real run, watching it
+  reach 29% of the first of 7 batch sizes after 5 minutes, and killing it
+  for zero usable output), so the full matrix would have cost several
+  times the phase's budget; both engines are reported at default
+  configuration only, a disclosed limitation decided with the user at a
+  $4.22 checkpoint. Six real fixes shipped mid-session: a
+  `DeepseekForCausalLM`-to-`DeepseekV2ForCausalLM` config shim both
+  tuners needed (neither recognizes the model's real V1 architecture), a
+  published `ServerArgs` SGLang's `fused_experts` requires
+  (`config namespace 'exec' not published`), a repeatability check
+  widened for `index_add_`'s non-bitwise-deterministic `atomicAdd` on
+  CUDA, a derived end-to-end latency for vLLM 0.29.0's `--save-detailed`
+  schema (which carries no per-request latency field at all), a
+  subprocess `PATH`-order fix for launching SGLang's server, and a
+  self-matching `pkill -f` pattern that had been silently killing its own
+  SSH session (fixed with the `[b]enchmark_moe.py` bracket trick). Cost:
+  **$4.5448** of a $10 cap, measured from RunPod's billing API rather
+  than rate x duration. Full account:
+  `docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md`;
+  runbook: `docs/runbooks/phase-6-final-benchmark.md`.
 - **Phase 5b (speculative decoding) complete**, 2026-09-18, merged via PR #6
   (`docs/plans/2026-09-17-phase-5b-speculative-decoding-plan.md`). A shared
   propose/verify/accept/rollback loop with two drafters (a 7B draft model,
@@ -34,9 +82,12 @@ once a phase's exit criteria are actually met).
   2/4 prompts in one process and 3/4 in another. Root-caused by a
   batch-width probe and a 10-trial determinism probe to a near-tied logit
   under the int8 kernel's floating-point precision, not a logic error.
-  Open, deliberately unaudited: Phase 5a's "perfect top-1/top-k agreement"
-  used the same kernel and a single-run check that could not see this.
-  Two findings-doc figures were also corrected after re-verifying against
+  Phase 5a's "perfect top-1/top-k agreement" used the same kernel but is a
+  small sample (29 positions, one run, `transformers==4.57.6`, so not
+  affected by the RoPE bug; its logit differences were finite and
+  non-zero, so it was not a degenerate-output pass) -- too few positions
+  to rule this sensitivity out, so Phase 6's correctness gate re-checks
+  agreement at scale rather than leaning on 5a's number. Two findings-doc figures were also corrected after re-verifying against
   the results JSONs (see the git history of the findings doc). Cost:
   **$12.26** across both sessions against a $10 cap that one pod exceeded
   and that was raised to $20 with disclosure. Also: `docs/findings/` split

CLAUDE.md

@@ -162,13 +162,53 @@ the 8-config sweep draft-model matched in 13 of 16 (prompt, k)
 combinations and prompt-lookup in 9 of 16, and the same k=4 prompt-lookup
 config matched 2/4 prompts in one process and 3/4 in another. Root cause,
 confirmed by two GPU probes: a genuine near-tied logit under the int8
-kernel's floating-point precision, not a logic bug in the loop. **Open
-question, deliberately not audited:** Phase 5a's own "perfect top-1/top-k
-agreement" claim used this same kernel and a single-run check that could
-not see this. Cost: **$12.26** across both sessions, against a $10 cap
+kernel's floating-point precision, not a logic bug in the loop. **Scope
+of Phase 5a's claim:** its "perfect top-1/top-k agreement" used this same
+kernel but is a small sample (29 positions, one run, on
+`transformers==4.57.6`, so unaffected by the RoPE bug; finite non-zero
+logit diffs, so not a degenerate-output pass) -- too few to rule out
+this sensitivity, so Phase 6's correctness gate re-checks agreement at
+scale on the benchmarked config instead of leaning on 5a's number. Cost: **$12.26** across both sessions, against a $10 cap
 that was exceeded on one pod and raised to $20 with disclosure. Full
 account: `docs/findings/phase-5b/2026-09-18-phase-5b-speculative-decoding-run.md`.
 
+**Phase 6 (final benchmark vs. vLLM and SGLang) is complete, 2026-09-20**,
+on branch `phase-6-final-benchmark` (PR pending). The at-scale correctness
+gate Phase 5b flagged as owed was run: naive, persistent and int8 all
+agree with the stock reference at 1,036 positions (97.1-97.3% top-1
+agreement), zero large-gap disagreements against a pre-registered
+1.0-logit threshold -- Phase 5a's 29-position claim is superseded by this
+number for these configs. **The kernel race is a real, disclosed loss for
+dispatch**: vLLM 0.29.0 beats dispatch's kernels at every one of the 28
+measured (precision, token count, distribution) combinations, bf16 and
+int8 alike, on the identical GPU, model, and `torch`/`triton` build as
+every contestant -- the gap is largest at 1 token (2.4-4.1x slower) and
+narrows to ~1.1-1.3x slower at 128 tokens and up, never closing. dispatch
+does beat SGLang specifically at 128 and 512 tokens, both precisions -- a
+real result against one contestant, not the other. vLLM and SGLang each
+served the real model correctly across
+concurrency 1/4/16/64 (vLLM roughly 6-9% ahead of SGLang throughout);
+dispatch has no served, batched engine of its own to place in that same
+ranked table (design doc §6 amended to say so explicitly), so its
+concurrency-1 numbers are reported alongside it, labeled non-comparable.
+Full 7-point tuning of vLLM's and SGLang's own MoE tuners was not run --
+a live check found each takes ~18 minutes per token count and saves only
+after every requested count finishes, so the full matrix would have cost
+several times the phase's budget; both engines are reported at default
+configuration only, a disclosed limitation decided with the user at a
+$4.22 checkpoint, not a fudge. Six real fixes shipped mid-session, all
+recorded in the runbook: a `DeepseekForCausalLM`-to-`DeepseekV2ForCausalLM`
+config shim both tuners needed, a published `ServerArgs` SGLang's
+`fused_experts` requires, a repeatability check widened for `index_add_`'s
+non-bitwise-deterministic `atomicAdd` on CUDA, a derived end-to-end latency
+for vLLM 0.29.0's `--save-detailed` schema (which carries no such field
+directly), a subprocess `PATH`-order fix, and a self-matching `pkill`
+pattern that had been silently killing its own SSH session. Cost: **$4.5448**
+of a $10 cap, measured from RunPod's billing API rather than rate x
+duration. Full account:
+`docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md`; runbook:
+`docs/runbooks/phase-6-final-benchmark.md`.
+
 ## One governing principle
 
 **Correctness before speed.** A kernel is not "fast" until it has been

README.md

@@ -30,9 +30,12 @@ result was a null or a mixed one, it's reported that way.
 > [#5](https://github.com/bsreecharanreddy/dispatch/pull/5) Phase 5a,
 > [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) Phase 5b) —
 > Phase 2 landed as docs only, its actual contribution being the
-> upstreamed vLLM PR below, not a dispatch-repo code change. Phase 6
-> (final benchmark vs. vLLM/SGLang) is next. Full task-by-task record:
-> [`docs/STATUS.md`](docs/STATUS.md). Design and phasing:
+> upstreamed vLLM PR below, not a dispatch-repo code change. **Phase 6
+> (final benchmark vs. vLLM/SGLang) is complete**, on branch
+> `phase-6-final-benchmark`, PR pending. This closes the system design's
+> full phase plan (0-6); only Phase 7 (productionization) remains. Full
+> task-by-task record: [`docs/STATUS.md`](docs/STATUS.md). Design and
+> phasing:
 > [`docs/design/2026-09-14-dispatch-system-design.md`](docs/design/2026-09-14-dispatch-system-design.md).
 
 ## In sixty seconds
@@ -48,7 +51,7 @@ the answer was a null or mixed result.
 serving path, an upstreamed open-source benchmark contribution, every
 rented-GPU session, and the write-ups of what broke along the way.
 
-**Measured across nine rented-GPU sessions (seven phases), $39.95 total,
+**Measured across ten rented-GPU sessions (eight phases), $44.49 total,
 every phase within its stated cap except Phase 5b, whose original $10 cap
 was exceeded on one pod and raised to $20 with disclosure:** a custom Triton kernel **~65-73% faster** than DeepSeek's own
 stock MoE forward pass at perfect logit agreement (Phase 1); an opt-in
@@ -67,9 +70,15 @@ project's own review, then root-caused to a `transformers` loading bug
 (uninitialized RoPE buffers) and re-measured — where only the model-free
 prompt-lookup drafter beat the baseline, and byte-exact agreement with
 greedy decoding turned out to be limited by near-tied logits in the int8
-kernel (Phase 5b).
-
-**170 tests, `make check` green throughout Phase 5b** (lint, `mypy --strict`,
+kernel (Phase 5b); and the head-to-head this whole project was built
+toward — **vLLM's and SGLang's own fused-MoE beat dispatch's kernel at
+its shipped default config on nearly every tested shape**, in both bf16
+and int8, on the same GPU, same model, same Triton compiler as every
+contestant, with the at-scale correctness gate this project owed since
+Phase 5a finally measured (97.1-97.3% top-1 agreement at 1,036 positions,
+zero large-gap disagreements) (Phase 6).
+
+**262 tests, `make check` green throughout Phase 6** (lint, `mypy --strict`,
 and the full non-GPU suite) — GPU-dependent tests are marked and excluded
 from CI by design, then run for real on rented hardware every phase.
 
@@ -165,11 +174,12 @@ including the ones that came back null or mixed:
 | **2 — vLLM contribution** | Do vLLM's or SGLang's official MoE benchmarks model skewed (zipf) expert load? | Neither did. Upstreamed an opt-in `--expert-load-distribution` flag to vLLM; on one rented RTX 3090, `--tune` under zipf vs. uniform routing picks a **different winning Triton config at 4 of 5 tested batch sizes**. Open: [vllm-project/vllm#57100](https://github.com/vllm-project/vllm/pull/57100) |
 | **3 — multi-GPU EP** | Does Phase 1's naive-vs-persistent crossover hold under DeepEP's real per-expert token distribution across GPUs? | **It doesn't apply at real scale.** Naive wins at every tested token count (16-2048) on H200, independent of EP. Real DeepEP dispatch across the model's 27 MoE layers produces per-local-expert counts (median 2, max ~20) far below Phase 1's smallest tested point (16) |
 | **4 — disaggregated prefill/decode** | Does separating prefill and decode across GPU pools relieve contention, holding GPU count fixed at 4? | **Mixed, not a clean result.** Disaggregated TTFT beats co-located at concurrency 4 (0.46s vs 1.12s) but loses at concurrency 8 (0.98s vs 0.70s) — most likely a kernel-warmup confound between independent process launches, reported as genuinely inconclusive rather than forced |
-| **5a — int8 quantization** | Does a self-computed, per-channel int8 weight-only kernel cut memory with no model-level correctness cost? | **Yes on both counts, no speedup.** **49.89%** expert-weight memory reduction, perfect top-1/mutual-top-k agreement vs. the naive bf16 kernel at every tested position. Throughput was a near-exact tie with naive (**-1.2%**), as expected — weight-only quantization saves memory bandwidth, not FLOPs. A real bug (quantizing without freeing the original bf16 weights, OOMing a 44GB L40) was found and fixed mid-session |
+| **5a — int8 quantization** | Does a self-computed, per-channel int8 weight-only kernel cut memory with no model-level correctness cost? | **Yes on both counts, no speedup.** **49.89%** expert-weight memory reduction, perfect top-1/mutual-top-k agreement vs. the naive bf16 kernel at every tested position (29 positions, one run -- a small sample; see 5b for the near-tie sensitivity it could not rule out). Throughput was a near-exact tie with naive (**-1.2%**), as expected — weight-only quantization saves memory bandwidth, not FLOPs. A real bug (quantizing without freeing the original bf16 weights, OOMing a 44GB L40) was found and fixed mid-session |
 | **5b — speculative decoding** | Does speculative decoding (a 7B draft model, or model-free prompt-lookup) speed up single-request decode on the int8 target while reproducing plain greedy output exactly? | **Partly, and not exactly.** A40, bf16, unbatched, baseline **25.18 tok/s**: only prompt-lookup beats it (**34.4-50.9 tok/s** across k=1-8, +77.7% at k=4); the 7B draft model is *below* baseline at every k (19.6-23.0) despite 2-7x higher acceptance. Output matched plain greedy in 13 of 16 (prompt, k) combinations for draft-model and 9 of 16 for prompt-lookup; the same k=4 config matched 2/4 prompts in one process and 3/4 in another. Root-caused to near-tied logits under the int8 kernel's floating-point precision, not a logic bug. The first session's numbers were withdrawn by this project's own review (degenerate baseline) and root-caused to a `transformers` bug leaving DeepSeek's RoPE buffers uninitialized |
+| **6 — final benchmark** | On the same L40, same model, same Triton compiler, how does dispatch's kernel compare with vLLM's and SGLang's own fused-MoE, at default and at scale, and does the correctness claim hold past a 29-position sample? | **vLLM's default config beats dispatch at every one of the 28 measured shapes**, bf16 and int8 alike -- the gap narrows from 2.4-4.1x slower at 1 token to ~1.1-1.3x slower at 128 tokens and up, but never closes (dispatch does beat SGLang specifically at 128/512 tokens, just not vLLM). The at-scale gate passed for every kernel: **97.1-97.3% top-1 agreement at 1,036 positions, zero large-gap disagreements** against a pre-registered 1.0-logit threshold — replacing Phase 5a's 29-position claim. vLLM and SGLang both served the real model correctly across concurrency 1-64 (vLLM ~6-9% ahead of SGLang throughout); dispatch's own harness has no served, batched engine to place in that same table. vLLM's and SGLang's own tuners were not run to completion (~18 min/token-count, all-or-nothing per run — confirmed live, not assumed) — both shown at default config only, disclosed rather than glossed over |
 
 Full method and every number: one folder per phase in
-[`docs/findings/`](docs/findings/) (`phase-0/` through `phase-5b/`), each
+[`docs/findings/`](docs/findings/) (`phase-0/` through `phase-6/`), each
 with a run doc and a cost doc.
 
 ## What it does not do
@@ -199,7 +209,7 @@ doc's own scope boundaries:
 | Kernel | Custom Triton grouped-GEMM (naive + persistent/cache-aware), int8 weight-only quantization (Phase 5a) |
 | Multi-GPU | DeepSeek's DeepEP — real all-to-all expert-parallel dispatch/combine over NVLink/SXM |
 | Serving | Continuous-batching prefill/decode workers, both co-located and disaggregated topologies; speculative decoding (draft-model and prompt-lookup drafters, Phase 5b) |
-| Benchmark methodology | vLLM/SGLang-style metrics — TTFT, inter-token latency, tokens/sec, cost per million tokens |
+| Benchmark methodology | Measured head-to-head against real vLLM 0.29.0 and SGLang 0.5.20, same GPU, same model, same Triton compiler (Phase 6) — TTFT, inter-token latency, tokens/sec, cost per million tokens |
 | Language | Python 3.12+ — `uv`, `ruff`, `mypy --strict`, `pytest` |
 | GPU provisioning | RunPod's API via lightweight scripts (`scripts/gpu/`) — not Terraform; see `CLAUDE.md`'s cost discipline |
 | CI | GitHub Actions — lint, `mypy --strict`, and the non-GPU suite on every push |
@@ -272,11 +282,16 @@ is the authoritative architecture, phasing, and scope document.
   and the open [vLLM PR](https://github.com/vllm-project/vllm/pull/57100): a
   real gap found by checking what a production project already ships, not
   duplicating it.
+- **Anyone who wants the head-to-head, not just the internal comparisons** —
+  [Phase 6's findings doc](docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md):
+  a real kernel race and served-engine comparison against vLLM 0.29.0 and
+  SGLang 0.5.20 on the same GPU, reported as a loss for dispatch's kernel
+  at most tested shapes, not reframed as a win.
 - **Anyone checking whether the claims hold** — [`docs/STATUS.md`](docs/STATUS.md)
   and [`docs/findings/`](docs/findings/), where a null or mixed result
   (Phase 1's persistent-kernel tie, Phase 3's crossover disappearing,
-  Phase 4's inconclusive disaggregation number) is reported as such rather
-  than smoothed over.
+  Phase 4's inconclusive disaggregation number, Phase 6's own kernel losing
+  most of the race) is reported as such rather than smoothed over.
 
 `CLAUDE.md` is instructions for AI coding assistants working in this repo,
 not a document for a human evaluating the project.