Phase 6: final benchmark vs. vLLM and SGLang

[bsreecharanreddy]

Summary

• At-scale correctness gate (owed since Phase 5b flagged Phase 5a's 29-position claim as too small): naive, persistent, and int8 all agree with the stock reference at 1,036 real-tokenizer positions -- 97.1-97.3% top-1 agreement, zero large-gap disagreements against the pre-registered 1.0-logit threshold. A stock-vs-stock control at the same 1,036 positions has zero disagreements of any kind, proving the measurement process itself introduces no spurious flips.
• Kernel-level race: dispatch's naive and persistent Triton grouped-GEMM kernels vs. vLLM 0.29.0's and SGLang 0.5.20's own fused-MoE, on identical seeded inputs, the same GPU/model/torch==2.13.0/triton==3.7.1 build, bf16 and int8, uniform and zipf routing, 7 token counts (1-2048). Zero results refused across all 224 combinations run.
• Labeled non-comparable engine-reference serving comparison: vLLM and SGLang each served the real model end-to-end at concurrency 1/4/16/64. dispatch has no serving stack of its own (system design §6, amended to say so explicitly); its concurrency-1 numbers are reported alongside, not folded into the ranked table.
• Six real fixes shipped mid-session (all in the runbook): a DeepseekForCausalLM->DeepseekV2ForCausalLM config shim both tuners needed, a published ServerArgs SGLang's fused_experts requires, a repeatability check widened for index_add_'s non-bitwise-deterministic atomicAdd on CUDA, a derived end-to-end latency for vLLM's --save-detailed schema (which carries no such field directly), a subprocess PATH-order fix, and a self-matching pkill pattern that was silently killing its own SSH session.
• Whole-branch review, done as a separate follow-up pass: found and fixed 20 real issues in this session's own new code (not the paid GPU results, which stand as measured) -- a reachable uncaught crash (dispatch-persistent + int8), a self-poisoning merge glob, several unguarded exceptions (ZeroDivisionError, StatisticsError, IndexError, raw KeyError) where the code's own docstrings promised a clean refusal, a silent no-op gate mode, non-deterministic table ordering, and a couple of duplicated-logic cleanups. All fixed, each its own commit -- see docs/STATUS.md, "Task 15 step 5" for the full list with commit hashes.
• A wrong headline claim, caught and corrected while writing this PR: the original write-up said "dispatch's naive kernel wins at int8 128/512 tokens." Checking the raw measured JSON directly found that claim compared dispatch only against SGLang and dropped vLLM out of the comparison. Corrected across the findings doc, CLAUDE.md, README.md, STATUS.md, and CHANGELOG.md -- see "Results" below and the last commit for the full account.

Cost: $4.5448 of the $10 cap, measured from RunPod's billing API rather than rate x duration.

Results

Correctness gate (1,036 positions each):

config	disagreements	large-gap	widest gap	top-1 agreement
stock vs. stock (control)	0	0	0.000	100.000%
naive (bf16)	28	0	0.750	97.297%
persistent (bf16)	30	0	0.750	97.104%
int8 (quantized)	29	0	0.750	97.201%

Kernel race — vLLM beats dispatch at every one of the 28 measured (precision, token count, distribution) combinations, bf16 and int8 alike. Gap by token count (dispatch's best tuned variant vs. vLLM):

tokens	bf16 ratio (dispatch/vLLM)	int8 ratio (dispatch/vLLM)
1	2.4x slower	3.5-4.1x slower
16	1.3x slower	1.7x slower
128	1.1x slower	1.3x slower
2048	1.2-1.3x slower	1.2-1.3x slower

dispatch does beat SGLang specifically, at 128 and 512 tokens in both bf16 and int8 (e.g. int8 128 tokens: dispatch 1.628ms vs. SGLang 2.111ms vs. vLLM 1.232ms) -- a real result against one contestant, not the other.

Engine reference (served, not raced):

engine	concurrency	out tok/s	p50 TTFT (ms)	$/M output tok
vLLM	1	115.5	37.9	1.97
vLLM	64	1269.2	174.1	0.18
SGLang	1	108.6	54.9	2.10
SGLang	64	1180.4	135.8	0.19
dispatch (naive, concurrency-1 only, non-comparable)	1	21.04	--	--

vLLM leads SGLang by roughly 6-9% throughout. dispatch's own concurrency-1 number is a genuine ~1.8x speedup over the stock model's eager moe_infer (21.04 vs. 11.87 tok/s), consistent with Phase 1, but not placed in the ranked table above: no HTTP layer, no CUDA-graph capture, no continuous batching -- comparing it ms-for-ms against a served number would compare two different things under one column header.

Things worth a reviewer's attention

• This is a real, disclosed loss for dispatch's kernel, not a caveat under a positive headline. The project's own correctness-before-speed and never-quote-an-unmeasured-number rules exist for exactly this outcome.
• A wrong comparative claim slipped into the original write-up ("dispatch wins at int8 128/512") and sat in five committed files (findings doc, CLAUDE.md, README.md, STATUS.md, CHANGELOG.md) before being caught during this PR's own preparation, by recomputing the winner at every cell directly from the raw per-engine JSON rather than trusting the hand-written prose. Fixed in the last commit; no measured numbers changed, only the narrative built on top of them.
• Full 7-point tuning of vLLM's and SGLang's own MoE tuners was not run -- confirmed live to cost ~18 minutes per token count with no partial credit on an early stop, several times the phase's budget. Both engines are reported at shipped default configuration only, a disclosed limitation decided with the user at a $4.22 checkpoint.
• dispatch has no served, batched engine of its own to place in the same ranked concurrency table as vLLM/SGLang -- design doc §6 amended to say so explicitly rather than imply an apples-to-apples comparison.
• Three commits from the original session (3acf14c, c31d1f4, 2536764) broke this repo's own same-commit-STATUS.md rule. Left as a documented gap in STATUS.md rather than rewritten via history rewriting, since nothing is pushed anywhere else and the risk of rewriting outweighed the paperwork gap itself.

Test plan

• make check: lint clean, mypy --strict clean, 262 passed / 3 skipped / 2 deselected (pytest -m "not gpu")
• Real GPU session: at-scale correctness gate, kernel race (7 token counts x 2 distributions x 2 precisions x 4 contestants), and the vLLM/SGLang serving reference, all recorded under docs/findings/phase-6/ with raw per-engine JSON evidence
• Whole-branch review (20 findings, all fixed) plus a raw-data recheck of every headline comparative claim before opening this PR