Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,36 @@ deletes its own evidence is not a retraction.
No phase shipped yet (`v0.1.0` is still the scaffold version; a tag lands
once a phase's exit criteria are actually met).

- **Phase 5b (speculative decoding) complete**, 2026-09-18, merged via PR #6
(`docs/plans/2026-09-17-phase-5b-speculative-decoding-plan.md`). A shared
propose/verify/accept/rollback loop with two drafters (a 7B draft model,
and model-free prompt-lookup) on Phase 5a's int8 target, with an
independent plain-greedy oracle for the baseline — `make check` green
throughout (170 tests). **The first GPU session's numbers (+21.8% /
+76.9%) were withdrawn** by the final whole-branch review: its baseline
was a degenerate 64 x token-id-0 on every prompt. Root cause, found on a
second session across four pods: `transformers==5.17.0` leaves
DeepSeek's remote-code RoPE `inv_freq` buffer uninitialized after
`from_pretrained`, poisoning attention with NaN on GPU;
`fix_rope_inv_freq()` now runs inside `load_model` for every caller.
Re-measured on an A40 (bf16, unbatched, 64 new tokens, 4 prompts x 5
reps), checked against the baseline at every k: baseline **25.18
tok/s**; **only prompt-lookup beats it** (34.4-50.9 tok/s across k=1-8,
+77.7% at k=4 in the sweep), while the 7B draft model is below baseline
at every k (19.6-23.0). **Output is not always byte-exact against plain
greedy**: draft-model matched in 13 of 16 (prompt, k) combinations,
prompt-lookup in 9 of 16, and the same k=4 prompt-lookup config matched
2/4 prompts in one process and 3/4 in another. Root-caused by a
batch-width probe and a 10-trial determinism probe to a near-tied logit
under the int8 kernel's floating-point precision, not a logic error.
Open, deliberately unaudited: Phase 5a's "perfect top-1/top-k agreement"
used the same kernel and a single-run check that could not see this.
Two findings-doc figures were also corrected after re-verifying against
the results JSONs (see the git history of the findings doc). Cost:
**$12.26** across both sessions against a $10 cap that one pod exceeded
and that was raised to $20 with disclosure. Also: `docs/findings/` split
into one subfolder per phase. Full account:
`docs/findings/phase-5b/2026-09-18-phase-5b-speculative-decoding-run.md`.
- **Phase 5a (int8 weight-only quantization) complete**, 2026-09-17
(`docs/plans/2026-09-16-phase-5a-quantization-plan.md`). Self-computed,
per-output-channel int8 quantization extending the Triton kernel
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,8 @@ Community Cloud pods that hit a real host-level GPU passthrough bug
before a Secure Cloud pod worked. Full account:
`docs/findings/phase-5a/2026-09-17-phase-5a-quantization-run.md`.

**Phase 5b (speculative decoding) is complete, 2026-09-18**, PR #6 open on
`phase-5b-speculative-decoding`. A shared propose/verify/accept/rollback
**Phase 5b (speculative decoding) is complete, 2026-09-18**, merged to
`main` via PR #6. A shared propose/verify/accept/rollback
loop with two drafters (a 7B draft model, and model-free prompt-lookup) on
top of Phase 5a's int8 target, with an independent plain-greedy oracle for
the baseline. The first GPU session's numbers were withdrawn by the final
Expand Down
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,16 @@ result was a null or a mixed one, it's reported that way.

---

> **Status:** Phases 0-5a merged to `main` across five PRs
> **Status:** Phases 0-5b merged to `main` across six PRs
> ([#1](https://github.com/bsreecharanreddy/dispatch/pull/1) Phase 0,
> [#2](https://github.com/bsreecharanreddy/dispatch/pull/2) Phase 1,
> [#3](https://github.com/bsreecharanreddy/dispatch/pull/3) Phases 2-3,
> [#4](https://github.com/bsreecharanreddy/dispatch/pull/4) Phase 4,
> [#5](https://github.com/bsreecharanreddy/dispatch/pull/5) Phase 5a) —
> [#5](https://github.com/bsreecharanreddy/dispatch/pull/5) Phase 5a,
> [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) Phase 5b) —
> Phase 2 landed as docs only, its actual contribution being the
> upstreamed vLLM PR below, not a dispatch-repo code change. **Phase 5b
> (speculative decoding) is complete**, PR
> [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) open. Full
> task-by-task record:
> upstreamed vLLM PR below, not a dispatch-repo code change. Phase 6
> (final benchmark vs. vLLM/SGLang) is next. Full task-by-task record:
> [`docs/STATUS.md`](docs/STATUS.md). Design and phasing:
> [`docs/design/2026-09-14-dispatch-system-design.md`](docs/design/2026-09-14-dispatch-system-design.md).

Expand Down
5 changes: 3 additions & 2 deletions docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -498,8 +498,9 @@ mode. Re-auditing Phase 5a was out of this task's scope.

Phase 5b is done: root cause fixed, real numbers measured, both
correctness gates and the full k-sweep checked and explained. Phase 4 is
merged (PR #4). Phase 5a and Phase 5b are both ready to push as PRs per
the one-branch-per-phase convention. Phase 3's PR (#3) already merged.
merged (PR #4), as are Phase 5a (PR #5) and Phase 5b (PR #6), each on its
own branch per the one-branch-per-phase convention. Phase 3's PR (#3) also
merged.
Phase 2's PR is still open and awaiting maintainer review; no further
work planned on it beyond responding to review feedback. Phase 6 (final
benchmark vs. vLLM/SGLang) planning can proceed, and should treat the
Expand Down
Loading