From 6ac4f1320023a282a20a2acc9d9e55ff820d4911 Mon Sep 17 00:00:00 2001 From: bsreecharanreddy <283869510+bsreecharanreddy@users.noreply.github.com> Date: Fri, 18 Sep 2026 16:28:01 -0400 Subject: [PATCH] docs: record Phase 5b as merged, add its missing CHANGELOG entry README, CLAUDE.md, and STATUS.md still described PR #6 as open. The CHANGELOG had no Phase 5b entry at all; adds one, keeping the withdrawn first-session numbers and their correction in the record per the changelog's own rule. --- CHANGELOG.md | 30 ++++++++++++++++++++++++++++++ CLAUDE.md | 4 ++-- README.md | 11 +++++------ docs/STATUS.md | 5 +++-- 4 files changed, 40 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 05ea933..1b5f65e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,36 @@ deletes its own evidence is not a retraction. No phase shipped yet (`v0.1.0` is still the scaffold version; a tag lands once a phase's exit criteria are actually met). +- **Phase 5b (speculative decoding) complete**, 2026-09-18, merged via PR #6 + (`docs/plans/2026-09-17-phase-5b-speculative-decoding-plan.md`). A shared + propose/verify/accept/rollback loop with two drafters (a 7B draft model, + and model-free prompt-lookup) on Phase 5a's int8 target, with an + independent plain-greedy oracle for the baseline — `make check` green + throughout (170 tests). **The first GPU session's numbers (+21.8% / + +76.9%) were withdrawn** by the final whole-branch review: its baseline + was a degenerate 64 x token-id-0 on every prompt. Root cause, found on a + second session across four pods: `transformers==5.17.0` leaves + DeepSeek's remote-code RoPE `inv_freq` buffer uninitialized after + `from_pretrained`, poisoning attention with NaN on GPU; + `fix_rope_inv_freq()` now runs inside `load_model` for every caller. + Re-measured on an A40 (bf16, unbatched, 64 new tokens, 4 prompts x 5 + reps), checked against the baseline at every k: baseline **25.18 + tok/s**; **only prompt-lookup beats it** (34.4-50.9 tok/s across k=1-8, + +77.7% at k=4 in the sweep), while the 7B draft model is below baseline + at every k (19.6-23.0). **Output is not always byte-exact against plain + greedy**: draft-model matched in 13 of 16 (prompt, k) combinations, + prompt-lookup in 9 of 16, and the same k=4 prompt-lookup config matched + 2/4 prompts in one process and 3/4 in another. Root-caused by a + batch-width probe and a 10-trial determinism probe to a near-tied logit + under the int8 kernel's floating-point precision, not a logic error. + Open, deliberately unaudited: Phase 5a's "perfect top-1/top-k agreement" + used the same kernel and a single-run check that could not see this. + Two findings-doc figures were also corrected after re-verifying against + the results JSONs (see the git history of the findings doc). Cost: + **$12.26** across both sessions against a $10 cap that one pod exceeded + and that was raised to $20 with disclosure. Also: `docs/findings/` split + into one subfolder per phase. Full account: + `docs/findings/phase-5b/2026-09-18-phase-5b-speculative-decoding-run.md`. - **Phase 5a (int8 weight-only quantization) complete**, 2026-09-17 (`docs/plans/2026-09-16-phase-5a-quantization-plan.md`). Self-computed, per-output-channel int8 quantization extending the Triton kernel diff --git a/CLAUDE.md b/CLAUDE.md index bc19e93..1865dd8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,8 +145,8 @@ Community Cloud pods that hit a real host-level GPU passthrough bug before a Secure Cloud pod worked. Full account: `docs/findings/phase-5a/2026-09-17-phase-5a-quantization-run.md`. -**Phase 5b (speculative decoding) is complete, 2026-09-18**, PR #6 open on -`phase-5b-speculative-decoding`. A shared propose/verify/accept/rollback +**Phase 5b (speculative decoding) is complete, 2026-09-18**, merged to +`main` via PR #6. A shared propose/verify/accept/rollback loop with two drafters (a 7B draft model, and model-free prompt-lookup) on top of Phase 5a's int8 target, with an independent plain-greedy oracle for the baseline. The first GPU session's numbers were withdrawn by the final diff --git a/README.md b/README.md index 8847f7e..61c7cc1 100644 --- a/README.md +++ b/README.md @@ -22,17 +22,16 @@ result was a null or a mixed one, it's reported that way. --- -> **Status:** Phases 0-5a merged to `main` across five PRs +> **Status:** Phases 0-5b merged to `main` across six PRs > ([#1](https://github.com/bsreecharanreddy/dispatch/pull/1) Phase 0, > [#2](https://github.com/bsreecharanreddy/dispatch/pull/2) Phase 1, > [#3](https://github.com/bsreecharanreddy/dispatch/pull/3) Phases 2-3, > [#4](https://github.com/bsreecharanreddy/dispatch/pull/4) Phase 4, -> [#5](https://github.com/bsreecharanreddy/dispatch/pull/5) Phase 5a) — +> [#5](https://github.com/bsreecharanreddy/dispatch/pull/5) Phase 5a, +> [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) Phase 5b) — > Phase 2 landed as docs only, its actual contribution being the -> upstreamed vLLM PR below, not a dispatch-repo code change. **Phase 5b -> (speculative decoding) is complete**, PR -> [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) open. Full -> task-by-task record: +> upstreamed vLLM PR below, not a dispatch-repo code change. Phase 6 +> (final benchmark vs. vLLM/SGLang) is next. Full task-by-task record: > [`docs/STATUS.md`](docs/STATUS.md). Design and phasing: > [`docs/design/2026-09-14-dispatch-system-design.md`](docs/design/2026-09-14-dispatch-system-design.md). diff --git a/docs/STATUS.md b/docs/STATUS.md index 2486848..260b3e3 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -498,8 +498,9 @@ mode. Re-auditing Phase 5a was out of this task's scope. Phase 5b is done: root cause fixed, real numbers measured, both correctness gates and the full k-sweep checked and explained. Phase 4 is -merged (PR #4). Phase 5a and Phase 5b are both ready to push as PRs per -the one-branch-per-phase convention. Phase 3's PR (#3) already merged. +merged (PR #4), as are Phase 5a (PR #5) and Phase 5b (PR #6), each on its +own branch per the one-branch-per-phase convention. Phase 3's PR (#3) also +merged. Phase 2's PR is still open and awaiting maintainer review; no further work planned on it beyond responding to review feedback. Phase 6 (final benchmark vs. vLLM/SGLang) planning can proceed, and should treat the