diff --git a/CHANGELOG.md b/CHANGELOG.md index d2bfd56..86d27d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,49 @@ Every number here is measured. Where one is later found wrong, it gets corrected **in place with the original kept**, because a retraction that deletes its own evidence is not a retraction. -## Unreleased +Every phase below has a matching git tag (`phase-0` through `phase-7`); +`v1.0.0` marks the full 8-phase system design complete. Compare any two +points in the project's history directly: +[`phase-0...v1.0.0`](https://github.com/bsreecharanreddy/dispatch/compare/phase-0...v1.0.0). -No phase shipped yet (`v0.1.0` is still the scaffold version; a tag lands -once a phase's exit criteria are actually met). +## [v1.0.0] / [phase-7] - 2026-09-21 + +**Phase 7 (productionization) complete.** This closes the system +design's entire 8-phase plan, Phase 0 through 7, merged to `main` via +[#9](https://github.com/bsreecharanreddy/dispatch/pull/9) +(`docs/plans/2026-09-20-phase-7-productionization-plan.md`). A Rust +router (tonic gRPC client, axum HTTP, Prometheus metrics) in front of +the existing Python model server, per ADR-0003's TGI-shaped split, +containerized (Docker multi-stage builds), deployed to a local `kind` +Kubernetes cluster with Prometheus/Grafana observability, demonstrated +once against a real rented GPU over an SSH tunnel — `make check` green +throughout (Python and the Rust router's own `fmt`/`clippy`/`test` +gate). The one real GPU session found and fixed two real bugs neither +anticipated by the plan: `python scripts/run_model_server.py` run +directly can't resolve its own lazy `scripts.gpu.*` import (`sys.path[0]` +is the script's directory, not the repo root — fixed by invoking +`python -m scripts.run_model_server` instead), and a genuine defect in +`deepseek-ai/deepseek-moe-16b-base`'s own shipped tokenizer — a +byte-level BPE vocabulary (47,723 of 100,000 entries carry the +GPT-2-style `Ġ` marker) wired to a SentencePiece `Metaspace` +pre-tokenizer/decoder expecting a `▁` marker that appears in zero vocab +entries, silently dropping every space in generated text. Root-caused +and fixed for real (not a pod-local shim) with +`fix_tokenizer_byte_level()` in `scripts/gpu/phase7_kernel_responder.py`, +verified by round-trip token-id identity — invisible to Phases 0-6 +because they only ever compared logits/token ids, never decoded text for +a human to read. Real demo request through the full stack (dev machine +-> `kind` router pod -> gRPC -> SSH tunnel -> real GPU inference -> +back): 136ms TTFT, correctly-spaced generated text, real Grafana +metrics. Cost: **$6.8563** of a $5 cap, exceeded and disclosed +mid-session, continued with explicit go-ahead. Full account: +`docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md`; +runbook: `docs/runbooks/phase-7-productionization.md`. + +Project total, all eleven rented-GPU sessions across all nine measured +phases (0, 1, 2, 3, 4, 5a, 5b, 6, 7): **$51.35**. + +## [phase-6] - 2026-09-20 - **Phase 6 (final benchmark vs. vLLM and SGLang) complete**, 2026-09-20, merged to `main` via @@ -61,6 +100,9 @@ once a phase's exit criteria are actually met). than rate x duration. Full account: `docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md`; runbook: `docs/runbooks/phase-6-final-benchmark.md`. + +## [phase-5b] - 2026-09-18 + - **Phase 5b (speculative decoding) complete**, 2026-09-18, merged via PR #6 (`docs/plans/2026-09-17-phase-5b-speculative-decoding-plan.md`). A shared propose/verify/accept/rollback loop with two drafters (a 7B draft model, @@ -94,6 +136,9 @@ once a phase's exit criteria are actually met). and that was raised to $20 with disclosure. Also: `docs/findings/` split into one subfolder per phase. Full account: `docs/findings/phase-5b/2026-09-18-phase-5b-speculative-decoding-run.md`. + +## [phase-5a] - 2026-09-17 + - **Phase 5a (int8 weight-only quantization) complete**, 2026-09-17 (`docs/plans/2026-09-16-phase-5a-quantization-plan.md`). Self-computed, per-output-channel int8 quantization extending the Triton kernel @@ -114,6 +159,9 @@ once a phase's exit criteria are actually met). cap, including two RunPod Community Cloud pods that hit a real host-level GPU passthrough bug. Full account: `docs/findings/phase-5a/2026-09-17-phase-5a-quantization-run.md`. + +## [phase-4] - 2026-09-17 + - **Phase 4 (disaggregated prefill/decode) complete**, 2026-09-17 (`docs/plans/2026-09-16-phase-4-disaggregated-prefill-decode-plan.md`). Continuous-batching prefill/decode workers across two real 4-GPU H100 @@ -127,6 +175,9 @@ once a phase's exit criteria are actually met). reported as genuinely inconclusive. Cost: **$10.94** of a $40 cap. Full account: `docs/findings/phase-4/2026-09-17-phase-4-disaggregated-prefill-decode-run.md`. + +## [phase-3] - 2026-09-16 + - **Phase 3 (multi-GPU expert-parallel serving) complete**, 2026-09-16 (`docs/plans/2026-09-15-phase-3-multi-gpu-expert-parallel-serving-plan.md`). Real 2x H200 SXM expert-parallel serving over DeepSeek's own DeepEP @@ -140,6 +191,9 @@ once a phase's exit criteria are actually met). median 2 max ~20, sit below Phase 1's smallest tested point of 16). Cost: **$13.03** of a $25 cap. Full account: `docs/findings/phase-3/2026-09-16-phase-3-multi-gpu-ep-run.md`. + +## [phase-2] - 2026-09-15 + - **Phase 2 (vLLM benchmark contribution) complete**, 2026-09-15 (`docs/plans/2026-09-15-phase-2-vllm-benchmark-contribution-plan.md`). Neither vLLM's nor SGLang's official MoE benchmark modeled skewed @@ -151,6 +205,9 @@ once a phase's exit criteria are actually met). Cost: **$0.77**. This phase landed no code in `dispatch` itself, only docs — its actual contribution is the upstreamed PR. Full account: `docs/findings/phase-2/2026-09-15-phase-2-vllm-benchmark-run.md`. + +## [phase-1] - 2026-09-15 + - **Phase 1 (custom Triton grouped-GEMM kernel) complete**, 2026-09-15 (`docs/plans/2026-09-15-phase-1-grouped-gemm-plan.md`). Built and tested: a CPU-only MoE reference and grouped-GEMM contract, a naive and @@ -180,6 +237,9 @@ once a phase's exit criteria are actually met). 512-2048 — a workload-specific result recorded rather than buried. Total GPU cost across both sessions: **$0.65**. Full account: `docs/findings/phase-1/2026-09-15-phase-1-grouped-gemm-run.md`. + +## [phase-0] - 2026-09-14 + - **Phase 0 (baseline) complete**, 2026-09-14 (`docs/plans/2026-09-14-phase-0-baseline-plan.md`). Built and tested: a RunPod REST client and provisioning CLI, a token-by-token-timed diff --git a/CLAUDE.md b/CLAUDE.md index 6f5deef..2e8c48f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -209,9 +209,10 @@ duration. Full account: `docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md`; runbook: `docs/runbooks/phase-6-final-benchmark.md`. -**Phase 7 (productionization) is complete, 2026-09-21**, on branch -`phase-7-productionization`, PR not yet opened. **This closes the system -design's entire 8-phase plan, Phase 0 through 7.** A Rust router (tonic +**Phase 7 (productionization) is complete, 2026-09-21**, merged to `main` +via [PR #9](https://github.com/bsreecharanreddy/dispatch/pull/9). **This +closes the system design's entire 8-phase plan, Phase 0 through 7.** A +Rust router (tonic gRPC client, axum HTTP, Prometheus metrics) in front of the existing Python model server, per ADR-0003's TGI-shaped split, containerized (Docker multi-stage builds), deployed to a local `kind` Kubernetes diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..882f707 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,106 @@ +# Contributing + +This is a solo portfolio project, not one actively seeking outside +contributions — but it's built with real conventions, and this doc exists +so anyone reading the code (or considering a PR) can see how it's meant +to be worked in. + +## Getting set up + +```bash +uv sync --all-extras --dev # Python +cd router && cargo build # Rust router (needs a stable toolchain) +``` + +## Running the checks + +```bash +make check # the full gate: lint, mypy --strict, pytest, and the + # Rust router's own fmt/clippy/test -- same steps CI runs +make check-fast # inner loop: skips slow-marked tests +make coverage # pytest --cov, term + xml report +``` + +`make check` runs before every push in this repo's own workflow, and +matches CI exactly (`.github/workflows/ci.yml`). + +## GPU-marked tests + +Any test that needs a real CUDA device is marked `gpu` and excluded from +both `make check` and CI — this repo's CI has no GPU runner, and marking +is the explicit, visible boundary of what CI can verify (never a silent +skip). GPU-marked tests are run for real, on rented hardware, once per +phase; each phase's findings doc under +[`docs/findings/`](docs/findings/) records exactly which GPU, for how +long, and at what cost. + +## Testing policy + +No implementation code is committed without tests. The testing table this +project settled on (from Phase 1's actual kernel work): + +| Layer | What must be covered | +|---|---| +| Kernel contract (CPU) | eager backend == `ReferenceMoE`; every row covered by exactly one tile; no tile crosses an expert boundary; experts that receive zero tokens | +| Kernels (`gpu`) | each kernel meets `assert_matches_reference` against an fp32 reference at toy, decode- and prefill-shaped dims, fp16 and bf16; a mutation must turn the suite red | +| End to end (`gpu`, paid) | mutual top-5 logit agreement with a same-session stock run; a kernel run that patches no layers refuses to run | +| Benchmarks | `triton.testing.do_bench`; a backend that disagrees with the eager one is refused, not timed; every JSON carries its full config | + +**Correctness before speed** governs all of it: a kernel isn't "fast" +until it's been proven numerically correct against a reference +implementation within a stated tolerance. A kernel that's fast because +it's silently wrong is a bug, not a result. + +**Never quote a benchmark number that wasn't measured** on the exact +hardware, at the exact config (batch size, sequence length, quantization +level, warm-up handled), by this repo. A benchmark claim states its +config alongside the number — "40% faster" isn't a result; "40% faster +at batch size 32, seq len 2048, on a single A100 80GB, vLLM 0.x.y as +baseline" is. + +## Conventions + +- **One branch per phase**, carrying the whole phase, pushed once as a + single PR. +- **One commit per completed task**, not one bundled commit per phase. +- **`docs/STATUS.md` updates in the same commit as the work it + describes.** +- Conventional commit prefixes (`feat:`, `fix:`, `docs:`, `test:`, + `refactor:`, `build:`). +- **Commit messages are plain ASCII** — `--`, never an em-dash. Docs use + `—` freely; the git log does not. +- Every phase gets a `docs/design/` entry (for a new subsystem) or slots + into the existing [system design](docs/design/2026-09-14-dispatch-system-design.md), + then a `docs/plans/` implementation plan, written before any code. + +## Language and tooling + +Python 3.12+ (`uv`, `ruff`, `mypy --strict`, `pytest`), Rust (stable, +`cargo fmt`, `cargo clippy -- -D warnings`, `cargo test`) for the Phase 7 +router. GPU provisioning (`scripts/gpu/`) calls RunPod's REST API +directly rather than through Terraform — the actual workflow (rent, run +a benchmark, tear down) fits a script better than Terraform state +management, and RunPod's own Terraform provider is much less mature than +AWS/Azure's. + +## Cost discipline + +GPU rental is real money. If you're running the paid (`gpu`-marked) test +suite or a benchmark script yourself: + +- Use marketplace/spot instances (RunPod Community Cloud, Vast.ai), not + dedicated on-demand. +- Set a budget cap before the first rental, not after. +- Spin up, run the measured session, capture the evidence, tear down + immediately — never leave a rented GPU idle between sessions or across + an unbounded wait. +- Pull every evidence file off the pod before `stop`, not after; a + stopped pod's disk is not guaranteed to come back. + +## Full picture + +[`CLAUDE.md`](CLAUDE.md) carries the complete set of engineering +patterns, phase-by-phase history, and the project's own written record of +what it got wrong and fixed along the way — written for an AI coding +assistant working in this repo, but equally readable as the fuller +version of this document. diff --git a/README.md b/README.md index 2babb74..5388024 100644 --- a/README.md +++ b/README.md @@ -22,25 +22,39 @@ result was a null or a mixed one, it's reported that way. --- -> **Status:** Phases 0-6 merged to `main` across seven PRs +> **Status:** All 8 phases (0-7) merged to `main` across eight PRs > ([#1](https://github.com/bsreecharanreddy/dispatch/pull/1) Phase 0, > [#2](https://github.com/bsreecharanreddy/dispatch/pull/2) Phase 1, > [#3](https://github.com/bsreecharanreddy/dispatch/pull/3) Phases 2-3, > [#4](https://github.com/bsreecharanreddy/dispatch/pull/4) Phase 4, > [#5](https://github.com/bsreecharanreddy/dispatch/pull/5) Phase 5a, > [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) Phase 5b, -> [#8](https://github.com/bsreecharanreddy/dispatch/pull/8) Phase 6) — +> [#8](https://github.com/bsreecharanreddy/dispatch/pull/8) Phase 6, +> [#9](https://github.com/bsreecharanreddy/dispatch/pull/9) Phase 7) — > Phase 2 landed as docs only, its actual contribution being the -> upstreamed vLLM PR below, not a dispatch-repo code change. **Phase 7 -> (productionization) is complete**, PR pending: a real Rust router in -> front of the Python model server, containerized, deployed to a local -> Kubernetes cluster, demonstrated once against a real rented GPU over an -> SSH tunnel — see [below](#phase-7-seeing-it-run). **This closes the -> system design's entire phase plan, Phase 0 through 7.** Full -> task-by-task record: [`docs/STATUS.md`](docs/STATUS.md). Design and -> phasing: +> upstreamed vLLM PR below, not a dispatch-repo code change. **This +> closes the system design's entire 8-phase plan.** Phase 7 +> (productionization) added a real Rust router in front of the Python +> model server, containerized, deployed to a local Kubernetes cluster, +> demonstrated once against a real rented GPU over an SSH tunnel — see +> [below](#phase-7-seeing-it-run). Every phase has its own git tag +> (`phase-0` through `phase-7`); `v1.0.0` marks this completed state. +> Full task-by-task record: [`docs/STATUS.md`](docs/STATUS.md); phase +> history: [`CHANGELOG.md`](CHANGELOG.md). Design and phasing: > [`docs/design/2026-09-14-dispatch-system-design.md`](docs/design/2026-09-14-dispatch-system-design.md). +**Jump to a section by interest:** + +| If you care about... | Start here | +|---|---| +| Inference / GPU-kernel engineering | [Phase 1 kernel](src/dispatch/kernels/grouped_gemm.py) + [findings](docs/findings/phase-1/2026-09-15-phase-1-grouped-gemm-run.md) | +| Multi-GPU / distributed serving | [Phase 3](docs/findings/phase-3/2026-09-16-phase-3-multi-gpu-ep-run.md) + [Phase 4](docs/findings/phase-4/2026-09-17-phase-4-disaggregated-prefill-decode-run.md) findings | +| Production serving / platform engineering | [Phase 7: seeing it run](#phase-7-seeing-it-run) | +| The vLLM/SGLang head-to-head | [Phase 6 findings](docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md) | +| Whether the claims actually hold up | [`docs/STATUS.md`](docs/STATUS.md) + [`docs/findings/`](docs/findings/) | + +(Full breakdown, one line per audience: [Who should look at what](#who-should-look-at-what).) + ## In sixty seconds **What it does.** A pure-PyTorch reference MoE forward pass, a from-scratch @@ -50,9 +64,10 @@ DeepEP library, and a disaggregated prefill/decode scheduler — each phase's thesis stated up front and reported honestly, including the phases where the answer was a null or mixed result. -**Built solo, start to finish** — design docs, the kernel, the multi-GPU -serving path, an upstreamed open-source benchmark contribution, every -rented-GPU session, and the write-ups of what broke along the way. +**Built entirely solo** — every Triton kernel, the multi-GPU serving path, +the Rust router, the Kubernetes manifests, an upstreamed open-source +benchmark contribution, every rented-GPU session, and every write-up of +what broke along the way, start to finish, no team. **Measured across eleven rented-GPU sessions (nine phases), $51.35 total, every phase within its stated cap except Phase 5b (its original $10 cap @@ -88,9 +103,13 @@ defect — the model's shipped vocabulary is byte-level BPE but wired to the wrong pre-tokenizer/decoder pair — found, root-caused, and fixed (Phase 7). -**262 tests, `make check` green throughout Phase 6** (lint, `mypy --strict`, -and the full non-GPU suite) — GPU-dependent tests are marked and excluded -from CI by design, then run for real on rented hardware every phase. +**277 tests** (268 Python + 9 Rust), `make check` green throughout — +lint, `mypy --strict`, the Rust router's own `fmt`/`clippy`/`test` gate, +and the full non-GPU suite. **86% coverage on CPU-testable logic** +(`pytest --cov`; the kernels themselves are GPU-only and covered by the +`gpu`-marked correctness suite instead, not by this number). +GPU-dependent tests are marked and excluded from CI by design, then run +for real on rented hardware every phase. ## Why this project @@ -289,6 +308,10 @@ They're run for real on rented hardware every phase; each phase's findings doc in [`docs/findings/`](docs/findings/) records exactly which GPU, for how long, and at what cost. +The full testing policy, conventions, and cost discipline this project +follows: [`CONTRIBUTING.md`](CONTRIBUTING.md). Phase-by-phase history, +one entry per release: [`CHANGELOG.md`](CHANGELOG.md). + ## Design and the paper trail [`docs/design/2026-09-14-dispatch-system-design.md`](docs/design/2026-09-14-dispatch-system-design.md) @@ -315,6 +338,12 @@ is the authoritative architecture, phasing, and scope document. [Phase 4's](docs/findings/phase-4/2026-09-17-phase-4-disaggregated-prefill-decode-run.md) findings docs for two results — one null, one mixed — found by measuring on real hardware rather than assumed. +- **Production serving / platform engineering** — `router/` (the Rust + gRPC/HTTP router) and `k8s/`, then + [Phase 7's findings doc](docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md): + a real router in front of a real GPU-backed model server, containerized + and deployed to Kubernetes, including a genuine tokenizer bug found and + fixed on the one real demo run. - **ML infra / open-source contribution** — [Phase 2's findings doc](docs/findings/phase-2/2026-09-15-phase-2-vllm-benchmark-run.md) and the open [vLLM PR](https://github.com/vllm-project/vllm/pull/57100): a diff --git a/docs/STATUS.md b/docs/STATUS.md index 00b640d..caf216d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -870,11 +870,10 @@ real-hardware surprise every prior phase's paid session also turned up. ## Next step -Phases 3 (PR #3), 4 (PR #4), 5a (PR #5), 5b (PR #6), and 6 (PR #8) are all -merged, each on its own branch per the one-branch-per-phase convention. -Phase 7 is complete on branch `phase-7-productionization` (see "Phase 7 -progress"), PR not yet opened -- the next action is asking the user for -push/PR approval, per this repo's standing convention, then opening it. +All nine measured phases (0, 1, 2, 3, 4, 5a, 5b, 6, 7) are merged to +`main` across PRs #1-#6, #8, and #9, each on its own branch per the +one-branch-per-phase convention. Phase 7 merged 2026-09-21 via +[#9](https://github.com/bsreecharanreddy/dispatch/pull/9). **The system design's phase table (§7) is now fully executed, Phase 0 through 7.** There is no next phase; `dispatch`'s implementation work @@ -885,3 +884,16 @@ brainstormed and written down before any code. Phase 2's PR is still open and awaiting maintainer review; no further work planned on it beyond responding to review feedback. + +**Post-completion portfolio polish, 2026-09-21:** repo made public; +annotated git tags added for `phase-0` through `phase-7` (at each PR's +merge commit) and `v1.0.0` (the completed 8-phase state, same commit as +`phase-7`); GitHub repo topics set; `CHANGELOG.md` restructured from a +single running "Unreleased" section into one dated version heading per +tag, with Phase 7's entry added (it had been missing); `CONTRIBUTING.md` +added (testing policy, conventions, cost discipline, lifted from this +file for a reader who wants the short version); README given a top-of-file +scan table, an explicit "built entirely solo" line, current test/coverage +numbers (277 tests: 268 Python + 9 Rust; 86% coverage on CPU-testable +logic), and a Phase 7 entry in "Who should look at what" (previously +missing). No code changed; `make check` green throughout.