diff --git a/.gitignore b/.gitignore index dc5b5fe..7d16d7a 100644 --- a/.gitignore +++ b/.gitignore @@ -54,3 +54,10 @@ graphify-out/ # Subagent-driven-development scratch workspace (ledgers, briefs, review # packages) -- process artifacts, not project history. .superpowers/ + +# Rust +/router/target/ + +# Playwright MCP browser-automation scratch output (page snapshots, +# console logs) -- tool-generated, not project content. +.playwright-mcp/ diff --git a/CLAUDE.md b/CLAUDE.md index cda95c5..6f5deef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,7 +33,7 @@ piece neither of them touches. **`docs/design/2026-09-14-dispatch-system-design.md`** is the authoritative architecture doc — model choice (deepseek-ai/deepseek-moe-16b-base), what's custom vs. reused, kernel scope, the multi-GPU/DeepEP plan and its real -NVLink hardware constraint, benchmark methodology, the 7-phase plan, cost +NVLink hardware constraint, benchmark methodology, the 8-phase plan, cost plan, and explicit scope boundaries. Read it before making any structural decision. `docs/adr/` carries the specific "why this over that" calls made along the way (0001: grouped-GEMM over another attention-kernel @@ -209,6 +209,43 @@ duration. Full account: `docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md`; runbook: `docs/runbooks/phase-6-final-benchmark.md`. +**Phase 7 (productionization) is complete, 2026-09-21**, on branch +`phase-7-productionization`, PR not yet opened. **This closes the system +design's entire 8-phase plan, Phase 0 through 7.** A Rust router (tonic +gRPC client, axum HTTP, Prometheus metrics) in front of the existing +Python model server, per ADR-0003's TGI-shaped split, containerized +(Docker multi-stage builds), deployed to a local `kind` Kubernetes +cluster with Prometheus/Grafana observability, demonstrated once against +a real rented GPU over an SSH tunnel. The one real GPU session found and +fixed two real bugs neither anticipated by the plan: `python +scripts/run_model_server.py` run directly can't resolve its own lazy +`scripts.gpu.*` import (`sys.path[0]` is the script's directory, not the +repo root -- fixed by invoking `python -m scripts.run_model_server` +instead), and a genuine defect in `deepseek-ai/deepseek-moe-16b-base`'s +own shipped tokenizer -- a byte-level BPE vocabulary (47,723 of 100,000 +entries carry the GPT-2-style `Ġ` marker) wired to a SentencePiece +`Metaspace` pre-tokenizer/decoder expecting a `▁` marker that appears in +zero vocab entries, silently dropping every space in generated text. +Root-caused and fixed for real (not a pod-local shim) with +`fix_tokenizer_byte_level()` in `scripts/gpu/phase7_kernel_responder.py`, +verified by round-trip token-id identity -- invisible to Phases 0-6 +because they only ever compared logits/token ids, never decoded text for +a human to read. The gpu-marked correctness test needed a second run with +bytecode caching disabled after a checksum-identical file reproduced its +pre-fix failure, traced to stale `.pyc` bytecode on the pod's +network-mounted `/workspace`. Real demo request through the full stack +(dev machine -> `kind` router pod -> gRPC -> SSH tunnel -> real GPU +inference -> back): 136ms TTFT, correctly-spaced generated text, real +Grafana metrics. Cost: **$6.8563** of a **$5 cap**, exceeded and disclosed +mid-session (mostly real debugging time: two long correctness-test runs +plus live tokenizer diagnosis), continued with the user's explicit +go-ahead rather than stopped early. PII (the presenter's local +username/hostname) found in the screen recording and RunPod console +screenshots was redacted, frame-by-frame verified, before publishing. +Full account: +`docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md`; +runbook: `docs/runbooks/phase-7-productionization.md`. + ## One governing principle **Correctness before speed.** A kernel is not "fast" until it has been diff --git a/Makefile b/Makefile index 2729555..b03e697 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: test test-fast lint fmt typecheck check check-fast coverage +.PHONY: test test-fast lint fmt typecheck check check-fast coverage router-lint router-test # GPU-dependent tests (anything that needs an actual CUDA device) are marked # `gpu` and excluded here -- this repo's CI has no GPU runner. They run @@ -24,22 +24,36 @@ fmt: typecheck: uv run mypy src tests scripts -# The full gate. Runs before every push -- CI runs the same three steps. +router-lint: + cd router && cargo fmt --check && cargo clippy --all-targets -- -D warnings + +router-test: + cd router && cargo test + +# The full gate. Runs before every push -- CI runs the same steps. check: - @echo "[1/3] lint" + @echo "[1/5] lint" @$(MAKE) lint - @echo "[2/3] typecheck" + @echo "[2/5] typecheck" @$(MAKE) typecheck - @echo "[3/3] test" + @echo "[3/5] test" @$(MAKE) test + @echo "[4/5] router-lint" + @$(MAKE) router-lint + @echo "[5/5] router-test" + @$(MAKE) router-test check-fast: - @echo "[1/3] lint" + @echo "[1/5] lint" @$(MAKE) lint - @echo "[2/3] typecheck" + @echo "[2/5] typecheck" @$(MAKE) typecheck - @echo "[3/3] test-fast" + @echo "[3/5] test-fast" @$(MAKE) test-fast + @echo "[4/5] router-lint" + @$(MAKE) router-lint + @echo "[5/5] router-test" + @$(MAKE) router-test coverage: uv run pytest -m "not gpu" --cov --cov-report=term --cov-report=xml diff --git a/README.md b/README.md index 6cf9b9d..2babb74 100644 --- a/README.md +++ b/README.md @@ -31,10 +31,12 @@ result was a null or a mixed one, it's reported that way. > [#6](https://github.com/bsreecharanreddy/dispatch/pull/6) Phase 5b, > [#8](https://github.com/bsreecharanreddy/dispatch/pull/8) Phase 6) — > Phase 2 landed as docs only, its actual contribution being the -> upstreamed vLLM PR below, not a dispatch-repo code change. **Phase 6 -> (final benchmark vs. vLLM/SGLang) is complete and merged.** This closes -> the system design's full phase plan (0-6); only Phase 7 -> (productionization) remains. Full +> upstreamed vLLM PR below, not a dispatch-repo code change. **Phase 7 +> (productionization) is complete**, PR pending: a real Rust router in +> front of the Python model server, containerized, deployed to a local +> Kubernetes cluster, demonstrated once against a real rented GPU over an +> SSH tunnel — see [below](#phase-7-seeing-it-run). **This closes the +> system design's entire phase plan, Phase 0 through 7.** Full > task-by-task record: [`docs/STATUS.md`](docs/STATUS.md). Design and > phasing: > [`docs/design/2026-09-14-dispatch-system-design.md`](docs/design/2026-09-14-dispatch-system-design.md). @@ -52,9 +54,10 @@ the answer was a null or mixed result. serving path, an upstreamed open-source benchmark contribution, every rented-GPU session, and the write-ups of what broke along the way. -**Measured across ten rented-GPU sessions (eight phases), $44.49 total, -every phase within its stated cap except Phase 5b, whose original $10 cap -was exceeded on one pod and raised to $20 with disclosure:** a custom Triton kernel **~65-73% faster** than DeepSeek's own +**Measured across eleven rented-GPU sessions (nine phases), $51.35 total, +every phase within its stated cap except Phase 5b (its original $10 cap +was exceeded on one pod and raised to $20 with disclosure) and Phase 7 +(its $5 cap was exceeded to $6.8563, disclosed mid-session):** a custom Triton kernel **~65-73% faster** than DeepSeek's own stock MoE forward pass at perfect logit agreement (Phase 1); an opt-in skewed-load benchmark flag upstreamed to vLLM, changing which kernel config its own tuner picks at 4 of 5 tested batch sizes (Phase 2); a real 2-GPU @@ -77,7 +80,13 @@ its shipped default config on nearly every tested shape**, in both bf16 and int8, on the same GPU, same model, same Triton compiler as every contestant, with the at-scale correctness gate this project owed since Phase 5a finally measured (97.1-97.3% top-1 agreement at 1,036 positions, -zero large-gap disagreements) (Phase 6). +zero large-gap disagreements) (Phase 6); and a real Rust router serving +real GPU-backed generations through a local Kubernetes cluster over an +SSH tunnel, where diagnosing why the very first demo request came back +with no spaces between words led to a genuine third-party tokenizer +defect — the model's shipped vocabulary is byte-level BPE but wired to +the wrong pre-tokenizer/decoder pair — found, root-caused, and fixed +(Phase 7). **262 tests, `make check` green throughout Phase 6** (lint, `mypy --strict`, and the full non-GPU suite) — GPU-dependent tests are marked and excluded @@ -160,8 +169,8 @@ flowchart LR The differentiated work is the grouped-GEMM kernel and the serving system around it. Phases 0-6 drive the kernel and model directly from lightweight Python test/benchmark harnesses; the Rust-router/Python-model-server split -is Phase 7's contribution, once there's a real service worth -productionizing. +is Phase 7's contribution, built and demonstrated once the kernel and +serving work behind it were proven. ## What has been measured @@ -178,11 +187,31 @@ including the ones that came back null or mixed: | **5a — int8 quantization** | Does a self-computed, per-channel int8 weight-only kernel cut memory with no model-level correctness cost? | **Yes on both counts, no speedup.** **49.89%** expert-weight memory reduction, perfect top-1/mutual-top-k agreement vs. the naive bf16 kernel at every tested position (29 positions, one run -- a small sample; see 5b for the near-tie sensitivity it could not rule out). Throughput was a near-exact tie with naive (**-1.2%**), as expected — weight-only quantization saves memory bandwidth, not FLOPs. A real bug (quantizing without freeing the original bf16 weights, OOMing a 44GB L40) was found and fixed mid-session | | **5b — speculative decoding** | Does speculative decoding (a 7B draft model, or model-free prompt-lookup) speed up single-request decode on the int8 target while reproducing plain greedy output exactly? | **Partly, and not exactly.** A40, bf16, unbatched, baseline **25.18 tok/s**: only prompt-lookup beats it (**34.4-50.9 tok/s** across k=1-8, +77.7% at k=4); the 7B draft model is *below* baseline at every k (19.6-23.0) despite 2-7x higher acceptance. Output matched plain greedy in 13 of 16 (prompt, k) combinations for draft-model and 9 of 16 for prompt-lookup; the same k=4 config matched 2/4 prompts in one process and 3/4 in another. Root-caused to near-tied logits under the int8 kernel's floating-point precision, not a logic bug. The first session's numbers were withdrawn by this project's own review (degenerate baseline) and root-caused to a `transformers` bug leaving DeepSeek's RoPE buffers uninitialized | | **6 — final benchmark** | On the same L40, same model, same Triton compiler, how does dispatch's kernel compare with vLLM's and SGLang's own fused-MoE, at default and at scale, and does the correctness claim hold past a 29-position sample? | **vLLM's default config beats dispatch at every one of the 28 measured shapes**, bf16 and int8 alike -- the gap narrows from 2.4-4.1x slower at 1 token to ~1.1-1.3x slower at 128 tokens and up, but never closes (dispatch does beat SGLang specifically at 128/512 tokens, just not vLLM). The at-scale gate passed for every kernel: **97.1-97.3% top-1 agreement at 1,036 positions, zero large-gap disagreements** against a pre-registered 1.0-logit threshold — replacing Phase 5a's 29-position claim. vLLM and SGLang both served the real model correctly across concurrency 1-64 (vLLM ~6-9% ahead of SGLang throughout); dispatch's own harness has no served, batched engine to place in that same table. vLLM's and SGLang's own tuners were not run to completion (~18 min/token-count, all-or-nothing per run — confirmed live, not assumed) — both shown at default config only, disclosed rather than glossed over | +| **7 — productionization** | Can the kernel and serving work from Phases 0-6 stand behind a real router, containerized and deployed to Kubernetes, serving one real request against a rented GPU end to end? | **Yes**, with two real bugs found and fixed along the way: a `sys.path` footgun in the model-server entrypoint, and a genuine defect in the model's own shipped tokenizer (a byte-level BPE vocabulary wired to a SentencePiece-style pre-tokenizer/decoder expecting the wrong marker character) that silently dropped every space in generated text until root-caused and fixed. Real demo request through router → gRPC → SSH tunnel → GPU: **136ms TTFT**, correctly-spaced generated text, real Prometheus/Grafana metrics. Cost **$6.8563** against a $5 cap, exceeded and disclosed | Full method and every number: one folder per phase in -[`docs/findings/`](docs/findings/) (`phase-0/` through `phase-6/`), each +[`docs/findings/`](docs/findings/) (`phase-0/` through `phase-7/`), each with a run doc and a cost doc. +## Phase 7: seeing it run + +A recorded demo instead of a standing public endpoint — a real GPU-backed +service running continuously costs roughly $500-600/month, which this +project's own cost-discipline rules rule out for a demo. Everything below +is from the one real session: `curl` against the router, gRPC to a +GPU-backed model server over an SSH tunnel, real generated text, real +Grafana metrics. + +![Grafana dashboard showing real request counts, queue depth, and time-to-first-token from the Phase 7 demo](docs/findings/phase-7/2026-09-21-phase-7-grafana-dashboard.png) + +![Terminal showing the real curl request through the router and its real, correctly-spaced generated response](docs/findings/phase-7/2026-09-21-phase-7-terminal-demo.png) + +[Screen recording](docs/findings/phase-7/2026-09-21-phase-7-demo-recording.mp4) +of the live request and the dashboard updating (presenter's local +username/hostname redacted before publishing). Full account, including +the two bugs found and fixed live: +[`docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md`](docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md). + ## What it does not do Stated here rather than left for a reader to discover — from the design @@ -211,7 +240,9 @@ doc's own scope boundaries: | Multi-GPU | DeepSeek's DeepEP — real all-to-all expert-parallel dispatch/combine over NVLink/SXM | | Serving | Continuous-batching prefill/decode workers, both co-located and disaggregated topologies; speculative decoding (draft-model and prompt-lookup drafters, Phase 5b) | | Benchmark methodology | Measured head-to-head against real vLLM 0.29.0 and SGLang 0.5.20, same GPU, same model, same Triton compiler (Phase 6) — TTFT, inter-token latency, tokens/sec, cost per million tokens | -| Language | Python 3.12+ — `uv`, `ruff`, `mypy --strict`, `pytest` | +| Serving front door | Rust router (`tonic` gRPC client, `axum` HTTP, Prometheus metrics) in front of the Python model server, per ADR-0003 (Phase 7) | +| Deployment | Docker (multi-stage builds), Kubernetes (`kind` locally), Prometheus + Grafana observability (Phase 7) | +| Language | Python 3.12+ — `uv`, `ruff`, `mypy --strict`, `pytest`; Rust 2024 edition — `cargo fmt`, `cargo clippy`, `cargo test` | | GPU provisioning | RunPod's API via lightweight scripts (`scripts/gpu/`) — not Terraform; see `CLAUDE.md`'s cost discipline | | CI | GitHub Actions — lint, `mypy --strict`, and the non-GPU suite on every push | @@ -222,19 +253,25 @@ src/dispatch/ benchmark/ metrics, generation harness, reference-logit capture/comparison kernels/ reference MoE, Triton grouped-GEMM (naive + persistent), int8 quantization, expert-parallel sharding, backend registry - serving/ KV-cache slice/handoff, continuous-batching prefill/decode, co-located worker + serving/ KV-cache slice/handoff, continuous-batching prefill/decode, co-located worker, + model_server.py (gRPC service wrapping the Responder protocol) speculative/ Drafter protocol (draft-model, prompt-lookup), shared verify/rollback loop, independent plain-greedy oracle, exact generated-token comparison +router/ Rust: tonic gRPC client, axum HTTP surface, Prometheus metrics (Phase 7) +docker/ router.Dockerfile, model-server.Dockerfile (Phase 7) +k8s/ namespace/deployment/service manifests, Prometheus + Grafana (Phase 7) scripts/ run_baseline.py latency/throughput CLI, stock or kernel-patched, --compare-reference gate run_speculative_bench.py speculative-decoding CLI, --compare-generated-tokens checks exact output run_kernel_bench.py pure kernel micro-benchmark, refuses to time a backend that disagrees + run_model_server.py Phase 7's gRPC model-server CLI (stub or real-kernel responder) + run_kind_demo.sh brings up the local kind cluster + router + observability stack (Phase 7) gpu/ RunPod API client, provisioning CLI, pod-live correctness/concurrency scripts tests/ unit + integration; `gpu`-marked tests need real CUDA, excluded from CI docs/design/ the authoritative architecture, phase plan, and scope document docs/adr/ decisions, each checked against real current material before being made docs/plans/ per-phase implementation plans, written before any code -docs/findings/ measured results and costs — one subfolder per phase (phase-0 ... phase-5b) +docs/findings/ measured results and costs — one subfolder per phase (phase-0 ... phase-7) docs/runbooks/ the exact GPU-session steps each phase's real run followed ``` diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..b139fbc --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,25 @@ +services: + model-server: + build: + context: . + dockerfile: docker/model-server.Dockerfile + command: ["--responder", "stub", "--port", "50051"] + ports: + - "50051:50051" + + router: + build: + context: . + dockerfile: docker/router.Dockerfile + environment: + MODEL_SERVER_ENDPOINT: "http://model-server:50051" + ROUTER_LISTEN_ADDR: "0.0.0.0:8080" + ROUTER_QUEUE_CAPACITY: "1" + ports: + - "8080:8080" + depends_on: + - model-server + # depends_on only waits for the container to *start*, not for the + # model server to finish syncing and bind its port -- the router's own + # startup retry (main.rs) is the primary fix; this is defense in depth. + restart: on-failure diff --git a/docker/model-server.Dockerfile b/docker/model-server.Dockerfile new file mode 100644 index 0000000..4470c20 --- /dev/null +++ b/docker/model-server.Dockerfile @@ -0,0 +1,19 @@ +FROM nvidia/cuda:13.3.0-runtime-ubuntu24.04 +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3.12 python3.12-venv curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* +RUN curl -LsSf https://astral.sh/uv/install.sh | sh +ENV PATH="/root/.local/bin:${PATH}" +WORKDIR /app +COPY pyproject.toml uv.lock README.md ./ +COPY proto ./proto +COPY src ./src +COPY scripts ./scripts +RUN uv sync --frozen --no-dev +EXPOSE 50051 +# --no-dev here too: `uv run` re-syncs by default, and without it, every +# container start re-installs mypy/ruff/pytest into a production image +# (found live: a compose run spent its first several seconds downloading +# and installing dev tools before the server ever bound its port). +ENTRYPOINT ["uv", "run", "--no-dev", "python", "scripts/run_model_server.py"] +CMD ["--responder", "stub", "--port", "50051"] diff --git a/docker/router.Dockerfile b/docker/router.Dockerfile new file mode 100644 index 0000000..5b7b202 --- /dev/null +++ b/docker/router.Dockerfile @@ -0,0 +1,17 @@ +FROM rust:1.98.1-slim-trixie AS build +WORKDIR /build +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* +COPY proto ./proto +COPY router/Cargo.toml router/Cargo.lock ./router/ +COPY router/build.rs ./router/build.rs +COPY router/src ./router/src +WORKDIR /build/router +RUN cargo build --release + +FROM debian:trixie-slim +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* +COPY --from=build /build/router/target/release/dispatch-router /usr/local/bin/dispatch-router +EXPOSE 8080 +ENTRYPOINT ["/usr/local/bin/dispatch-router"] diff --git a/docs/STATUS.md b/docs/STATUS.md index ed4b9ed..00b640d 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -649,18 +649,239 @@ disclosed limitation, not a fudge. Total cost: **\$4.5448 of the \$10 cap** `docs/findings/phase-6/2026-09-20-phase-6-final-benchmark-run.md`; runbook: `docs/runbooks/phase-6-final-benchmark.md`. +## Phase 7 progress + +Design: `docs/design/2026-09-20-phase-7-productionization.md`. Plan: +`docs/plans/2026-09-20-phase-7-productionization-plan.md`, 14 tasks. Branch +`phase-7-productionization` (worktree). + +- [x] Task 1, 2026-09-20: `kind` installed (v0.33.0). Connectivity spike on + this machine (macOS, Docker Desktop): a pod inside a throwaway `kind` + cluster reached a plain `python3 -m http.server` listener on the dev + machine via `host.docker.internal:` on the first try -- no + fallback host-gateway config needed. This is the hostname + `k8s/model-server-external.yaml` (Task 11) hardcodes for the + `ExternalName` Service the router uses to reach the (later, + SSH-tunneled) model server. +- [x] Task 2, 2026-09-20: shared `proto/dispatch.proto` and the Rust router + crate scaffold (`router/`), Rust 1.98.1 installed via rustup. One real + plan correction found live: `tonic_build::compile_protos` (the plan's + originally-written `build.rs` call) does not exist in tonic 0.14 -- + that release split prost integration out of `tonic-build` into two + new crates, `tonic-prost-build` (build-time codegen) and `tonic-prost` + (the runtime `ProstCodec` the generated code references). Fixed by + switching to `tonic_prost_build::compile_protos` and adding + `tonic-prost` as a normal dependency; confirmed by reading + tonic-build 0.14.6's actual source after the original call failed to + compile, not guessed. `tonic::async_trait` (planned for later tasks' + mock gRPC servers) is unaffected -- still re-exported at tonic's + crate root, confirmed the same way. +- [x] Task 3, 2026-09-20: Python side of the contract -- + `dispatch.serving.model_server` (`Responder` protocol, `StubResponder`, + `ModelServerServicer`, `serve`) and `scripts/run_model_server.py`. + `make check` green (267 tests, up from 262; lint and `mypy --strict` + clean), but getting there needed three real mypy fixes beyond the + plan's own code: (1) `--python_out` alone generates message classes + via runtime reflection, invisible to mypy without the matching + `--pyi_out` stub, now part of `scripts/gen_proto.py`; (2) the plan's + `ignore_errors = true` override for `dispatch.proto.*` only suppresses + errors *inside* that module, not "call to untyped function" errors at + its *importers* -- switched to `ignore_missing_imports` + + `follow_imports = "skip"`, the same pattern already used for + triton/deep_ep/vllm/sglang; (3) that per-module override only applies + when a module is *imported*, not when mypy walks `src` and hits it + directly as a root target, so the two generated files also needed an + explicit `[tool.mypy] exclude` pattern. All three confirmed by reading + mypy's actual error output and its own documented `follow_imports` + semantics, not guessed. Two intentional forward references to Task 9's + not-yet-existing `scripts/gpu/phase7_kernel_responder.py` are marked + with `# type: ignore` comments Task 9 is expected to remove (mypy's + `warn_unused_ignores` will flag them once that module exists and is + typed). +- [x] Task 4, 2026-09-20: router admission queue (`router/src/queue.rs`) -- + a capacity-1 `tokio::sync::Semaphore` plus an `AtomicUsize` waiting + counter for the `queue_depth` gauge Task 5 wires up. `cargo test`, + `cargo fmt`, `cargo clippy -D warnings` all clean. +- [x] Task 5, 2026-09-20: router Prometheus metrics (`router/src/metrics.rs`) + -- `RouterMetrics` registers `dispatch_router_requests_total`, + `..._ttft_seconds`, `..._inter_token_latency_seconds`, + `..._queue_depth` on its own `Registry`, with `.encode()` for the + `/metrics` HTTP handler Task 7 adds. `cargo test`/`fmt`/`clippy` clean. +- [x] Task 6, 2026-09-20: router gRPC client (`router/src/client.rs`) and a + test-only mock `ModelServer` (`router/src/test_support.rs`, gated + `#[cfg(test)]`). `ModelServerClient::generate` records TTFT and + inter-token-latency into `RouterMetrics` as it streams. The one thing + Task 2 flagged as uncertain -- whether `#[tonic::async_trait]` still + matches tonic 0.14's generated `ModelServer` trait after the + tonic-prost-build split -- compiled and passed on the first try, no + further correction needed. `cargo test`/`fmt`/`clippy` clean. +- [x] Task 7, 2026-09-20: router HTTP surface (`router/src/app.rs`: + `/generate`, `/healthz`, `/readyz`, `/metrics`) and `main.rs` wiring + (env-configured: `MODEL_SERVER_ENDPOINT`, `ROUTER_LISTEN_ADDR`, + `ROUTER_QUEUE_CAPACITY`). All 3 tests passed on the first try. + Manually smoke-tested end to end (stub model server + real router + binary, both running): `POST /generate` returned real streamed text + and timing, `/metrics` showed real recorded TTFT/inter-token-latency/ + queue-depth/requests-total values. `cargo test`/`fmt`/`clippy` clean. +- [x] Task 8, 2026-09-20: cross-language router integration test + (`router/tests/cross_language_integration.rs`) -- spawns the real + `scripts/run_model_server.py --responder stub` as a subprocess and + drives it through the real router over real gRPC. Found and fixed a + real process-lifecycle bug while getting it stable: the spawned + `uv run python ...` forks rather than exec'ing on this machine, so + killing only the immediate child (the `uv` process) left the actual + model server running, orphaned and reparented to `launchd`, still + listening on the test's port after the test had already passed -- + confirmed live (`ps` showed `PPID=1`) after a manually + `tail`-piped run hung for 27+ minutes because the orphan also held + an inherited stderr fd the pipe was waiting on for EOF. Fixed by + putting the child in its own process group + (`std::os::unix::process::CommandExt::process_group(0)`) and killing + the negative-pid group in `Drop`, plus switching `stderr` from + `Stdio::inherit()` to `Stdio::piped()` so the fd-inheritance hole + can't recur even if the group-kill ever fails. Verified clean over 3 + repeated runs (no orphaned process after any of them). All 9 router + tests (8 lib + this one) pass; `cargo fmt`/`clippy -D warnings` clean. +- [x] Task 9, 2026-09-20: the real `KernelResponder` + (`scripts/gpu/phase7_kernel_responder.py`, GPU-only) -- wraps the + real `deepseek-ai/deepseek-moe-16b-base`, `fix_rope_inv_freq`, + `patch_moe_infer(resolve_backend("naive"))` (Phase 6's chosen + kernel), and a single-request `ColocatedWorker` into + `model_server.py`'s `Responder` protocol; single-flight only, per + the design's non-goals. Additive `ColocatedWorker.snapshot_active_tokens()` + (`colocated.py`) exposes in-flight (not just completed) generated + token ids so streaming can emit each token as it's produced. Its + `gpu`-marked correctness test compares the streamed output against + a same-session stock greedy generation, byte-exact -- matches this + repo's actual established correctness-gate pattern (same-session + stock reference) more closely than the design doc's shorthand + wording, which mentioned a stored Phase 6 reference file instead. + Confirmed the two `# type: ignore` comments Task 3 added for this + exact forward reference became unused once this module landed, and + removed both, exactly as predicted. `make check` green throughout + (268 tests, up from 267; the new `gpu` test skips cleanly off-GPU). +- [x] Task 10, 2026-09-20: `docker/router.Dockerfile`, + `docker/model-server.Dockerfile`, `docker-compose.yml` -- both images + build and the full stack works end to end, but not on the first + attempt. Three real bugs found and fixed against the actual built + images, none visible from reading the Dockerfiles alone: (1) + `pyproject.toml` declares `readme = "README.md"`, which `uv sync`'s + build step needs but the model-server Dockerfile never copied in -- + `uv sync` failed outright; (2) the generated `dispatch_pb2.py` + imports `google.protobuf` directly at runtime, previously available + only transitively via the dev-only `grpcio-tools` -- a `--no-dev` + image had `grpcio` but not `protobuf`, `ModuleNotFoundError` on + startup; added `protobuf>=7.36.2` (the version already resolved in + `uv.lock`) as an explicit runtime dependency; (3) `uv run` (no + `--no-dev`) re-syncs and reinstalls dev tools (mypy, ruff, pytest) + into the container on every start despite the image being built + with `--no-dev` -- switched the entrypoint to `uv run --no-dev`. + A fourth issue surfaced once the first three were fixed and the + stack actually ran: the router's one-shot startup connect raced the + model server's own startup and exited immediately on the first + failure (`depends_on` only waits for the container to start, not + for the app inside it to be ready) -- fixed with a bounded 30x1s + retry loop in `main.rs` (`connect_with_retry`), plus + `restart: on-failure` on the compose service as defense in depth. + Verified: `POST /generate` through the containerized router and + model server returned real streamed text and timing, `/metrics` + showed real recorded values, matching the earlier bare-binary smoke + test from Task 7. `make check` green throughout (Rust: `cargo + test`/`fmt`/`clippy` all clean, 9 tests; Python: 268 tests, `mypy + --strict` and `ruff` clean). +- [x] Task 11, 2026-09-20: K8s manifests (`k8s/namespace.yaml`, + `router-deployment.yaml`, `model-server-external.yaml`, + `prometheus.yaml`, `grafana.yaml`) and `scripts/run_kind_demo.sh`, + fully rehearsed end to end against the stub responder (no GPU): + `kind create cluster` -> build and `kind load` both images -> apply + manifests -> both rollouts succeed -> port-forward -> a real + `POST /generate` through the router pod, the `ExternalName` Service, + and `host.docker.internal` (Task 1's verified finding) reached the + locally-running stub server and returned real generated text -> + Prometheus's own API confirmed it was actually scraping the + router's real `dispatch_router_requests_total` (value 1, matching + the one request fired) -> Grafana's `/api/health` responded -> + `kind delete cluster` torn down cleanly, no orphaned processes. + One deviation from the plan's own snippet, applied before the + rehearsal rather than found by it: the router's Task 10 startup + retry loop can take up to 30s, so the router Deployment's liveness/ + readiness probes needed `periodSeconds`/`failureThreshold` past + the plan's original `initialDelaySeconds: 2` alone, or K8s would + restart the pod mid-retry; set to tolerate the full 30s window. + Confirmed live: the rehearsal's rollout succeeded on the first try + with the wider probes. `ExternalName`'s pure-DNS semantics (no + port remapping) mean this same manifest set, unmodified, is what + Task 13 points at the real SSH tunnel instead of the local stub. +- [x] Task 12, 2026-09-20: `make check`/`make check-fast` are Rust-aware + (`router-lint`: `cargo fmt --check` + `cargo clippy -D warnings`; + `router-test`: `cargo test`) -- "green before push" is one command + covering both languages again. Ran `make check` for real: all 5 + steps pass (268 Python tests, 9 Rust tests). +- [x] Task 13, 2026-09-21: the one real paid GPU session. Pod + `maljfft9qy5iaa`, RunPod Secure Cloud, NVIDIA A40 ($0.49/hr, the + quoted L40 sold out during pod creation). The gpu-marked + correctness test (`test_kernel_responder_matches_same_session_stock_greedy_text`) + passed -- but only after a second run with `__pycache__` cleared + and `python -B`, tracing a same-checksum re-run's identical + failure to stale bytecode on the pod's network-mounted + `/workspace`. Two real bugs found and fixed live, neither in code + any earlier task had exercised: `python scripts/run_model_server.py` + run directly can't resolve the lazy `scripts.gpu.*` import + (`sys.path[0]` is the script's own directory, not the repo root -- + fixed by invoking `python -m scripts.run_model_server` instead, a + runbook-level fix); and the model's own shipped `tokenizer.json` + is a byte-level BPE vocabulary (47,723 of 100,000 entries carry + the GPT-2-style `Ġ` marker) wired to a SentencePiece `Metaspace` + pre-tokenizer/decoder expecting a `▁` marker that appears in zero + vocab entries -- encode() silently dropped every word boundary, + decode() produced glued-together or literal-`Ġ`-leaking text. + Fixed for real (not a pod-local shim) with + `fix_tokenizer_byte_level()` in + `scripts/gpu/phase7_kernel_responder.py`, verified by round-trip + token-id identity. The real demo then worked end to end: the SSH + tunnel, `scripts/run_kind_demo.sh` (unmodified from Task 11's + stub rehearsal), a real `POST /generate` returning real generated + text and real `ttft_ms`/`inter_token_latencies_ms`, a Grafana + screenshot and screen recording captured as evidence (PII + redacted before publishing, at the user's request -- terminal + username/hostname in the prompt and title bar on every visible + line, plus one app-switcher tooltip flash, all found by + frame-by-frame review of the raw recording). Cost: **$6.8563** + (RunPod's billing API) against a **$5 cap**, exceeded and + disclosed mid-session -- see the findings doc for why. Pod + terminated and verified gone. Full account: + `docs/findings/phase-7/2026-09-21-phase-7-productionization-run.md`; + runbook: `docs/runbooks/phase-7-productionization.md`. +- [x] Task 14, 2026-09-21: findings doc, runbook, this STATUS entry, + README and CLAUDE.md refreshed with Phase 7's summary, final + `make check` green. + +**Phase 7 is complete.** The system design's 8-phase plan (SS7) is now +fully executed, Phase 0 through 7 -- the productionization phase closes +the gap Phase 7's own design doc was added to fill (a real serving path, +not just kernels and benchmarks): a Rust router (tonic gRPC, axum HTTP, +Prometheus metrics) in front of the existing Python model server per +ADR-0003, containerized, deployed to a local `kind` Kubernetes cluster, +observed with Prometheus/Grafana, demonstrated once against a real rented +GPU over an SSH tunnel -- with two real bugs (one a `sys.path` footgun, +one a genuine third-party tokenizer defect traced to its exact missing +vocabulary data) found and fixed along the way, exactly the kind of +real-hardware surprise every prior phase's paid session also turned up. + ## Next step Phases 3 (PR #3), 4 (PR #4), 5a (PR #5), 5b (PR #6), and 6 (PR #8) are all merged, each on its own branch per the one-branch-per-phase convention. - -**Phase 6 merged to `main` via PR #8, 2026-09-20** (design, plan, all 15 -tasks, findings, runbook, a whole-branch review that found and fixed 20 -real issues, and a corrected headline result -- see "Phase 6 progress"). -Branch `phase-6-final-benchmark` deleted post-merge. The system design's -phase table (§7) is now fully executed, Phase 0 through 6; Phase 7 -(productionization: Rust router, Docker, K8s demoed once) is the only -phase left unstarted. +Phase 7 is complete on branch `phase-7-productionization` (see "Phase 7 +progress"), PR not yet opened -- the next action is asking the user for +push/PR approval, per this repo's standing convention, then opening it. + +**The system design's phase table (§7) is now fully executed, Phase 0 +through 7.** There is no next phase; `dispatch`'s implementation work +against the original design doc is done. Any further work is +maintenance, responding to Phase 2's open upstream PR review, or a +deliberately new scope decided the same way every phase here started -- +brainstormed and written down before any code. Phase 2's PR is still open and awaiting maintainer review; no further work planned on it beyond responding to review feedback. diff --git a/docs/design/2026-09-20-phase-7-productionization.md b/docs/design/2026-09-20-phase-7-productionization.md new file mode 100644 index 0000000..96901fe --- /dev/null +++ b/docs/design/2026-09-20-phase-7-productionization.md @@ -0,0 +1,237 @@ +# Phase 7 design: productionization (Rust router + Docker + K8s) + +Per `docs/design/2026-09-14-dispatch-system-design.md` §7 (phase plan, row +7) and §8 (Phase 7 detail), and `docs/adr/0003-rust-router-over-python-only-server.md` +(the TGI-shaped Rust-router/Python-model-server split, already accepted). +Phases 0-6 are complete and merged; the system design's phase table is now +fully executed except this one. This design fills in what §8 left at +architecture-sketch level: the concrete split of router responsibilities, +the demo topology, and the implementation plan's starting point. + +## 1. Purpose and thesis + +Every phase through Phase 6 answered a kernel- or engine-level performance +question. This phase answers a different one: **can dispatch's kernel work +be reached as a real, observable, deployable service**, the way a +production inference-engineering team would actually ship it -- not a new +performance claim, and not a rerun of Phase 6's race. + +Added specifically because checking real inference-engineer job postings +(national and Atlanta-metro) against the original design found a gap this +project didn't otherwise cover: containerized, production-shaped +deployment with live observability, distinct from a one-shot benchmark +script. ADR-0003 already settled *how* to close that gap (TGI's own +documented three-tier split: Rust router, Python model server, gRPC +between them) and *why Rust* specifically (the router is the "last mile" +layer where Python's GIL and interpreter overhead matter most, per the +research ADR-0003 cites). This design is the *what exactly gets built*. + +## 2. Scope, decided during brainstorming + +- **The Python model server runs real inference on a real rented GPU**, + not a stub. Reuses dispatch's naive bf16 kernel -- the exact config + Phase 6 already used for its own concurrency-1 served row (21.04 + tok/s, `docs/findings/phase-6`), not re-decided from scratch here -- + serving the real `deepseek-ai/deepseek-moe-16b-base`. Consistent with + CLAUDE.md's "never + quote a benchmark number that wasn't measured" -- the Grafana dashboard + this phase produces shows real TTFT/ITL from a real run, not a + fixed-latency fake. Rejected: a lightweight/stub model, which would + prove the plumbing but make the observability demo cosmetic. +- **K8s target is a local `kind` cluster**, not a rented cloud control + plane. Free, reproducible, scriptable teardown -- proves the K8s + manifests are real and deployable without the cost-discipline overhead + a cloud cluster would add on top of the GPU rental this phase already + needs. Matches design doc §8's "K8s demoed once and torn down, not run + continuously." +- **Router-to-model-server connectivity is an SSH local-forward**, not a + publicly exposed port on the rented pod. The in-cluster router reaches + the tunnel endpoint through a Kubernetes `ExternalName` Service pointing + at `localhost:` on the dev machine. This is the same + SSH-based access pattern Phase 5b and Phase 6 already use to pull + evidence off a rented pod -- no new public-exposure surface, no new + auth mechanism to build. Rejected: a public gRPC port on the pod + (would need real authn/allowlisting to be responsible with a real + inference server); running the whole stack on the GPU pod itself + (would need K8s tooling installed fresh on a rented box each session + instead of persisting locally, and revisits the local-`kind` decision). +- **The router does not reimplement continuous batching.** Phase 4 + (`colocated.py`) already has a tested continuous-batching worker. The + router owns the client-facing concerns TGI's own router actually owns + -- HTTP/gRPC ingress, admission queueing, forwarding to the model + server's streaming `Generate` call, and Prometheus metrics -- while the + model server keeps the batching loop. Keeps "Rust is scoped to the + router only" (ADR-0003) from quietly expanding into a second batching + implementation. +- **Plain K8s manifests, not Helm.** No Helm is installed in this + environment, and this project already prefers lightweight, direct + tooling over heavier IaC for infrastructure it doesn't need to + templatize (see CLAUDE.md's reasoning for RunPod scripts over + Terraform, same logic applied here: one router, one model-server + `ExternalName`, one Prometheus, one Grafana -- nothing that benefits + from a chart's parameterization). +- **$5 GPU cap.** Smaller than every prior phase's cap because this phase + doesn't run a new benchmark sweep -- it runs one real model server for + long enough to demo a live request through the full stack and capture + metrics, matching Phase 5a's actual cost class. +- **Screenshot capture is automated.** Once Grafana is port-forwarded + locally, a browser-automation pass (Playwright/Chrome DevTools) + navigates to the dashboard and saves the screenshot into + `docs/findings/phase-7/` directly, rather than a manual step. +- **No standing public deployment -- considered and explicitly declined.** + A real GPU-backed public endpoint would cost real money continuously + (an L40/A40-class GPU left running prices out around $500-600/month) + and directly contradicts this project's own cost-discipline rules + (CLAUDE.md: "never leave a rented GPU idle," "budget cap set before the + first rental") that every prior phase followed. In place of a live URL, + the recruiter-facing artifact is a **short screen recording** of the + live `kind` demo (terminal bring-up, then Grafana panels populating + with real request traffic) captured alongside the screenshot and + embedded in the README -- free to produce, no standing infrastructure, + and no honesty-labeling burden (an always-on public endpoint that + wasn't actually backed by the real model at click-time would need + careful labeling to not misrepresent what's live, which this project's + "never quote a number that wasn't measured" ethos leans hard against). + +## 3. Components + +```text +proto/dispatch.proto shared gRPC contract: streaming Generate + RPC (prompt in; per-token {text, t_first, + t_emit} out). Compiled by both sides -- + tonic-build/prost-build (Rust) and + grpcio-tools (Python) each vendor their + own protoc, no new system dependency. + +router/ new Cargo workspace + src/main.rs tonic gRPC + axum HTTP entrypoints + src/queue.rs FIFO admission queue, backpressure + src/client.rs tonic client -> model server's Generate + src/metrics.rs Prometheus registry: TTFT, inter-token + latency, queue depth, requests/sec + src/health.rs /healthz, /readyz for K8s probes + +src/dispatch/serving/model_server.py new: gRPC server adapting Phase 4's + colocated continuous-batching worker + + Phase 6's chosen kernel to the shared + proto's Generate RPC + +docker/router.Dockerfile multi-stage Rust build -> slim runtime +docker/model-server.Dockerfile Python + CUDA base, installs dispatch +docker-compose.yml router + model server, local + verification before any K8s step + +k8s/ + namespace.yaml + router-deployment.yaml Deployment + Service + ConfigMap + model-server-external.yaml ExternalName Service -> SSH-tunnel + endpoint (host.docker.internal or the + kind node's host-gateway, checked live) + prometheus.yaml minimal Deployment + ConfigMap (scrape + config pointed at the router's /metrics) + grafana.yaml Deployment + one imported dashboard JSON + +scripts/run_kind_demo.sh scripted bring-up: kind create, apply + manifests, open the SSH tunnel, port- + forward Grafana, fire one demo request, + scripted teardown + +docs/findings/phase-7/ demo evidence: metrics JSON/exported + Grafana panel data, the screenshot, + the screen recording, cost record +``` + +## 4. Data flow + +Client -> router (`Generate`, HTTP or gRPC) -> admission queue -> router's +tonic client streams to the tunneled model server -> model server runs +continuous-batched decode on the real GPU using Phase 6's chosen kernel -> +tokens stream back through the router, each hop timestamped -> router +records TTFT/inter-token-latency/queue-depth into its Prometheus registry +-> Prometheus scrapes `/metrics` -> Grafana dashboard (imported once) +renders it live during the demo window -> one screenshot captured as +evidence. + +## 5. Testing + +Extends the repo's existing table (CLAUDE.md, "Testing policy") across the +new language boundary: + +| Layer | What must be covered | Runs in CI? | +|---|---|---| +| Router unit (`cargo test`) | queue admission/ordering under backpressure; metrics recorded correctly per request; proto (de)serialization round-trips | yes | +| Router integration | router's tonic client against a fake in-process Python `Generate` server (grpcio test server, no real weights, no GPU) -- proves the gRPC wiring itself | yes | +| Model server unit (pytest) | the proto adapter around `colocated.py`'s already-tested loop; a malformed request is refused, not silently dropped | yes | +| Real end-to-end (`gpu`-marked, paid) | router -> SSH tunnel -> real model server -> real GPU -> one real streamed response, checked against a known-good token sequence from Phase 6's reference | no -- pod-only, excluded from CI like every other phase's `gpu` tests, reason visible in the test | +| K8s demo | scripted `kind` bring-up (`scripts/run_kind_demo.sh`), port-forward, one live request through the full stack, Grafana screenshot as evidence | manual/scripted, once, not part of the automated suite | + +`make check` gains a Rust-aware counterpart (`cargo test` and `cargo +clippy` invoked alongside the existing lint/typecheck/test gate) so "green +before push" stays one rule covering both languages, not two separate +gates to remember. + +## 6. Non-goals + +- **A second benchmark race.** Phase 6 already answered the + dispatch-vs-vLLM-vs-SGLang performance question. This phase's metrics + are operational telemetry from one live demo run, not a new comparative + claim. +- **Autoscaling, multi-replica routing, or a real load balancer.** One + router replica, one model server, demoed once. Production-shaped + plumbing, not a production-scale deployment. +- **A public-facing deployment.** Considered explicitly during + brainstorming and declined -- see §2's "no standing public deployment" + entry for the cost and honesty-labeling reasoning. The SSH-tunnel + topology is deliberately local-demo-only; no cloud ingress, no TLS + termination, no auth layer beyond what the tunnel itself provides. The + recorded demo (§2, §7) is the recruiter-facing artifact instead. +- **Rewriting the model server's inference path in Rust.** ADR-0003 + already ruled this out; Rust stays scoped to the router. +- **A continuously running K8s deployment.** `kind` is created, demoed, + and torn down in one scripted session, per design doc §8. + +## 7. Risk, cost, and rollout + +**Cost.** One rented GPU pod (RunPod, L40-class, checked live at rental +time), just long enough to bring up the model server and run the K8s demo +against it. **Budget cap: $5**, set before the first rental -- smaller +than every prior phase because this is a demo session, not a sweep. + +**Cost discipline, from CLAUDE.md, applied to this phase:** + +- Develop and test the router, the proto, and the K8s manifests entirely + against the fake in-process Python server first -- rent only for the + final real-GPU demo session. +- Open the SSH tunnel and run the full demo (including the Grafana + screenshot capture) in one continuous session; never leave the pod + idle waiting on a decision mid-session. +- Pull the demo evidence (metrics export, screenshot, cost record) before + `stop`, not after -- same rule Phase 5b's incident established. + +**Risks:** + +- **Rust is new tooling for this environment** (no `cargo`/`rustc` + installed). Budgeted as real setup time in the implementation plan, not + assumed free -- per ADR-0003's own "Consequences" section. +- **`kind`'s network path to a host-machine SSH tunnel is environment- + specific** (Docker Desktop's `host.docker.internal` vs. a Linux kind + node's host-gateway alias). The plan's first task verifies this + concretely on this machine before any other router/K8s work depends on + it, rather than assuming a document's example works unmodified. +- **Streaming gRPC across the tunnel adds a real latency hop** the + dashboard will show honestly (SSH forwarding overhead on top of the + model's own TTFT). Reported as part of the topology, not hidden or + subtracted out. +- **Phase 6's chosen kernel may need re-verifying on whatever GPU this + phase actually rents** (a different L40 instance, possibly a different + driver). The real end-to-end `gpu` test re-checks this before the demo + runs, the same correctness-before-speed rule every prior phase followed. + +**Rollout.** Branch `phase-7-productionization`, one commit per plan task, +`docs/STATUS.md` updated in the same commit as each piece of work, one PR +at the end. Findings, the Grafana screenshot, and the demo screen recording +in `docs/findings/phase-7/`. README, CLAUDE.md and the story-bank gist +refreshed at completion -- the README refresh specifically embeds the demo +recording, since it's this phase's main recruiter-facing artifact -- and +since this closes the system design's phase table (§7) entirely, that +completion note goes in CLAUDE.md's "Current status" too. diff --git a/docs/findings/phase-7/2026-09-21-phase-7-demo-recording.mp4 b/docs/findings/phase-7/2026-09-21-phase-7-demo-recording.mp4 new file mode 100644 index 0000000..d02c10f Binary files /dev/null and b/docs/findings/phase-7/2026-09-21-phase-7-demo-recording.mp4 differ diff --git a/docs/findings/phase-7/2026-09-21-phase-7-gpu-correctness-test.log b/docs/findings/phase-7/2026-09-21-phase-7-gpu-correctness-test.log new file mode 100644 index 0000000..56f4ad5 --- /dev/null +++ b/docs/findings/phase-7/2026-09-21-phase-7-gpu-correctness-test.log @@ -0,0 +1,28 @@ +============================= test session starts ============================== +platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0 -- /workspace/dispatch/.venv/bin/python +cachedir: .pytest_cache +rootdir: /workspace/dispatch +configfile: pyproject.toml +plugins: anyio-4.15.1, cov-7.1.0 +collecting ... collected 1 item + +tests/unit/test_phase7_kernel_responder.py::test_kernel_responder_matches_same_session_stock_greedy_text PASSED [100%] + +=============================== warnings summary =============================== +.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1345 + /workspace/dispatch/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py:1345: PytestAssertRewriteWarning: Module already imported so cannot be rewritten; anyio + self._mark_plugins_for_rewrite(hook, disable_autoload) + +tests/unit/test_phase7_kernel_responder.py::test_kernel_responder_matches_same_session_stock_greedy_text +tests/unit/test_phase7_kernel_responder.py::test_kernel_responder_matches_same_session_stock_greedy_text + /workspace/dispatch/.venv/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:71: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`. + warnings.warn(DEPRECATION_MESSAGE, FutureWarning) + +tests/unit/test_phase7_kernel_responder.py::test_kernel_responder_matches_same_session_stock_greedy_text +tests/unit/test_phase7_kernel_responder.py::test_kernel_responder_matches_same_session_stock_greedy_text + /workspace/dispatch/.venv/lib/python3.12/site-packages/transformers/modeling_attn_mask_utils.py:281: FutureWarning: The attention mask API under `transformers.modeling_attn_mask_utils` (`AttentionMaskConverter`) is deprecated and will be removed in Transformers v5.10. Please use the new API in `transformers.masking_utils`. + warnings.warn(DEPRECATION_MESSAGE, FutureWarning) + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================== 1 passed, 5 warnings in 1422.60s (0:23:42) ================== +GPUTEST_EXIT_CODE=0 diff --git a/docs/findings/phase-7/2026-09-21-phase-7-gpu-env-info.log b/docs/findings/phase-7/2026-09-21-phase-7-gpu-env-info.log new file mode 100644 index 0000000..5ec9212 --- /dev/null +++ b/docs/findings/phase-7/2026-09-21-phase-7-gpu-env-info.log @@ -0,0 +1,24 @@ +Mon Sep 21 14:26:26 2026 ++-----------------------------------------------------------------------------------------+ +| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 | ++-----------------------------------------+------------------------+----------------------+ +| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | +| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | +| | | MIG M. | +|=========================================+========================+======================| +| 0 NVIDIA A40 On | 00000000:56:00.0 Off | 0 | +| 0% 48C P0 119W / 300W | 34491MiB / 46068MiB | 0% Default | +| | | N/A | ++-----------------------------------------+------------------------+----------------------+ + ++-----------------------------------------------------------------------------------------+ +| Processes: | +| GPU GI CI PID Type Process name GPU Memory | +| ID ID Usage | +|=========================================================================================| +| No running processes found | ++-----------------------------------------------------------------------------------------+ +--- +PRETTY_NAME="Ubuntu 24.04.3 LTS" +NAME="Ubuntu" +VERSION_ID="24.04" diff --git a/docs/findings/phase-7/2026-09-21-phase-7-grafana-dashboard.png b/docs/findings/phase-7/2026-09-21-phase-7-grafana-dashboard.png new file mode 100644 index 0000000..f42b0fa Binary files /dev/null and b/docs/findings/phase-7/2026-09-21-phase-7-grafana-dashboard.png differ diff --git a/docs/findings/phase-7/2026-09-21-phase-7-metrics.txt b/docs/findings/phase-7/2026-09-21-phase-7-metrics.txt new file mode 100644 index 0000000..2d9116a --- /dev/null +++ b/docs/findings/phase-7/2026-09-21-phase-7-metrics.txt @@ -0,0 +1,38 @@ +# HELP dispatch_router_inter_token_latency_seconds Gap between consecutive tokens, seconds +# TYPE dispatch_router_inter_token_latency_seconds histogram +dispatch_router_inter_token_latency_seconds_bucket{le="0.005"} 7 +dispatch_router_inter_token_latency_seconds_bucket{le="0.01"} 7 +dispatch_router_inter_token_latency_seconds_bucket{le="0.025"} 7 +dispatch_router_inter_token_latency_seconds_bucket{le="0.05"} 81 +dispatch_router_inter_token_latency_seconds_bucket{le="0.1"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="0.25"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="0.5"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="1"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="2.5"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="5"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="10"} 118 +dispatch_router_inter_token_latency_seconds_bucket{le="+Inf"} 118 +dispatch_router_inter_token_latency_seconds_sum 5.464371751000002 +dispatch_router_inter_token_latency_seconds_count 118 +# HELP dispatch_router_queue_depth Requests currently waiting for the single-flight model server +# TYPE dispatch_router_queue_depth gauge +dispatch_router_queue_depth 0 +# HELP dispatch_router_requests_total Total requests admitted by the router +# TYPE dispatch_router_requests_total counter +dispatch_router_requests_total 9 +# HELP dispatch_router_ttft_seconds Time to first token, seconds +# TYPE dispatch_router_ttft_seconds histogram +dispatch_router_ttft_seconds_bucket{le="0.005"} 0 +dispatch_router_ttft_seconds_bucket{le="0.01"} 0 +dispatch_router_ttft_seconds_bucket{le="0.025"} 0 +dispatch_router_ttft_seconds_bucket{le="0.05"} 0 +dispatch_router_ttft_seconds_bucket{le="0.1"} 0 +dispatch_router_ttft_seconds_bucket{le="0.25"} 6 +dispatch_router_ttft_seconds_bucket{le="0.5"} 6 +dispatch_router_ttft_seconds_bucket{le="1"} 7 +dispatch_router_ttft_seconds_bucket{le="2.5"} 7 +dispatch_router_ttft_seconds_bucket{le="5"} 7 +dispatch_router_ttft_seconds_bucket{le="10"} 9 +dispatch_router_ttft_seconds_bucket{le="+Inf"} 9 +dispatch_router_ttft_seconds_sum 14.428644759 +dispatch_router_ttft_seconds_count 9 diff --git a/docs/findings/phase-7/2026-09-21-phase-7-model-server.log b/docs/findings/phase-7/2026-09-21-phase-7-model-server.log new file mode 100644 index 0000000..0a4604f --- /dev/null +++ b/docs/findings/phase-7/2026-09-21-phase-7-model-server.log @@ -0,0 +1,14 @@ +Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. + Loading weights: 0%| | 0/5466 [00:00>> tok.tokenize("hello world") +['hell', 'ow', 'orld'] # no space marker anywhere +>>> tok.backend_tokenizer.pre_tokenizer +Metaspace(replacement="▁", prepend_scheme=always, split=False) +>>> sum(1 for k in vocab if '▁' in k) # SentencePiece's own marker +0 # ...out of 100,000 vocab entries +>>> sum(1 for k in vocab if 'Ġ' in k) # GPT-2 byte-level BPE's marker +47723 # ...nearly half the vocab +``` + +`deepseek-ai/deepseek-moe-16b-base`'s shipped `tokenizer.json` is a +byte-level BPE vocabulary (GPT-2/RoBERTa-style, using `Ġ`/`Ċ` for +space/newline) wired to a SentencePiece-style `Metaspace` pre-tokenizer +and decoder — which look for a `▁` marker that appears in zero of the +vocabulary's 100,000 entries. `encode()` silently loses every word +boundary before the model ever sees the prompt; `decode()` either drops +the (already-lost) spaces or, on tokens that do carry a literal `Ġ`/`Ċ` +character, leaks it through unconverted. Every prior phase fed this same +model the same mangled encoding without noticing, because Phases 0-6 +never decoded generated text for a human to read — they compared logits +and token ids, which this bug does not disturb. + +Rewiring both the pre-tokenizer and decoder to `ByteLevel` (matching what +the vocabulary actually is) fixes it completely, verified by round-trip +identity: + +```python +tok.backend_tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel( + add_prefix_space=False, use_regex=True +) +tok.backend_tokenizer.decoder = decoders.ByteLevel() +assert tok(tok.decode(ids))["input_ids"] == ids # True, on every prompt tested +``` + +Landed as `fix_tokenizer_byte_level()` in +`scripts/gpu/phase7_kernel_responder.py`, called once in +`build_kernel_responder()` right after the tokenizer loads — a real, +committed fix (not a pod-local monkeypatch like the `transformers` +version shims), since it corrects the tokenizer's own wiring rather than +working around a dependency-version mismatch, and applies on any machine, +any `transformers` version. Scoped to this file only, not to the shared +benchmark harness — Phases 0-6 are already merged and never relied on +decoded-text correctness, so there was nothing there to fix. + +## GPU-marked correctness gate + +`test_kernel_responder_matches_same_session_stock_greedy_text` passed — +but not on the first re-run after the tokenizer fix shipped. A checksum +of the fixed file matched byte-for-byte between the dev machine and the +pod, yet the second run reproduced the exact pre-fix failure and took +longer (1422s vs. the first run's 1247s). Hypothesis: stale `.pyc` +bytecode on `/workspace`, a network-mounted FUSE filesystem +(`mfs#ca-mtl-1.runpod.net`) whose mtime-based cache invalidation isn't +reliable for Python's bytecode cache. Cleared every `__pycache__` under +`/workspace/dispatch` and re-ran with `python -B` (bytecode caching +disabled outright, eliminating the variable rather than trusting the +clear alone) — passed. The test itself is orthogonal to the tokenizer bug +above: it compares dispatch's kernel-patched generation path against a +same-session stock-`moe_infer` reference using identical token ids, never +decoded text, so the tokenizer's correctness (or lack of it) does not +affect what this gate proves. + +## The real demo + +`scripts/run_kind_demo.sh`, unmodified from Task 11's stub rehearsal, +pointed at an SSH tunnel (`ssh -L 50051:localhost:50051 root@`) +instead of a local stub. The router's `model-server` `ExternalName` +Service resolves `host.docker.internal` exactly as Task 1 found it would, +reaching the tunnel with no manifest changes. `kind` cluster up, both +Docker images built and loaded, router/Prometheus/Grafana all rolled out, +port-forwards up. + +Real request through the full stack (dev machine → `kind` router pod → +gRPC → SSH tunnel → pod → real GPU inference → back), captured in the +screen recording and matching the Grafana dashboard at the same moment: + +``` +curl -s -X POST localhost:8080/generate \ + -H 'content-type: application/json' \ + -d '{"prompt":"The quick brown fox","max_new_tokens":20}' +``` + +```json +{ + "text": " jumps over the lazy dog.\nThe quick brown fox jumps over the lazy dog.\nThe quick", + "ttft_ms": 136.455917, + "inter_token_latencies_ms": [47.916833, 49.81525, 47.626792, 47.981041, + 50.226084, 47.229666, 49.312167, 47.594209, 47.434166, 49.490209, + 48.392291, 42.585334, 40.931416, 40.329042, 42.170208, 45.041084, + 45.057458, 43.674833] +} +``` + +Real, correctly-spaced generated text (post-fix), real measured timing +from actual GPU inference — not the stub's canned tokens. + +**Grafana dashboard** (embedded below), confirming Prometheus scraped the +router's real metrics, not synthetic data: + +![Grafana dashboard showing real Phase 7 demo metrics](2026-09-21-phase-7-grafana-dashboard.png) + +**Screen recording** (PII redacted — see below): +[`2026-09-21-phase-7-demo-recording.mp4`](2026-09-21-phase-7-demo-recording.mp4) + +Aggregate metrics across the full session (`/metrics`, exported to +`2026-09-21-phase-7-metrics.txt`), 9 requests admitted total: + +| Metric | Value | +|---|---| +| Requests total | 9 | +| Mean TTFT | 1.603s (skewed by 2 cold starts: 6.76s the very first request, 6.07s after the model-server restart to ship the tokenizer fix; the other 7 requests were all 136-663ms) | +| Mean inter-token latency | 46.3ms (118 samples, 5.464s summed) | +| Queue depth at rest | 0 | + +The cold-start inflation matches every prior phase's first-request +pattern (CUDA kernel compilation/Triton autotuning on first invocation); +not re-measured as a clean "warm" number here since Phase 7's goal is a +working demo, not a new latency benchmark — Phase 6 already owns that +claim on this same kernel. + +**PII redaction, requested by the user before publishing:** the raw +recording showed the presenter's local username and machine hostname in +the terminal's title bar and shell prompt (every line), plus one +`Cmd+Tab` app-switcher tooltip mid-recording that briefly exposed a VS +Code workspace name containing the same username. Rectangular redaction +boxes, timed to the terminal's two scroll states (verified frame-by-frame +against the raw recording) cover the prompt on every line without +touching the commands or output typed after it; the ~1.8s window +containing the actual window-switch animation (rotated/warped geometry a +static box can't track) plus the tooltip flash is replaced with a solid +black frame instead of attempting pixel-tracked redaction. Re-encoded +with `ffmpeg`/`libx264`; the raw, unredacted capture was deleted after +the redacted version was verified frame-by-frame at every relevant +timestamp. + +## Deviations from the plan + +- **Task 1's `host.docker.internal` finding held under real load**: the + `ExternalName` Service needed zero changes between the stub rehearsal + and the real GPU-backed tunnel, exactly as predicted when it was first + verified. +- **Task 13 Step 5's literal invocation doesn't work** — see bug 1 above; + the runbook records `python -m scripts.run_model_server`, not + `python scripts/run_model_server.py`. +- **A tokenizer defect neither the plan nor any earlier phase + anticipated** — see bug 2 above — required an unplanned fix landed in + `scripts/gpu/phase7_kernel_responder.py` before the demo could produce + readable output. +- **The correctness gate needed a second run** with bytecode caching + disabled, not anticipated by Step 4's plain `pytest -m gpu` invocation + — see the correctness-gate section above. +- **Budget cap exceeded** ($6.8563 against $5), disclosed mid-session, + user chose to continue rather than stop early — see Infrastructure + above. + +## What this closes + +Phase 7 is the last phase in the system design's 8-phase plan (§7). +`dispatch` now has, in addition to the custom kernels and multi-GPU +serving work of Phases 1-6: a real Rust router in front of a real Python +model server (ADR-0003's TGI-shaped split), containerized, deployed to a +local Kubernetes cluster, observed with Prometheus/Grafana, demonstrated +once against a real rented GPU over an SSH tunnel — and, along the way, +a genuine third-party tokenizer defect found, root-caused to the exact +missing vocabulary data, and fixed. diff --git a/docs/findings/phase-7/2026-09-21-phase-7-runpod-pod-details.png b/docs/findings/phase-7/2026-09-21-phase-7-runpod-pod-details.png new file mode 100644 index 0000000..dcf2505 Binary files /dev/null and b/docs/findings/phase-7/2026-09-21-phase-7-runpod-pod-details.png differ diff --git a/docs/findings/phase-7/2026-09-21-phase-7-terminal-demo.png b/docs/findings/phase-7/2026-09-21-phase-7-terminal-demo.png new file mode 100644 index 0000000..96c35be Binary files /dev/null and b/docs/findings/phase-7/2026-09-21-phase-7-terminal-demo.png differ diff --git a/docs/plans/2026-09-20-phase-7-productionization-plan.md b/docs/plans/2026-09-20-phase-7-productionization-plan.md new file mode 100644 index 0000000..a26a817 --- /dev/null +++ b/docs/plans/2026-09-20-phase-7-productionization-plan.md @@ -0,0 +1,2591 @@ +# Phase 7 productionization (Rust router + Docker + K8s) -- Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Task 13 rents a GPU and spends real money: never start it without the user's explicit go-ahead, and read "Cost and stop rules" first.** + +**Goal:** Put dispatch's real kernel work behind a real, observable, deployable service -- a Rust router (gRPC to a Python model server, HTTP to clients, Prometheus metrics) demoed once through a local `kind` Kubernetes cluster against the real `deepseek-ai/deepseek-moe-16b-base` model on a rented GPU, with Grafana showing live TTFT/inter-token-latency/queue-depth from that real run. + +**Architecture:** A Cargo binary+library crate (`router/`) owns HTTP ingress, single-slot admission, a tonic gRPC client, and a Prometheus registry. A Python gRPC server (`src/dispatch/serving/model_server.py`) implements the same shared `proto/dispatch.proto` contract behind a `Responder` protocol with two implementations: `StubResponder` (canned, no model, used everywhere except the final paid session) and `KernelResponder` (real model + Phase 6's naive kernel + Phase 4's `ColocatedWorker`, GPU-only). Everything is built and proven against the stub first -- unit tests, a cross-language integration test, Docker Compose, and a full `kind` rehearsal -- before the one real GPU session swaps the stub for the kernel behind an SSH tunnel. + +**Tech Stack:** Rust (stable, via rustup), tonic 0.14.6, prost 0.14.4, axum 0.8.9, tokio 1.53.1, the `prometheus` crate 0.14.0 (all checked live 2026-09-20 against crates.io); Python 3.12+, `uv`, `ruff`, `mypy --strict`, `pytest`, grpcio/grpcio-tools 1.84.0 (checked live against pypi.org); Docker; `kind` v0.33.0 (checked live against GitHub releases); `kubectl` (already installed, client v1.30.5). + +**Spec:** `docs/design/2026-09-20-phase-7-productionization.md`. + +## Global Constraints + +Every task's requirements implicitly include this section. + +- Python side: 3.12+; `uv`, `ruff` (line length 100), `mypy --strict`, `pytest`. Rust side: stable toolchain via `rustup`, `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, `cargo test`. `make check` (Task 12 onward) runs both gates and is green before every commit and before any push. +- **Correctness before speed, still.** The real GPU session (Task 13) re-verifies `KernelResponder` against a same-session stock generation before it's ever wired behind the router -- the same bar every prior phase's kernel work met. +- **Never quote a number that wasn't measured.** The Grafana dashboard's TTFT/inter-token-latency/queue-depth values come only from the real Task 13 session; nothing in this plan fabricates or estimates them. +- `gpu`-marked pytest tests are excluded from CI, reason visible in the test, same as every earlier phase. +- One commit per task. `docs/STATUS.md` is updated in the same commit as the work it describes. +- **Commit messages are plain ASCII: `--`, never an em-dash. No attribution lines in commit messages or PR descriptions.** +- One branch for the phase: `phase-7-productionization`. One PR at the end. Do not push or open the PR without asking the user. +- Model: `deepseek-ai/deepseek-moe-16b-base`. Kernel: dispatch's **naive bf16** backend -- the exact config Phase 6 already used for its own concurrency-1 served row (21.04 tok/s), not re-decided here. +- Router-to-model-server connectivity: an SSH local-forward from the dev machine into the rented pod, reached from the local `kind` cluster via an `ExternalName` Service pointing at `host.docker.internal` (Task 1 verifies this concretely on this machine before anything else depends on it). +- Hardware: **one GPU pod, RunPod**, rented only for Task 13. Budget cap **$5**, a ceiling not a target, set before the first rental. +- Findings go to `docs/findings/phase-7/`. + +## Pre-registered design decisions + +Fixed here, carried over from the approved design doc and the API research done while writing this plan -- not re-decided mid-implementation. + +- **Single-flight, not multi-request batching.** `KernelResponder` and the router's `AdmissionQueue` both enforce at most one in-flight generation at a time (a lock on the Python side, a capacity-1 semaphore on the Rust side). Phase 7's non-goals explicitly rule out concurrent multi-replica serving; this keeps the implementation honest about that scope rather than building unused concurrency machinery. +- **Router exposes HTTP to clients, gRPC to the model server only.** The design doc's data-flow line ("Client -> router, HTTP or gRPC") is resolved concretely to HTTP-only for the client-facing side, matching TGI's own actual router surface and avoiding a second, unused gRPC-facing listener. +- **`router/` is a combined library+binary Cargo crate** (`src/lib.rs` re-exporting `app`, `client`, `metrics`, `pb`, `queue`; `src/main.rs` as a thin entrypoint), not binary-only -- required so `router/tests/` integration tests (Task 8) can reach the same modules the unit tests use. This is a refinement of the design doc's "Cargo workspace" phrasing; a multi-crate workspace would be unused ceremony for one binary. +- **`protoc` is vendored via the `protoc-bin-vendored` crate** (Rust) and `grpcio-tools`' own bundled `protoc` (Python) -- neither side needs a system `protoc` install, confirmed against this machine's environment check (no system `protoc` found). +- **Real end-to-end correctness (Task 9's `gpu` test) compares `KernelResponder` against a same-session stock generation on the identical prompt**, not a previously-committed Phase 6 reference file -- this matches every earlier phase's actual correctness-gate pattern (Phase 1/3/4/5a/5b all compare against a same-session stock run, never a stale committed file) more closely than the design doc's shorthand wording. +- **The screen recording is a manual capture, the Grafana screenshot is automated.** Reliably automating a multi-window (terminal + browser) screen recording from this environment isn't something to overpromise; the Grafana dashboard screenshot alone (one page, one browser tab) is reliably automatable via browser tooling and stays automated as agreed. Task 13 gives the user the exact commands to run on-camera so the manual recording is clean. + +## Cost and stop rules + +- **State the live hourly price and get the user's explicit go-ahead before creating the pod (Task 13, Step 1).** Estimated cost is not a measurement. +- **Budget cap: $5.** This is a short demo session (bring up the model server, run the gate test, run the K8s demo, capture evidence), not a sweep -- expect well under an hour of rental. +- **Never leave the pod running across an unbounded wait.** Stop it first, restart on the answer, per CLAUDE.md's cost discipline and its own cited incident. +- **Pull every evidence file off the pod before `stop` or terminate.** +- Develop and test everything (router, proto, model server, Docker, K8s manifests) against `StubResponder` first -- rent only for Task 13. + +## File structure + +| File | Responsibility | Task | +|---|---|---| +| `proto/dispatch.proto` | shared gRPC contract | 2 | +| `router/Cargo.toml`, `router/build.rs`, `router/src/lib.rs`, `router/src/pb.rs` | Rust crate scaffold, protoc-vendored codegen | 2 | +| `src/dispatch/proto/dispatch_pb2.py`, `dispatch_pb2_grpc.py`, `scripts/gen_proto.py` | Python side of the contract | 3 | +| `src/dispatch/serving/model_server.py`, `scripts/run_model_server.py` | Responder protocol, StubResponder, gRPC servicer, CLI | 3 | +| `router/src/queue.rs` | admission (single-flight semaphore + depth gauge) | 4 | +| `router/src/metrics.rs` | Prometheus registry | 5 | +| `router/src/client.rs`, `router/src/test_support.rs` | tonic client to the model server; test-only mock server helper | 6 | +| `router/src/app.rs`, `router/src/main.rs` | axum HTTP surface, wiring | 7 | +| `router/tests/cross_language_integration.rs` | router <-> real Python stub server, cross-language proof | 8 | +| `src/dispatch/serving/colocated.py` (modify), `scripts/gpu/phase7_kernel_responder.py` | real model server, GPU-only | 9 | +| `docker/router.Dockerfile`, `docker/model-server.Dockerfile`, `docker-compose.yml` | containers, local verification | 10 | +| `k8s/*.yaml`, `scripts/run_kind_demo.sh` | K8s manifests, scripted bring-up, rehearsed against the stub | 11 | +| `Makefile` (modify) | Rust-aware `make check` | 12 | +| `docs/runbooks/phase-7-productionization.md`, `docs/findings/phase-7/` | the paid session and its evidence | 13, 14 | + +Tasks 1-12 are local, free, and TDD where code is involved. Task 13 is the paid pod session. Task 14 is findings, docs, and the PR. + +--- + +### Task 1: Verify kind <-> host connectivity, install prerequisites + +**Files:** +- Modify: `docs/STATUS.md` (new "Phase 7 progress" section, this task's finding) + +**Interfaces:** +- Consumes: nothing. +- Produces: a recorded, verified answer to "what hostname does a pod inside a local `kind` cluster on this machine use to reach a listener on the dev machine's own localhost" -- every later task (11, 13) that writes `k8s/model-server-external.yaml` depends on this exact value. + +- [ ] **Step 1: Install `kind` and confirm Docker is running** + +```bash +brew install kind +kind version # expect v0.33.0 or newer +docker info >/dev/null && echo "docker OK" +``` + +- [ ] **Step 2: Create a throwaway cluster** + +```bash +kind create cluster --name connectivity-check +kubectl config use-context kind-connectivity-check +kubectl get nodes +``` + +- [ ] **Step 3: Start a listener on the dev machine standing in for the future SSH-tunneled model server** + +```bash +python3 -m http.server 50051 & +LISTENER_PID=$! +``` + +- [ ] **Step 4: From inside the cluster, try to reach it via `host.docker.internal`** + +```bash +kubectl run connectivity-probe --image=busybox --rm -it --restart=Never -- \ + wget -qO- --timeout=3 http://host.docker.internal:50051/ || echo "FAILED: host.docker.internal" +``` + +If this succeeds (prints the directory listing HTML `python3 -m http.server` serves), `host.docker.internal` is the answer and Step 5 is skipped. + +- [ ] **Step 5 (only if Step 4 failed): try the node's host-gateway alias** + +```bash +kubectl get nodes -o jsonpath='{.items[0].status.addresses}' +# then probe the printed InternalIP-style address the same way as Step 4, +# and/or recreate the cluster with a kind config adding +# extraPortMappings / a node-level extraHosts entry for host-gateway, +# per kind's own networking docs -- record exactly which config worked. +``` + +- [ ] **Step 6: Record the finding and clean up** + +Add a new "## Phase 7 progress" section to `docs/STATUS.md` stating: kind version installed, which hostname reached the dev-machine listener (`host.docker.internal` or the recorded alternative from Step 5), and that this value is what `k8s/model-server-external.yaml` (Task 11) hardcodes. + +```bash +kill $LISTENER_PID +kind delete cluster --name connectivity-check +``` + +- [ ] **Step 7: Commit** + +```bash +git checkout -b phase-7-productionization +git add docs/STATUS.md +git commit -m "docs: record Phase 7's kind-to-host connectivity finding" +``` + +--- + +### Task 2: Shared gRPC contract and Rust crate scaffold + +**Files:** +- Create: `proto/dispatch.proto` +- Create: `router/Cargo.toml`, `router/build.rs`, `router/src/lib.rs`, `router/src/pb.rs`, `router/src/main.rs` +- Modify: `.gitignore` (add `/router/target/`) + +**Interfaces:** +- Consumes: nothing. +- Produces: `dispatch_router::pb::dispatch_v1::{GenerateRequest, GenerateResponse}` (prost message types) and `dispatch_router::pb::dispatch_v1::model_server_client::ModelServerClient` / `model_server_server::{ModelServer, ModelServerServer}` (tonic-generated client/server), used by every later router task. + +- [ ] **Step 1: Install the Rust toolchain** + +```bash +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y +source "$HOME/.cargo/env" +rustc --version && cargo --version +``` + +- [ ] **Step 2: Write the proto contract** + +Create `proto/dispatch.proto`: + +```proto +syntax = "proto3"; + +package dispatch.v1; + +// The gRPC contract between the Rust router and the Python model server +// (docs/design/2026-09-20-phase-7-productionization.md SS3-4). Compiled +// by both sides -- tonic-build/prost-build (Rust, router/build.rs) and +// grpcio-tools (Python, scripts/gen_proto.py) -- each vendors its own +// protoc, so no system protoc install is required. +service ModelServer { + // Streams generated text token-by-token. The final message in the + // stream always has is_final=true and empty text; every message + // before it has is_final=false and carries exactly one token's text. + rpc Generate(GenerateRequest) returns (stream GenerateResponse); +} + +message GenerateRequest { + string request_id = 1; + string prompt = 2; + uint32 max_new_tokens = 3; +} + +message GenerateResponse { + string request_id = 1; + string text = 2; + bool is_final = 3; + double t_emit_unix = 4; +} +``` + +- [ ] **Step 3: Scaffold the Cargo crate** + +Create `router/Cargo.toml`: + +```toml +[package] +name = "dispatch-router" +version = "0.1.0" +edition = "2021" + +[dependencies] +tonic = "0.14.6" +prost = "0.14.4" +tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "sync", "time"] } +axum = "0.8.9" +serde = { version = "1.0.229", features = ["derive"] } +serde_json = "1.0.151" +prometheus = "0.14.0" + +[build-dependencies] +tonic-build = "0.14.6" +protoc-bin-vendored = "3.2.0" + +[dev-dependencies] +tower = { version = "0.5.3", features = ["util"] } +http-body-util = "0.1.5" +tokio-stream = "0.1.19" +futures-core = "0.3.34" +``` + +Create `router/build.rs`: + +```rust +fn main() -> Result<(), Box> { + let protoc_path = protoc_bin_vendored::protoc_bin_path()?; + std::env::set_var("PROTOC", protoc_path); + tonic_build::compile_protos("../proto/dispatch.proto")?; + Ok(()) +} +``` + +Create `router/src/pb.rs`: + +```rust +pub mod dispatch_v1 { + tonic::include_proto!("dispatch.v1"); +} + +#[cfg(test)] +mod tests { + use prost::Message; + + use super::dispatch_v1::GenerateRequest; + + #[test] + fn generate_request_round_trips_through_prost_encoding() { + let original = GenerateRequest { + request_id: "r1".to_string(), + prompt: "hello".to_string(), + max_new_tokens: 16, + }; + + let mut buf = Vec::new(); + original.encode(&mut buf).expect("encode"); + let decoded = GenerateRequest::decode(buf.as_slice()).expect("decode"); + + assert_eq!(decoded, original); + } +} +``` + +Create `router/src/lib.rs`: + +```rust +pub mod pb; +``` + +Create `router/src/main.rs`: + +```rust +fn main() { + println!("dispatch-router (scaffold) -- HTTP/gRPC wiring lands in Task 7"); +} +``` + +- [ ] **Step 4: Add the Rust build directory to `.gitignore`** + +In `.gitignore`, add a new section: + +``` +# Rust +/router/target/ +``` + +- [ ] **Step 5: Build and run the test** + +```bash +cd router +cargo build +cargo test +``` + +Expected: the build succeeds (protoc is fetched via `protoc-bin-vendored`, no system `protoc` needed) and `generate_request_round_trips_through_prost_encoding` passes. If `tonic_build::compile_protos`'s exact method name has moved since this was written, `cargo doc -p tonic-build --open` (against the pinned 0.14.6) shows the current API -- adjust `build.rs` accordingly; this is the one place in this plan where a fast-moving crate's exact surface can't be guaranteed word-for-word, the same class of risk Phase 6's plan flagged for SGLang's API. + +- [ ] **Step 6: Commit** + +```bash +cd .. +git add proto/dispatch.proto router/Cargo.toml router/Cargo.lock router/build.rs router/src .gitignore +git commit -m "feat: add the shared gRPC contract and Rust router crate scaffold" +``` + +--- + +### Task 3: Python side of the contract -- model server skeleton + +**Files:** +- Create: `scripts/gen_proto.py` +- Create: `src/dispatch/proto/__init__.py`, `src/dispatch/proto/dispatch_pb2.py`, `src/dispatch/proto/dispatch_pb2_grpc.py` (generated, then committed) +- Create: `src/dispatch/serving/model_server.py` +- Create: `scripts/run_model_server.py` +- Test: `tests/unit/test_model_server.py`, `tests/unit/test_run_model_server.py` +- Modify: `pyproject.toml` (add `grpcio` dependency, `grpcio-tools` dev dependency, mypy override, ruff extend-exclude) + +**Interfaces:** +- Consumes: `proto/dispatch.proto` (Task 2). +- Produces: `dispatch.serving.model_server.{Responder, TokenEvent, StubResponder, ModelServerServicer, ServerBusyError, serve}`; `scripts.run_model_server.{build_responder, parse_args, main}`. Task 9 provides a second `Responder` implementation (`KernelResponder`) matching this same protocol; Task 8's Rust integration test spawns `scripts/run_model_server.py --responder stub` as a subprocess. + +- [ ] **Step 1: Add Python dependencies** + +In `pyproject.toml`'s `dependencies` list, add: + +```toml + "grpcio>=1.84.0", +``` + +In `[dependency-groups]` `dev`, add: + +```toml + "grpcio-tools>=1.84.0", +``` + +- [ ] **Step 2: Write the proto codegen script** + +Create `scripts/gen_proto.py`: + +```python +"""Regenerates the Python gRPC stubs from proto/dispatch.proto. grpc_tools +emits a bare `import dispatch_pb2` in the generated _grpc.py file, which +breaks once the file lives inside the dispatch.proto package (not on +sys.path directly) -- a known grpc_tools limitation, worked around here +by rewriting that one import to a relative one after generation. +""" + +from __future__ import annotations + +import re +import subprocess +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +PROTO_DIR = REPO_ROOT / "proto" +OUT_DIR = REPO_ROOT / "src" / "dispatch" / "proto" + + +def main() -> None: + OUT_DIR.mkdir(parents=True, exist_ok=True) + (OUT_DIR / "__init__.py").touch(exist_ok=True) + subprocess.run( + [ + sys.executable, + "-m", + "grpc_tools.protoc", + f"-I{PROTO_DIR}", + f"--python_out={OUT_DIR}", + f"--grpc_python_out={OUT_DIR}", + str(PROTO_DIR / "dispatch.proto"), + ], + check=True, + cwd=REPO_ROOT, + ) + grpc_file = OUT_DIR / "dispatch_pb2_grpc.py" + text = grpc_file.read_text() + patched = re.sub( + r"^import dispatch_pb2 as dispatch__pb2$", + "from . import dispatch_pb2 as dispatch__pb2", + text, + flags=re.MULTILINE, + ) + if patched == text: + raise RuntimeError( + "expected bare 'import dispatch_pb2' line not found -- " + "grpc_tools output format changed since this script was written" + ) + grpc_file.write_text(patched) + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 2: Run codegen and commit the generated files** + +```bash +uv sync +uv run python scripts/gen_proto.py +``` + +Expected: `src/dispatch/proto/dispatch_pb2.py` and `dispatch_pb2_grpc.py` now exist, and `dispatch_pb2_grpc.py`'s import line reads `from . import dispatch_pb2 as dispatch__pb2`. + +- [ ] **Step 3: Exempt the generated files from ruff and mypy strict** + +In `pyproject.toml`'s `[tool.ruff]` section, add: + +```toml +extend-exclude = ["src/dispatch/proto/dispatch_pb2.py", "src/dispatch/proto/dispatch_pb2_grpc.py"] +``` + +In `[tool.mypy]`, add a new override (following the existing `triton`/`deep_ep`/`vllm`/`sglang` pattern): + +```toml +# Generated protobuf/grpc code -- untyped by construction, same boundary +# as triton/deep_ep/vllm/sglang above. +[[tool.mypy.overrides]] +module = ["dispatch.proto.*"] +ignore_errors = true +``` + +- [ ] **Step 4: Write the failing tests** + +Create `tests/unit/test_model_server.py`: + +```python +"""No GPU, no torch -- StubResponder and the servicer wiring only.""" + +from __future__ import annotations + +import grpc +import pytest + +from dispatch.proto import dispatch_pb2, dispatch_pb2_grpc +from dispatch.serving.model_server import ServerBusyError, StubResponder, serve + + +def test_stub_responder_yields_requested_token_count_then_final() -> None: + responder = StubResponder(tokens=("a", "b", "c", "d"), delay_seconds=0.0) + + events = list(responder.generate("ignored", max_new_tokens=2)) + + assert [e.text for e in events] == ["a", "b", ""] + assert [e.is_final for e in events] == [False, False, True] + + +def test_serve_streams_stub_responses_over_real_grpc() -> None: + server, port = serve(StubResponder(tokens=("hi",), delay_seconds=0.0), port=0) + try: + channel = grpc.insecure_channel(f"localhost:{port}") + stub = dispatch_pb2_grpc.ModelServerStub(channel) + request = dispatch_pb2.GenerateRequest(request_id="r1", prompt="hello", max_new_tokens=1) + + responses = list(stub.Generate(request)) + + assert [r.text for r in responses] == ["hi", ""] + assert [r.is_final for r in responses] == [False, True] + assert all(r.request_id == "r1" for r in responses) + finally: + server.stop(grace=None) + + +def test_generate_aborts_with_resource_exhausted_on_server_busy_error() -> None: + class _BusyResponder: + def generate(self, prompt: str, max_new_tokens: int): # type: ignore[no-untyped-def] + raise ServerBusyError("only one stream at a time") + yield # pragma: no cover -- makes this a generator function + + server, port = serve(_BusyResponder(), port=0) # type: ignore[arg-type] + try: + channel = grpc.insecure_channel(f"localhost:{port}") + stub = dispatch_pb2_grpc.ModelServerStub(channel) + request = dispatch_pb2.GenerateRequest(request_id="r1", prompt="x", max_new_tokens=1) + + with pytest.raises(grpc.RpcError) as exc_info: + list(stub.Generate(request)) + + assert exc_info.value.code() == grpc.StatusCode.RESOURCE_EXHAUSTED + finally: + server.stop(grace=None) +``` + +- [ ] **Step 5: Run the tests to verify they fail** + +```bash +uv run pytest tests/unit/test_model_server.py -v +``` + +Expected: `ModuleNotFoundError: No module named 'dispatch.serving.model_server'`. + +- [ ] **Step 6: Implement `model_server.py`** + +Create `src/dispatch/serving/model_server.py`: + +```python +"""gRPC model server for Phase 7's router demo: streams token-by-token +generation over the shared proto/dispatch.proto contract. Two Responders +share one Servicer so the wire contract is identical whether responses +come from StubResponder (canned, no model, used by tests/dev/docker- +compose) or the real KernelResponder (scripts/gpu/phase7_kernel_responder.py, +GPU-only, used only for the real demo session) -- this module itself +never imports torch, so it stays importable and testable anywhere. +""" + +from __future__ import annotations + +import time +from collections.abc import Iterator +from concurrent import futures +from dataclasses import dataclass +from typing import Protocol + +import grpc + +from dispatch.proto import dispatch_pb2, dispatch_pb2_grpc + + +class ServerBusyError(RuntimeError): + """Raised by a Responder that can only stream one request at a time + when a second request arrives while the first is still in flight.""" + + +@dataclass(frozen=True) +class TokenEvent: + text: str + is_final: bool + t_emit: float + + +class Responder(Protocol): + def generate(self, prompt: str, max_new_tokens: int) -> Iterator[TokenEvent]: ... + + +class StubResponder: + """Canned, deterministic tokens with a fixed per-token delay. No + torch, no model weights -- used by unit tests, the cross-language + router integration test (Task 8), and docker-compose local + verification, all without a GPU. + """ + + def __init__( + self, + *, + tokens: tuple[str, ...] = ("The", " quick", " brown", " fox", " jumps"), + delay_seconds: float = 0.01, + ) -> None: + self._tokens = tokens + self._delay_seconds = delay_seconds + + def generate(self, prompt: str, max_new_tokens: int) -> Iterator[TokenEvent]: + del prompt # canned output does not depend on the prompt + for text in self._tokens[:max_new_tokens]: + time.sleep(self._delay_seconds) + yield TokenEvent(text=text, is_final=False, t_emit=time.perf_counter()) + yield TokenEvent(text="", is_final=True, t_emit=time.perf_counter()) + + +class ModelServerServicer(dispatch_pb2_grpc.ModelServerServicer): + def __init__(self, responder: Responder) -> None: + self._responder = responder + + def Generate( # noqa: N802 -- overrides a grpc_tools-generated method name + self, request: dispatch_pb2.GenerateRequest, context: grpc.ServicerContext + ) -> Iterator[dispatch_pb2.GenerateResponse]: + try: + for event in self._responder.generate(request.prompt, request.max_new_tokens): + yield dispatch_pb2.GenerateResponse( + request_id=request.request_id, + text=event.text, + is_final=event.is_final, + t_emit_unix=event.t_emit, + ) + except ServerBusyError as exc: + context.abort(grpc.StatusCode.RESOURCE_EXHAUSTED, str(exc)) + + +def serve(responder: Responder, *, port: int = 0) -> tuple[grpc.Server, int]: + server = grpc.server(futures.ThreadPoolExecutor(max_workers=4)) + dispatch_pb2_grpc.add_ModelServerServicer_to_server(ModelServerServicer(responder), server) + bound_port = server.add_insecure_port(f"[::]:{port}") + server.start() + return server, bound_port +``` + +- [ ] **Step 7: Run the tests to verify they pass** + +```bash +uv run pytest tests/unit/test_model_server.py -v +``` + +- [ ] **Step 8: Write the CLI, its failing test, then implement it** + +Create `tests/unit/test_run_model_server.py`: + +```python +"""CPU-only: proves --responder stub needs nothing but this repo's normal +dependencies. --responder kernel is exercised only on a real GPU +(Task 9's gpu-marked test), never here. +""" + +from __future__ import annotations + +from scripts.run_model_server import build_responder, parse_args + + +def test_default_responder_is_stub() -> None: + args = parse_args([]) + + assert args.responder == "stub" + assert args.port == 50051 + + +def test_build_responder_stub_needs_no_gpu_imports() -> None: + args = parse_args(["--responder", "stub"]) + + responder = build_responder(args) + + events = list(responder.generate("hi", max_new_tokens=1)) + assert events[-1].is_final is True +``` + +Run it (`uv run pytest tests/unit/test_run_model_server.py -v`) and confirm it fails with `ModuleNotFoundError: No module named 'scripts.run_model_server'`. + +Create `scripts/run_model_server.py`: + +```python +"""CLI entrypoint for Phase 7's model server. `--responder stub` (the +default) needs nothing but this repo's normal dependencies and is what +tests, the cross-language router integration test, and docker-compose +local verification all run. `--responder kernel` additionally imports +scripts/gpu/phase7_kernel_responder.py (torch, the real model, Phase 6's +naive kernel) -- lazily, only when selected, so this module stays +importable on a machine with neither GPU nor torch installed. +""" + +from __future__ import annotations + +import argparse + +from dispatch.serving.model_server import Responder, StubResponder, serve + + +def build_responder(args: argparse.Namespace) -> Responder: + if args.responder == "stub": + return StubResponder() + from scripts.gpu.phase7_kernel_responder import build_kernel_responder # noqa: PLC0415 + + return build_kernel_responder(moe_kernel=args.moe_kernel) + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--responder", choices=["stub", "kernel"], default="stub") + parser.add_argument("--port", type=int, default=50051) + parser.add_argument( + "--moe-kernel", + default="naive", + help="Only used with --responder kernel; passed to resolve_backend.", + ) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> None: + args = parse_args(argv) + responder = build_responder(args) + server, bound_port = serve(responder, port=args.port) + print(f"model server listening on port {bound_port}, responder={args.responder}") + server.wait_for_termination() + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 9: Run the tests, lint, and typecheck** + +```bash +uv run pytest tests/unit/test_run_model_server.py -v +uv run ruff check . && uv run ruff format --check . +uv run mypy src tests scripts +``` + +- [ ] **Step 10: Commit** + +```bash +git add pyproject.toml uv.lock scripts/gen_proto.py src/dispatch/proto \ + src/dispatch/serving/model_server.py scripts/run_model_server.py \ + tests/unit/test_model_server.py tests/unit/test_run_model_server.py +git commit -m "feat: add the Python model server skeleton (StubResponder, gRPC servicer, CLI)" +``` + +--- + +### Task 4: Router admission queue + +**Files:** +- Create: `router/src/queue.rs` +- Modify: `router/src/lib.rs` (add `pub mod queue;`) + +**Interfaces:** +- Consumes: nothing. +- Produces: `dispatch_router::queue::{AdmissionQueue, AdmissionTicket}` -- `AdmissionQueue::new(capacity: usize)`, `.depth() -> usize`, `async .admit() -> AdmissionTicket<'_>`. Task 7 consumes this directly. + +- [ ] **Step 1: Write the failing tests** + +Create `router/src/queue.rs`: + +```rust +use std::sync::atomic::{AtomicUsize, Ordering}; + +use tokio::sync::{Semaphore, SemaphorePermit}; + +pub struct AdmissionQueue { + semaphore: Semaphore, + waiting: AtomicUsize, +} + +pub struct AdmissionTicket<'a> { + _permit: SemaphorePermit<'a>, +} + +impl AdmissionQueue { + pub fn new(capacity: usize) -> Self { + Self { + semaphore: Semaphore::new(capacity), + waiting: AtomicUsize::new(0), + } + } + + pub fn depth(&self) -> usize { + self.waiting.load(Ordering::SeqCst) + } + + pub async fn admit(&self) -> AdmissionTicket<'_> { + self.waiting.fetch_add(1, Ordering::SeqCst); + let permit = self + .semaphore + .acquire() + .await + .expect("semaphore is never closed"); + self.waiting.fetch_sub(1, Ordering::SeqCst); + AdmissionTicket { _permit: permit } + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize as StdAtomicUsize, Ordering as StdOrdering}; + + use super::*; + + #[tokio::test] + async fn admit_serializes_single_capacity_access() { + let queue = Arc::new(AdmissionQueue::new(1)); + let concurrent = Arc::new(StdAtomicUsize::new(0)); + let max_concurrent = Arc::new(StdAtomicUsize::new(0)); + + let mut handles = Vec::new(); + for _ in 0..5 { + let queue = queue.clone(); + let concurrent = concurrent.clone(); + let max_concurrent = max_concurrent.clone(); + handles.push(tokio::spawn(async move { + let _ticket = queue.admit().await; + let now = concurrent.fetch_add(1, StdOrdering::SeqCst) + 1; + max_concurrent.fetch_max(now, StdOrdering::SeqCst); + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + concurrent.fetch_sub(1, StdOrdering::SeqCst); + })); + } + for handle in handles { + handle.await.expect("task did not panic"); + } + + assert_eq!(max_concurrent.load(StdOrdering::SeqCst), 1); + } + + #[tokio::test] + async fn depth_reflects_waiting_tasks() { + let queue = Arc::new(AdmissionQueue::new(1)); + let held = queue.admit().await; + + let queue2 = queue.clone(); + let waiter = tokio::spawn(async move { + let _ticket = queue2.admit().await; + }); + + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + assert_eq!(queue.depth(), 1); + + drop(held); + waiter.await.expect("task did not panic"); + } +} +``` + +- [ ] **Step 2: Add the module to `lib.rs`** + +In `router/src/lib.rs`, add: + +```rust +pub mod queue; +``` + +- [ ] **Step 3: Run the tests** + +```bash +cd router && cargo test queue:: +``` + +Expected: both tests pass (they were written alongside the implementation above rather than red-then-green, since the queue's behavior is simple enough to write once correctly -- but confirm by temporarily commenting out the `fetch_sub` in `admit` and re-running to see `depth_reflects_waiting_tasks` fail, then restore it). + +- [ ] **Step 4: Lint** + +```bash +cargo fmt --check +cargo clippy --all-targets -- -D warnings +``` + +- [ ] **Step 5: Commit** + +```bash +cd .. +git add router/src/queue.rs router/src/lib.rs +git commit -m "feat: add the router's single-flight admission queue" +``` + +--- + +### Task 5: Router Prometheus metrics + +**Files:** +- Create: `router/src/metrics.rs` +- Modify: `router/src/lib.rs` (add `pub mod metrics;`) + +**Interfaces:** +- Consumes: nothing. +- Produces: `dispatch_router::metrics::RouterMetrics` with public fields `requests_total: IntCounter`, `ttft_seconds: Histogram`, `inter_token_latency_seconds: Histogram`, `queue_depth: IntGauge`, and `.encode() -> String`. Tasks 6 and 7 consume this. + +- [ ] **Step 1: Write the metrics module with its test** + +Create `router/src/metrics.rs`: + +```rust +use prometheus::{Encoder, Histogram, HistogramOpts, IntCounter, IntGauge, Opts, Registry, TextEncoder}; + +pub struct RouterMetrics { + registry: Registry, + pub requests_total: IntCounter, + pub ttft_seconds: Histogram, + pub inter_token_latency_seconds: Histogram, + pub queue_depth: IntGauge, +} + +impl RouterMetrics { + pub fn new() -> Self { + let registry = Registry::new(); + + let requests_total = IntCounter::with_opts(Opts::new( + "dispatch_router_requests_total", + "Total requests admitted by the router", + )) + .expect("valid metric opts"); + registry + .register(Box::new(requests_total.clone())) + .expect("first registration of this metric"); + + let ttft_seconds = Histogram::with_opts(HistogramOpts::new( + "dispatch_router_ttft_seconds", + "Time to first token, seconds", + )) + .expect("valid metric opts"); + registry + .register(Box::new(ttft_seconds.clone())) + .expect("first registration of this metric"); + + let inter_token_latency_seconds = Histogram::with_opts(HistogramOpts::new( + "dispatch_router_inter_token_latency_seconds", + "Gap between consecutive tokens, seconds", + )) + .expect("valid metric opts"); + registry + .register(Box::new(inter_token_latency_seconds.clone())) + .expect("first registration of this metric"); + + let queue_depth = IntGauge::with_opts(Opts::new( + "dispatch_router_queue_depth", + "Requests currently waiting for the single-flight model server", + )) + .expect("valid metric opts"); + registry + .register(Box::new(queue_depth.clone())) + .expect("first registration of this metric"); + + Self { + registry, + requests_total, + ttft_seconds, + inter_token_latency_seconds, + queue_depth, + } + } + + pub fn encode(&self) -> String { + let metric_families = self.registry.gather(); + let mut buffer = Vec::new(); + TextEncoder::new() + .encode(&metric_families, &mut buffer) + .expect("prometheus text encoding never fails for valid metric families"); + String::from_utf8(buffer).expect("prometheus TextEncoder always emits valid utf-8") + } +} + +impl Default for RouterMetrics { + fn default() -> Self { + Self::new() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn encode_reports_recorded_values() { + let metrics = RouterMetrics::new(); + metrics.requests_total.inc(); + metrics.ttft_seconds.observe(0.25); + metrics.queue_depth.set(3); + + let text = metrics.encode(); + + assert!(text.contains("dispatch_router_requests_total 1")); + assert!(text.contains("dispatch_router_queue_depth 3")); + assert!(text.contains("dispatch_router_ttft_seconds_sum 0.25")); + } +} +``` + +- [ ] **Step 2: Add the module to `lib.rs`** + +In `router/src/lib.rs`, add: + +```rust +pub mod metrics; +``` + +- [ ] **Step 3: Run the test, then lint** + +```bash +cd router +cargo test metrics:: +cargo fmt --check +cargo clippy --all-targets -- -D warnings +``` + +- [ ] **Step 4: Commit** + +```bash +cd .. +git add router/src/metrics.rs router/src/lib.rs +git commit -m "feat: add the router's Prometheus metrics registry" +``` + +--- + +### Task 6: Router gRPC client and test-only mock server + +**Files:** +- Create: `router/src/client.rs`, `router/src/test_support.rs` +- Modify: `router/src/lib.rs` (add `pub mod client;` and `#[cfg(test)] pub mod test_support;`) + +**Interfaces:** +- Consumes: `dispatch_router::pb::dispatch_v1::*` (Task 2), `dispatch_router::metrics::RouterMetrics` (Task 5). +- Produces: `dispatch_router::client::{ModelServerClient, StreamedGeneration}` -- `ModelServerClient::connect(endpoint: String) -> Result`, `async .generate(request_id: &str, prompt: &str, max_new_tokens: u32, metrics: &RouterMetrics) -> Result`. `dispatch_router::test_support::spawn_mock_model_server(chunks: Vec<&'static str>) -> String` (test-only). Task 7 and Task 8 consume `ModelServerClient`; Task 7's tests consume `test_support`. + +- [ ] **Step 1: Write the test-only mock server helper** + +Create `router/src/test_support.rs`: + +```rust +use std::pin::Pin; + +use tokio::net::TcpListener; +use tokio::sync::mpsc; +use tokio_stream::wrappers::{ReceiverStream, TcpListenerStream}; +use tonic::{Request, Response, Status}; + +use crate::pb::dispatch_v1::model_server_server::{ModelServer, ModelServerServer}; +use crate::pb::dispatch_v1::{GenerateRequest, GenerateResponse}; + +struct MockModelServer { + chunks: Vec<&'static str>, +} + +#[tonic::async_trait] +impl ModelServer for MockModelServer { + type GenerateStream = + Pin> + Send>>; + + async fn generate( + &self, + request: Request, + ) -> Result, Status> { + let request_id = request.into_inner().request_id; + let chunks = self.chunks.clone(); + let (tx, rx) = mpsc::channel(8); + tokio::spawn(async move { + for text in chunks { + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + let _ = tx + .send(Ok(GenerateResponse { + request_id: request_id.clone(), + text: text.to_string(), + is_final: false, + t_emit_unix: 0.0, + })) + .await; + } + let _ = tx + .send(Ok(GenerateResponse { + request_id, + text: String::new(), + is_final: true, + t_emit_unix: 0.0, + })) + .await; + }); + Ok(Response::new(Box::pin(ReceiverStream::new(rx)))) + } +} + +/// Starts an in-process, real gRPC server implementing ModelServer that +/// streams `chunks` back as separate non-final tokens followed by one +/// final empty message, then returns its `http://host:port` endpoint. +/// Test-only (router/src/lib.rs gates this module behind `#[cfg(test)]`). +pub async fn spawn_mock_model_server(chunks: Vec<&'static str>) -> String { + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind ephemeral port"); + let addr = listener.local_addr().expect("local addr"); + tokio::spawn(async move { + tonic::transport::Server::builder() + .add_service(ModelServerServer::new(MockModelServer { chunks })) + .serve_with_incoming(TcpListenerStream::new(listener)) + .await + .expect("mock server exited unexpectedly"); + }); + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + format!("http://{addr}") +} +``` + +- [ ] **Step 2: Write the client with its test** + +Create `router/src/client.rs`: + +```rust +use std::time::{Duration, Instant}; + +use tonic::Request; +use tonic::transport::Channel; + +use crate::metrics::RouterMetrics; +use crate::pb::dispatch_v1::model_server_client::ModelServerClient as GrpcClient; +use crate::pb::dispatch_v1::GenerateRequest; + +pub struct ModelServerClient { + inner: GrpcClient, +} + +#[derive(Debug, Clone)] +pub struct StreamedGeneration { + pub full_text: String, + pub ttft: Duration, + pub inter_token_latencies: Vec, +} + +impl ModelServerClient { + pub async fn connect(endpoint: String) -> Result { + let inner = GrpcClient::connect(endpoint).await?; + Ok(Self { inner }) + } + + pub async fn generate( + &mut self, + request_id: &str, + prompt: &str, + max_new_tokens: u32, + metrics: &RouterMetrics, + ) -> Result { + let start = Instant::now(); + let mut stream = self + .inner + .generate(Request::new(GenerateRequest { + request_id: request_id.to_string(), + prompt: prompt.to_string(), + max_new_tokens, + })) + .await? + .into_inner(); + + let mut full_text = String::new(); + let mut ttft: Option = None; + let mut inter_token_latencies = Vec::new(); + let mut last_token_at = start; + + while let Some(chunk) = stream.message().await? { + if chunk.is_final { + break; + } + let now = Instant::now(); + match ttft { + None => { + let elapsed = now.duration_since(start); + metrics.ttft_seconds.observe(elapsed.as_secs_f64()); + ttft = Some(elapsed); + } + Some(_) => { + let gap = now.duration_since(last_token_at); + metrics.inter_token_latency_seconds.observe(gap.as_secs_f64()); + inter_token_latencies.push(gap); + } + } + last_token_at = now; + full_text.push_str(&chunk.text); + } + + Ok(StreamedGeneration { + full_text, + ttft: ttft.unwrap_or_default(), + inter_token_latencies, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_support::spawn_mock_model_server; + + #[tokio::test] + async fn generate_accumulates_text_and_records_latencies() { + let endpoint = spawn_mock_model_server(vec!["Hello", " world", "!"]).await; + let mut client = ModelServerClient::connect(endpoint).await.expect("connect"); + let metrics = RouterMetrics::new(); + + let result = client + .generate("req-1", "hi", 3, &metrics) + .await + .expect("generate succeeds"); + + assert_eq!(result.full_text, "Hello world!"); + assert_eq!(result.inter_token_latencies.len(), 2); + assert!(metrics.encode().contains("dispatch_router_ttft_seconds_count 1")); + } +} +``` + +- [ ] **Step 3: Add the modules to `lib.rs`** + +In `router/src/lib.rs`: + +```rust +pub mod client; +pub mod metrics; +pub mod pb; +pub mod queue; + +#[cfg(test)] +pub mod test_support; +``` + +- [ ] **Step 4: Run the test, then lint** + +```bash +cd router +cargo test client:: +cargo fmt --check +cargo clippy --all-targets -- -D warnings +``` + +- [ ] **Step 5: Commit** + +```bash +cd .. +git add router/src/client.rs router/src/test_support.rs router/src/lib.rs router/Cargo.toml router/Cargo.lock +git commit -m "feat: add the router's gRPC client and a test-only mock model server" +``` + +--- + +### Task 7: Router HTTP surface and wiring + +**Files:** +- Create: `router/src/app.rs` +- Modify: `router/src/main.rs` (full rewrite), `router/src/lib.rs` (add `pub mod app;`) + +**Interfaces:** +- Consumes: `dispatch_router::queue::AdmissionQueue`, `dispatch_router::client::ModelServerClient`, `dispatch_router::metrics::RouterMetrics` (Tasks 4-6). +- Produces: `dispatch_router::app::{AppState, GenerateHttpRequest, GenerateHttpResponse, build_app}`. Task 8's integration test and the router binary (`main.rs`) both consume `build_app`. + +- [ ] **Step 1: Write `app.rs` with its tests** + +Create `router/src/app.rs`: + +```rust +use std::sync::Arc; + +use axum::extract::State; +use axum::response::IntoResponse; +use axum::routing::{get, post}; +use axum::{Json, Router}; +use serde::{Deserialize, Serialize}; +use tokio::sync::Mutex; + +use crate::client::ModelServerClient; +use crate::metrics::RouterMetrics; +use crate::queue::AdmissionQueue; + +#[derive(Clone)] +pub struct AppState { + pub queue: Arc, + pub client: Arc>, + pub metrics: Arc, +} + +#[derive(Debug, Deserialize)] +pub struct GenerateHttpRequest { + pub prompt: String, + pub max_new_tokens: u32, +} + +#[derive(Debug, Serialize, Deserialize)] +pub struct GenerateHttpResponse { + pub text: String, + pub ttft_ms: f64, + pub inter_token_latencies_ms: Vec, +} + +pub fn build_app(state: AppState) -> Router { + Router::new() + .route("/generate", post(generate)) + .route("/healthz", get(healthz)) + .route("/readyz", get(readyz)) + .route("/metrics", get(metrics_handler)) + .with_state(state) +} + +async fn generate( + State(state): State, + Json(req): Json, +) -> Result, axum::http::StatusCode> { + let _ticket = state.queue.admit().await; + state.metrics.queue_depth.set(state.queue.depth() as i64); + state.metrics.requests_total.inc(); + + let mut client = state.client.lock().await; + let request_id = request_id(); + let result = client + .generate(&request_id, &req.prompt, req.max_new_tokens, &state.metrics) + .await + .map_err(|_| axum::http::StatusCode::BAD_GATEWAY)?; + + Ok(Json(GenerateHttpResponse { + text: result.full_text, + ttft_ms: result.ttft.as_secs_f64() * 1000.0, + inter_token_latencies_ms: result + .inter_token_latencies + .iter() + .map(|d| d.as_secs_f64() * 1000.0) + .collect(), + })) +} + +async fn healthz() -> &'static str { + "ok" +} + +/// Startup blocks on a successful connection to the model server (see +/// main.rs) -- by the time this process is serving HTTP at all, it is +/// also ready, so readyz and healthz report the same thing here. Kept +/// as a separate endpoint because K8s conventionally wires liveness and +/// readiness probes to different paths. +async fn readyz() -> &'static str { + "ok" +} + +async fn metrics_handler(State(state): State) -> impl IntoResponse { + state.metrics.encode() +} + +fn request_id() -> String { + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system clock is after the epoch") + .as_nanos(); + format!("{nanos:x}") +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use http_body_util::BodyExt; + use tokio::sync::Mutex; + use tower::ServiceExt; + + use super::*; + use crate::client::ModelServerClient; + use crate::metrics::RouterMetrics; + use crate::queue::AdmissionQueue; + use crate::test_support::spawn_mock_model_server; + + async fn test_state(chunks: Vec<&'static str>) -> AppState { + let endpoint = spawn_mock_model_server(chunks).await; + let client = ModelServerClient::connect(endpoint).await.expect("connect"); + AppState { + queue: Arc::new(AdmissionQueue::new(1)), + client: Arc::new(Mutex::new(client)), + metrics: Arc::new(RouterMetrics::new()), + } + } + + #[tokio::test] + async fn generate_endpoint_returns_full_text_and_timing() { + let app = build_app(test_state(vec!["Hello", " world"]).await); + + let request = axum::http::Request::builder() + .method("POST") + .uri("/generate") + .header("content-type", "application/json") + .body(axum::body::Body::from(r#"{"prompt":"hi","max_new_tokens":2}"#)) + .expect("build request"); + + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK); + + let body = response.into_body().collect().await.expect("read body").to_bytes(); + let parsed: GenerateHttpResponse = + serde_json::from_slice(&body).expect("valid json response"); + assert_eq!(parsed.text, "Hello world"); + assert!(parsed.ttft_ms >= 0.0); + } + + #[tokio::test] + async fn healthz_and_readyz_return_ok() { + for path in ["/healthz", "/readyz"] { + let app = build_app(test_state(vec!["ok"]).await); + let request = axum::http::Request::builder() + .uri(path) + .body(axum::body::Body::empty()) + .expect("build request"); + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK, "path: {path}"); + } + } + + #[tokio::test] + async fn metrics_endpoint_exposes_prometheus_text_format() { + let app = build_app(test_state(vec!["ok"]).await); + + let request = axum::http::Request::builder() + .uri("/metrics") + .body(axum::body::Body::empty()) + .expect("build request"); + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK); + + let body = response.into_body().collect().await.expect("read body").to_bytes(); + let text = String::from_utf8(body.to_vec()).expect("utf8"); + assert!(text.contains("dispatch_router_queue_depth")); + } +} +``` + +- [ ] **Step 2: Rewrite `main.rs`** + +Replace `router/src/main.rs` entirely: + +```rust +use std::env; +use std::sync::Arc; + +use tokio::sync::Mutex; + +use dispatch_router::app::{build_app, AppState}; +use dispatch_router::client::ModelServerClient; +use dispatch_router::metrics::RouterMetrics; +use dispatch_router::queue::AdmissionQueue; + +#[tokio::main] +async fn main() -> Result<(), Box> { + let model_server_endpoint = env::var("MODEL_SERVER_ENDPOINT") + .unwrap_or_else(|_| "http://127.0.0.1:50051".to_string()); + let listen_addr = env::var("ROUTER_LISTEN_ADDR").unwrap_or_else(|_| "0.0.0.0:8080".to_string()); + let queue_capacity: usize = env::var("ROUTER_QUEUE_CAPACITY") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(1); + + println!("connecting to model server at {model_server_endpoint}"); + let client = ModelServerClient::connect(model_server_endpoint).await?; + + let state = AppState { + queue: Arc::new(AdmissionQueue::new(queue_capacity)), + client: Arc::new(Mutex::new(client)), + metrics: Arc::new(RouterMetrics::new()), + }; + + let app = build_app(state); + let listener = tokio::net::TcpListener::bind(&listen_addr).await?; + println!("dispatch-router listening on {listen_addr}"); + axum::serve(listener, app).await?; + Ok(()) +} +``` + +- [ ] **Step 3: Add the module to `lib.rs`** + +In `router/src/lib.rs`, add `pub mod app;` (alongside the existing `client`, `metrics`, `pb`, `queue`). + +- [ ] **Step 4: Run the tests, then lint, then a manual smoke run** + +```bash +cd router +cargo test app:: +cargo fmt --check +cargo clippy --all-targets -- -D warnings +cargo build +``` + +- [ ] **Step 5: Commit** + +```bash +cd .. +git add router/src/app.rs router/src/main.rs router/src/lib.rs +git commit -m "feat: add the router's HTTP surface (generate/healthz/readyz/metrics)" +``` + +--- + +### Task 8: Cross-language router integration test + +**Files:** +- Create: `router/tests/cross_language_integration.rs` + +**Interfaces:** +- Consumes: `dispatch_router::app::{build_app, AppState}`, `dispatch_router::client::ModelServerClient`, `dispatch_router::metrics::RouterMetrics`, `dispatch_router::queue::AdmissionQueue` (Task 7); `scripts/run_model_server.py --responder stub` (Task 3). +- Produces: proof that the Rust client and Python grpcio server actually agree on the wire format -- the one thing Task 6/7's Rust-only mock tests can't show. + +- [ ] **Step 1: Write the integration test** + +Create `router/tests/cross_language_integration.rs`: + +```rust +use std::path::PathBuf; +use std::process::{Child, Command, Stdio}; +use std::sync::Arc; +use std::time::Duration; + +use tokio::sync::Mutex; + +use dispatch_router::app::{build_app, AppState}; +use dispatch_router::client::ModelServerClient; +use dispatch_router::metrics::RouterMetrics; +use dispatch_router::queue::AdmissionQueue; + +const TEST_PORT: u16 = 50099; + +struct StubServerGuard(Child); + +impl Drop for StubServerGuard { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn repo_root() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("router/ has a parent directory") + .to_path_buf() +} + +fn spawn_stub_server() -> StubServerGuard { + let child = Command::new("uv") + .args([ + "run", + "python", + "scripts/run_model_server.py", + "--responder", + "stub", + "--port", + &TEST_PORT.to_string(), + ]) + .current_dir(repo_root()) + .stdout(Stdio::piped()) + .stderr(Stdio::inherit()) + .spawn() + .expect("failed to spawn scripts/run_model_server.py -- is uv on PATH?"); + StubServerGuard(child) +} + +async fn wait_for_server_ready(endpoint: &str) { + for _ in 0..50 { + if ModelServerClient::connect(endpoint.to_string()).await.is_ok() { + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + panic!("stub model server never became reachable at {endpoint}"); +} + +#[tokio::test] +async fn router_streams_real_python_stub_server_responses_over_http() { + let _guard = spawn_stub_server(); + let endpoint = format!("http://127.0.0.1:{TEST_PORT}"); + wait_for_server_ready(&endpoint).await; + + let client = ModelServerClient::connect(endpoint) + .await + .expect("connect to the real Python stub server"); + let state = AppState { + queue: Arc::new(AdmissionQueue::new(1)), + client: Arc::new(Mutex::new(client)), + metrics: Arc::new(RouterMetrics::new()), + }; + let app = build_app(state); + + let request = axum::http::Request::builder() + .method("POST") + .uri("/generate") + .header("content-type", "application/json") + .body(axum::body::Body::from(r#"{"prompt":"hi","max_new_tokens":3}"#)) + .expect("build request"); + + use tower::ServiceExt; + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK); + + use http_body_util::BodyExt; + let body = response.into_body().collect().await.expect("read body").to_bytes(); + let parsed: dispatch_router::app::GenerateHttpResponse = + serde_json::from_slice(&body).expect("valid json response"); + + // StubResponder's default tokens, from model_server.py. + assert_eq!(parsed.text, "The quick brown"); +} +``` + +- [ ] **Step 2: Run it** + +```bash +cd router +cargo test --test cross_language_integration +``` + +Expected: the test spawns the real `scripts/run_model_server.py --responder stub` via `uv run`, connects a real tonic client to a real grpcio server, and gets back `"The quick brown"` (the first three of `StubResponder`'s default tokens) over the router's actual HTTP endpoint. If `uv` isn't resolvable from `cargo test`'s environment, the test's own panic message says so explicitly (`spawn_stub_server`'s `.expect(...)`) rather than failing silently. + +- [ ] **Step 3: Lint** + +```bash +cargo fmt --check +cargo clippy --all-targets -- -D warnings +``` + +- [ ] **Step 4: Commit** + +```bash +cd .. +git add router/tests/cross_language_integration.rs +git commit -m "test: prove the router and the real Python stub server agree on the wire" +``` + +--- + +### Task 9: Real KernelResponder (GPU-only) + +**Files:** +- Modify: `src/dispatch/serving/colocated.py` (add `snapshot_active_tokens`) +- Create: `scripts/gpu/phase7_kernel_responder.py` +- Test: `tests/unit/test_colocated.py` (add a case), `tests/unit/test_phase7_kernel_responder.py` (`gpu`-marked) + +**Interfaces:** +- Consumes: `dispatch.serving.model_server.{Responder, TokenEvent, ServerBusyError}` (Task 3), `dispatch.serving.disaggregated.{Request, PrefillFn, DecodeFn}`, `dispatch.serving.colocated.ColocatedWorker` (Phase 4), `dispatch.benchmark.harness.load_model`, `dispatch.kernels.backends.resolve_backend`, `dispatch.kernels.integration.{fix_rope_inv_freq, patch_moe_infer}` (Phases 0-1). +- Produces: `scripts.gpu.phase7_kernel_responder.{KernelResponder, build_kernel_responder, MODEL_NAME}` -- `build_kernel_responder(*, moe_kernel: str = "naive") -> KernelResponder`, consumed by `scripts/run_model_server.py`'s lazy import (Task 3). + +- [ ] **Step 1: Add `snapshot_active_tokens` to `ColocatedWorker`, with its test** + +In `tests/unit/test_colocated.py`, add: + +```python +def test_snapshot_active_tokens_reports_in_flight_generated_ids_so_far() -> None: + worker = ColocatedWorker(_fake_prefill_fn, _fake_decode_fn, batch_size=4) + worker.submit(Request("a", torch.tensor([[1, 2, 3]]), max_new_tokens=3)) + + worker.step() # prefilled and decoded once: 2 of 3 tokens so far, still active + + assert worker.snapshot_active_tokens() == {"a": [100, 101]} + + worker.step() # completes; no longer active + assert worker.snapshot_active_tokens() == {} +``` + +Run it (`uv run pytest tests/unit/test_colocated.py -v`) and confirm it fails with `AttributeError: 'ColocatedWorker' object has no attribute 'snapshot_active_tokens'`. + +In `src/dispatch/serving/colocated.py`, add this method to `ColocatedWorker` (after `step`): + +```python + def snapshot_active_tokens(self) -> dict[str, list[int]]: + """Every currently-active request's full generated-token-id list + so far, keyed by request_id -- returns copies, never internal + references. Additive to Phase 4's step()/submit(): step() + already returns *completed* RequestResults; this exposes the + still-in-flight ones too, which Phase 7's streaming model server + needs to emit each token as soon as it's produced rather than + only at completion. + """ + return {r.request_id: list(r.generated_token_ids) for r in self._active} +``` + +Run the test again and confirm it passes. + +- [ ] **Step 2: Write `phase7_kernel_responder.py`** + +Create `scripts/gpu/phase7_kernel_responder.py`: + +```python +"""GPU-only real inference for Phase 7's model server demo: wraps the +real deepseek-ai/deepseek-moe-16b-base model, Phase 6's naive bf16 +kernel, and Phase 4's ColocatedWorker into model_server.py's Responder +protocol. Imported lazily by scripts/run_model_server.py only when +--responder kernel is passed -- never imported by CPU tests or CI. + +Single-flight only: ColocatedWorker's continuous-batching machinery is +reused, but this responder enforces at most one in-flight stream at a +time (a lock, not a queue) -- Phase 7's design explicitly scopes out +concurrent multi-stream serving (design doc SS6, non-goals). +""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Iterator +from typing import Any + +import torch +from transformers import DynamicCache + +from dispatch.benchmark.harness import load_model +from dispatch.kernels.backends import resolve_backend +from dispatch.kernels.integration import fix_rope_inv_freq, patch_moe_infer +from dispatch.serving.colocated import ColocatedWorker +from dispatch.serving.disaggregated import DecodeFn, PrefillFn, Request +from dispatch.serving.model_server import ServerBusyError, TokenEvent + +MODEL_NAME = "deepseek-ai/deepseek-moe-16b-base" +DEMO_REQUEST_ID = "demo" + + +# Same pre-v5 DynamicCache shim Phase 3/4's own GPU scripts use -- +# DeepSeek's remote-code modeling file still calls the removed +# get_usable_length even against this repo's transformers>=5.17.0 floor. +def _get_usable_length( + self: DynamicCache, new_seq_length: int | None = None, layer_idx: int = 0 +) -> int: + return self.get_seq_length(layer_idx) + + +DynamicCache.get_usable_length = _get_usable_length # type: ignore[attr-defined] + + +def make_prefill_fn(model: torch.nn.Module, device: str) -> PrefillFn: + def prefill_fn(input_ids: torch.Tensor, attention_mask: torch.Tensor) -> Any: + with torch.no_grad(): + outputs = model( + input_ids=input_ids.to(device), + attention_mask=attention_mask.to(device), + use_cache=True, + ) + first_tokens = outputs.logits[:, -1, :].argmax(dim=-1) + cache = DynamicCache.from_legacy_cache(outputs.past_key_values) # type: ignore[attr-defined] + return first_tokens.cpu(), cache + + return prefill_fn + + +def make_decode_fn(model: torch.nn.Module, device: str) -> DecodeFn: + def decode_fn( + next_input_ids: torch.Tensor, + cache: DynamicCache, + attention_mask: torch.Tensor, + position_ids: torch.Tensor, + ) -> Any: + with torch.no_grad(): + outputs = model( + input_ids=next_input_ids.to(device), + past_key_values=cache.to_legacy_cache(), # type: ignore[attr-defined] + attention_mask=attention_mask.to(device), + position_ids=position_ids.to(device), + use_cache=True, + ) + next_tokens = outputs.logits[:, -1, :].argmax(dim=-1) + updated_cache = DynamicCache.from_legacy_cache(outputs.past_key_values) # type: ignore[attr-defined] + return next_tokens.cpu(), updated_cache + + return decode_fn + + +class KernelResponder: + def __init__(self, model: torch.nn.Module, tokenizer: Any, *, device: str = "cuda") -> None: + self.tokenizer = tokenizer + self._device = device + self._prefill_fn = make_prefill_fn(model, device) + self._decode_fn = make_decode_fn(model, device) + self._lock = threading.Lock() + + def generate(self, prompt: str, max_new_tokens: int) -> Iterator[TokenEvent]: + if not self._lock.acquire(blocking=False): + raise ServerBusyError("model server is already streaming one request") + try: + input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids + worker = ColocatedWorker(self._prefill_fn, self._decode_fn, batch_size=1) + worker.submit( + Request( + request_id=DEMO_REQUEST_ID, + prompt_ids=input_ids, + max_new_tokens=max_new_tokens, + eos_token_id=self.tokenizer.eos_token_id, + ) + ) + emitted = 0 + while True: + completed = worker.step() + active = worker.snapshot_active_tokens() + if DEMO_REQUEST_ID in active: + new_ids = active[DEMO_REQUEST_ID][emitted:] + for token_id in new_ids: + yield TokenEvent( + text=self.tokenizer.decode([token_id]), + is_final=False, + t_emit=time.perf_counter(), + ) + emitted = len(active[DEMO_REQUEST_ID]) + if completed: + tail = completed[0].generated_token_ids[emitted:] + for token_id in tail: + yield TokenEvent( + text=self.tokenizer.decode([token_id]), + is_final=False, + t_emit=time.perf_counter(), + ) + yield TokenEvent(text="", is_final=True, t_emit=time.perf_counter()) + return + finally: + self._lock.release() + + +def build_kernel_responder(*, moe_kernel: str = "naive") -> KernelResponder: + model, tokenizer = load_model( + MODEL_NAME, device="cuda", dtype=torch.bfloat16, trust_remote_code=True + ) + fix_rope_inv_freq(model) + patched = patch_moe_infer(model, resolve_backend(moe_kernel)) + if patched == 0: + raise RuntimeError(f"patch_moe_infer patched zero layers for backend {moe_kernel!r}") + return KernelResponder(model, tokenizer, device="cuda") +``` + +- [ ] **Step 3: Write the `gpu`-marked correctness test** + +Create `tests/unit/test_phase7_kernel_responder.py`: + +```python +"""Real end-to-end model-server correctness on a real GPU: KernelResponder's +streamed token sequence matches a same-session stock-model greedy +generation on the identical prompt, exactly -- Phase 4's own bar for its +correctness gate, applied here to the streaming responder. Excluded from +CI -- no GPU runner there. +""" + +from __future__ import annotations + +import pytest +import torch + +pytestmark = pytest.mark.gpu +pytest.importorskip("triton", reason="triton ships Linux wheels only") +if not torch.cuda.is_available(): + pytest.skip("needs a CUDA device", allow_module_level=True) + +from dispatch.benchmark.harness import load_model +from dispatch.kernels.integration import fix_rope_inv_freq +from scripts.gpu.phase7_kernel_responder import MODEL_NAME, build_kernel_responder + +PROMPT = "The quick brown fox jumps over the lazy dog." +MAX_NEW_TOKENS = 8 + + +def _stock_greedy_text(prompt: str, max_new_tokens: int) -> str: + model, tokenizer = load_model( + MODEL_NAME, device="cuda", dtype=torch.bfloat16, trust_remote_code=True + ) + fix_rope_inv_freq(model) + model.eval() + input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda") + generated: list[int] = [] + past = None + next_input = input_ids + with torch.no_grad(): + for _ in range(max_new_tokens): + outputs = model(input_ids=next_input, past_key_values=past, use_cache=True) + past = outputs.past_key_values + next_token = outputs.logits[:, -1, :].argmax(dim=-1, keepdim=True) + generated.append(int(next_token.item())) + next_input = next_token + return str(tokenizer.decode(generated)) + + +def test_kernel_responder_matches_same_session_stock_greedy_text() -> None: + expected_text = _stock_greedy_text(PROMPT, MAX_NEW_TOKENS) + + responder = build_kernel_responder(moe_kernel="naive") + events = list(responder.generate(PROMPT, MAX_NEW_TOKENS)) + generated_text = "".join(e.text for e in events if not e.is_final) + + assert generated_text == expected_text +``` + +- [ ] **Step 4: Lint and typecheck the CPU-reachable parts** + +```bash +uv run ruff check . && uv run ruff format --check . +uv run mypy src tests scripts +uv run pytest -m "not gpu" -v +``` + +This test does not run yet (it needs a GPU); Task 13 runs it for real on the rented pod. + +- [ ] **Step 5: Commit** + +```bash +git add src/dispatch/serving/colocated.py tests/unit/test_colocated.py \ + scripts/gpu/phase7_kernel_responder.py tests/unit/test_phase7_kernel_responder.py +git commit -m "feat: add the real KernelResponder wrapping Phase 4's worker and Phase 6's kernel" +``` + +--- + +### Task 10: Dockerfiles and local Compose verification + +**Files:** +- Create: `docker/router.Dockerfile`, `docker/model-server.Dockerfile`, `docker-compose.yml` + +**Interfaces:** +- Consumes: `router/` (Task 7), `scripts/run_model_server.py` (Task 3). +- Produces: two runnable container images and a Compose file, verified against `StubResponder` end-to-end; Task 11's `kind` rehearsal reuses both Dockerfiles. + +- [ ] **Step 1: Write the router Dockerfile** + +Create `docker/router.Dockerfile`: + +```dockerfile +FROM rust:1.98.1-slim-trixie AS build +WORKDIR /build +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* +COPY proto ./proto +COPY router/Cargo.toml router/Cargo.lock ./router/ +COPY router/build.rs ./router/build.rs +COPY router/src ./router/src +WORKDIR /build/router +RUN cargo build --release + +FROM debian:trixie-slim +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates \ + && rm -rf /var/lib/apt/lists/* +COPY --from=build /build/router/target/release/dispatch-router /usr/local/bin/dispatch-router +EXPOSE 8080 +ENTRYPOINT ["/usr/local/bin/dispatch-router"] +``` + +- [ ] **Step 2: Write the model server Dockerfile** + +Create `docker/model-server.Dockerfile`: + +```dockerfile +FROM nvidia/cuda:13.3.0-runtime-ubuntu24.04 +RUN apt-get update && apt-get install -y --no-install-recommends \ + python3.12 python3.12-venv curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* +RUN curl -LsSf https://astral.sh/uv/install.sh | sh +ENV PATH="/root/.local/bin:${PATH}" +WORKDIR /app +COPY pyproject.toml uv.lock ./ +COPY proto ./proto +COPY src ./src +COPY scripts ./scripts +RUN uv sync --frozen --no-dev +EXPOSE 50051 +ENTRYPOINT ["uv", "run", "python", "scripts/run_model_server.py"] +CMD ["--responder", "stub", "--port", "50051"] +``` + +`CMD` defaults to the stub responder -- local Compose verification (this task) and the `kind` rehearsal (Task 11) never need a GPU. Task 13's real session overrides `CMD` (`docker run ... --responder kernel --moe-kernel naive`) on the rented pod only. + +- [ ] **Step 3: Write `docker-compose.yml`** + +Create `docker-compose.yml` at the repo root: + +```yaml +services: + model-server: + build: + context: . + dockerfile: docker/model-server.Dockerfile + command: ["--responder", "stub", "--port", "50051"] + ports: + - "50051:50051" + + router: + build: + context: . + dockerfile: docker/router.Dockerfile + environment: + MODEL_SERVER_ENDPOINT: "http://model-server:50051" + ROUTER_LISTEN_ADDR: "0.0.0.0:8080" + ROUTER_QUEUE_CAPACITY: "1" + ports: + - "8080:8080" + depends_on: + - model-server +``` + +- [ ] **Step 4: Build and verify locally** + +```bash +docker compose up -d --build +sleep 3 +curl -s -X POST localhost:8080/generate \ + -H 'content-type: application/json' \ + -d '{"prompt":"hi","max_new_tokens":3}' +``` + +Expected: a JSON body with `"text":"The quick brown"` (StubResponder's default tokens) and non-negative `ttft_ms`. + +```bash +curl -s localhost:8080/healthz +curl -s localhost:8080/metrics | grep dispatch_router +docker compose down +``` + +- [ ] **Step 5: Commit** + +```bash +git add docker/router.Dockerfile docker/model-server.Dockerfile docker-compose.yml +git commit -m "feat: containerize the router and model server, verified via docker compose" +``` + +--- + +### Task 11: K8s manifests and scripted kind demo + +**Files:** +- Create: `k8s/namespace.yaml`, `k8s/router-deployment.yaml`, `k8s/model-server-external.yaml`, `k8s/prometheus.yaml`, `k8s/grafana.yaml`, `scripts/run_kind_demo.sh` + +**Interfaces:** +- Consumes: `docker/router.Dockerfile`, `docker/model-server.Dockerfile` (Task 10); Task 1's verified `host.docker.internal` (or its recorded alternative) finding. +- Produces: a scripted, repeatable `kind` bring-up that Task 13 reuses unmodified, pointing the same `ExternalName` Service at the real SSH tunnel instead of a local stub process. + +- [ ] **Step 1: Write the namespace and router manifests** + +Create `k8s/namespace.yaml`: + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: dispatch-demo +``` + +Create `k8s/router-deployment.yaml`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: router-config + namespace: dispatch-demo +data: + MODEL_SERVER_ENDPOINT: "http://model-server.dispatch-demo.svc.cluster.local:50051" + ROUTER_LISTEN_ADDR: "0.0.0.0:8080" + ROUTER_QUEUE_CAPACITY: "1" +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: router + namespace: dispatch-demo +spec: + replicas: 1 + selector: + matchLabels: + app: dispatch-router + template: + metadata: + labels: + app: dispatch-router + spec: + containers: + - name: router + image: dispatch-router:demo + imagePullPolicy: IfNotPresent + ports: + - containerPort: 8080 + envFrom: + - configMapRef: + name: router-config + livenessProbe: + httpGet: + path: /healthz + port: 8080 + initialDelaySeconds: 2 + readinessProbe: + httpGet: + path: /readyz + port: 8080 + initialDelaySeconds: 2 +--- +apiVersion: v1 +kind: Service +metadata: + name: router + namespace: dispatch-demo +spec: + selector: + app: dispatch-router + ports: + - name: http + port: 8080 + targetPort: 8080 +``` + +- [ ] **Step 2: Write the model-server `ExternalName` Service** + +Create `k8s/model-server-external.yaml` -- the hostname below is Task 1's verified finding (`host.docker.internal`, or the recorded alternative if Step 5 of Task 1 was needed): + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: model-server + namespace: dispatch-demo +spec: + type: ExternalName + externalName: host.docker.internal + ports: + - port: 50051 +``` + +Note: `ExternalName` is pure DNS (a CNAME) -- it does not remap ports, so the router still connects on port 50051 directly to whatever `host.docker.internal` resolves to. That real destination is a local stub process in this task's rehearsal, and the SSH-tunneled real model server in Task 13; nothing else in this manifest changes between them. + +- [ ] **Step 3: Write the Prometheus manifest** + +Create `k8s/prometheus.yaml`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + namespace: dispatch-demo +data: + prometheus.yml: | + global: + scrape_interval: 5s + scrape_configs: + - job_name: dispatch-router + static_configs: + - targets: ["router:8080"] + metrics_path: /metrics +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus + namespace: dispatch-demo +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + containers: + - name: prometheus + image: prom/prometheus:v3.13.3 + args: ["--config.file=/etc/prometheus/prometheus.yml"] + ports: + - containerPort: 9090 + volumeMounts: + - name: config + mountPath: /etc/prometheus + volumes: + - name: config + configMap: + name: prometheus-config +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus + namespace: dispatch-demo +spec: + selector: + app: prometheus + ports: + - port: 9090 + targetPort: 9090 +``` + +- [ ] **Step 4: Write the Grafana manifest with an imported dashboard** + +Create `k8s/grafana.yaml`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-datasource + namespace: dispatch-demo +data: + datasource.yaml: | + apiVersion: 1 + datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-dashboard-provider + namespace: dispatch-demo +data: + dashboards.yaml: | + apiVersion: 1 + providers: + - name: dispatch + type: file + options: + path: /var/lib/grafana/dashboards +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-dashboard-dispatch + namespace: dispatch-demo +data: + dispatch-router.json: | + { + "title": "Dispatch Router - Phase 7 Demo", + "uid": "dispatch-router-demo", + "schemaVersion": 39, + "refresh": "5s", + "time": { "from": "now-5m", "to": "now" }, + "panels": [ + { + "id": 1, + "title": "Requests total", + "type": "stat", + "gridPos": { "h": 6, "w": 6, "x": 0, "y": 0 }, + "targets": [{ "expr": "dispatch_router_requests_total", "refId": "A" }] + }, + { + "id": 2, + "title": "Queue depth", + "type": "timeseries", + "gridPos": { "h": 8, "w": 9, "x": 6, "y": 0 }, + "targets": [{ "expr": "dispatch_router_queue_depth", "refId": "A" }] + }, + { + "id": 3, + "title": "Mean time to first token (s)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 9, "x": 15, "y": 0 }, + "targets": [ + { + "expr": "dispatch_router_ttft_seconds_sum / dispatch_router_ttft_seconds_count", + "refId": "A" + } + ] + } + ] + } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: grafana + namespace: dispatch-demo +spec: + replicas: 1 + selector: + matchLabels: + app: grafana + template: + metadata: + labels: + app: grafana + spec: + containers: + - name: grafana + image: grafana/grafana:13.2.2 + ports: + - containerPort: 3000 + env: + # Anonymous admin access is fine here: this cluster is local- + # only, demoed once, and torn down (design doc SS6/SS7, + # "no standing public deployment") -- it avoids a login step + # in the automated screenshot capture (Task 13/14) for a + # cluster that never leaves the dev machine. + - name: GF_AUTH_ANONYMOUS_ENABLED + value: "true" + - name: GF_AUTH_ANONYMOUS_ORG_ROLE + value: "Admin" + volumeMounts: + - name: datasource + mountPath: /etc/grafana/provisioning/datasources + - name: dashboard-provider + mountPath: /etc/grafana/provisioning/dashboards + - name: dashboard-dispatch + mountPath: /var/lib/grafana/dashboards + volumes: + - name: datasource + configMap: + name: grafana-datasource + - name: dashboard-provider + configMap: + name: grafana-dashboard-provider + - name: dashboard-dispatch + configMap: + name: grafana-dashboard-dispatch +--- +apiVersion: v1 +kind: Service +metadata: + name: grafana + namespace: dispatch-demo +spec: + selector: + app: grafana + ports: + - port: 3000 + targetPort: 3000 +``` + +- [ ] **Step 5: Write the demo script** + +Create `scripts/run_kind_demo.sh`: + +```bash +#!/usr/bin/env bash +# Brings up the K8s side of Phase 7's demo: a kind cluster with the +# router, Prometheus, and Grafana. Assumes something is already +# listening on the dev machine's port 50051 for the model-server +# ExternalName Service to reach -- the stub responder for this task's +# rehearsal (`uv run python scripts/run_model_server.py --responder stub +# --port 50051`, in another terminal), or the real SSH tunnel for +# Task 13's paid session. This script's own job is K8s bring-up only. +set -euo pipefail + +CLUSTER_NAME="dispatch-demo" +NAMESPACE="dispatch-demo" + +echo "[1/6] Creating kind cluster ${CLUSTER_NAME} (if not already up)" +if ! kind get clusters | grep -qx "${CLUSTER_NAME}"; then + kind create cluster --name "${CLUSTER_NAME}" +fi +kubectl config use-context "kind-${CLUSTER_NAME}" + +echo "[2/6] Building and loading images into the kind cluster" +docker build -t dispatch-router:demo -f docker/router.Dockerfile . +docker build -t dispatch-model-server:demo -f docker/model-server.Dockerfile . +kind load docker-image dispatch-router:demo --name "${CLUSTER_NAME}" +kind load docker-image dispatch-model-server:demo --name "${CLUSTER_NAME}" + +echo "[3/6] Applying manifests" +kubectl apply -f k8s/namespace.yaml +kubectl apply -f k8s/router-deployment.yaml +kubectl apply -f k8s/model-server-external.yaml +kubectl apply -f k8s/prometheus.yaml +kubectl apply -f k8s/grafana.yaml + +echo "[4/6] Waiting for rollouts" +kubectl -n "${NAMESPACE}" rollout status deployment/router --timeout=120s +kubectl -n "${NAMESPACE}" rollout status deployment/prometheus --timeout=120s +kubectl -n "${NAMESPACE}" rollout status deployment/grafana --timeout=120s + +echo "[5/6] Port-forwarding router (8080) and Grafana (3000) to localhost" +kubectl -n "${NAMESPACE}" port-forward svc/router 8080:8080 & +ROUTER_PF_PID=$! +kubectl -n "${NAMESPACE}" port-forward svc/grafana 3000:3000 & +GRAFANA_PF_PID=$! +trap 'kill ${ROUTER_PF_PID} ${GRAFANA_PF_PID} 2>/dev/null || true' EXIT +sleep 3 + +echo "[6/6] Ready." +echo " Demo request: curl -s -X POST localhost:8080/generate -H 'content-type: application/json' -d '{\"prompt\":\"The quick brown fox\",\"max_new_tokens\":16}'" +echo " Grafana: http://localhost:3000 (anonymous admin)" +echo "Press Ctrl-C to stop the port-forwards. Tear down the cluster separately with:" +echo " kind delete cluster --name ${CLUSTER_NAME}" +wait +``` + +```bash +chmod +x scripts/run_kind_demo.sh +``` + +- [ ] **Step 6: Rehearse the full demo against the stub, no GPU** + +In one terminal: + +```bash +uv run python scripts/run_model_server.py --responder stub --port 50051 +``` + +In another terminal: + +```bash +./scripts/run_kind_demo.sh +``` + +In a third terminal, once the script prints "Ready.": + +```bash +curl -s -X POST localhost:8080/generate \ + -H 'content-type: application/json' \ + -d '{"prompt":"The quick brown fox","max_new_tokens":5}' +open http://localhost:3000 # confirm the Dispatch Router dashboard renders and updates +``` + +Expected: the curl call returns `"The quick brown fox jumps"` and Grafana's "Requests total" stat increments. Tear down: + +```bash +kind delete cluster --name dispatch-demo +``` + +(Ctrl-C the stub model server and the demo script's port-forwards too.) + +- [ ] **Step 7: Commit** + +```bash +git add k8s/ scripts/run_kind_demo.sh +git commit -m "feat: add K8s manifests and a scripted kind demo, rehearsed against the stub" +``` + +--- + +### Task 12: `make check` goes Rust-aware + +**Files:** +- Modify: `Makefile` + +**Interfaces:** +- Consumes: `router/` (Tasks 2-8). +- Produces: `make router-lint`, `make router-test`, folded into `make check`/`make check-fast` so "green before push" stays one command covering both languages. + +- [ ] **Step 1: Add Rust targets** + +In `Makefile`, add (near the existing `lint`/`test` targets): + +```makefile +router-lint: + cd router && cargo fmt --check && cargo clippy --all-targets -- -D warnings + +router-test: + cd router && cargo test +``` + +- [ ] **Step 2: Fold them into `check` and `check-fast`** + +Replace the existing `check` target: + +```makefile +# The full gate. Runs before every push -- CI runs the same steps. +check: + @echo "[1/5] lint" + @$(MAKE) lint + @echo "[2/5] typecheck" + @$(MAKE) typecheck + @echo "[3/5] test" + @$(MAKE) test + @echo "[4/5] router-lint" + @$(MAKE) router-lint + @echo "[5/5] router-test" + @$(MAKE) router-test +``` + +Replace the existing `check-fast` target the same way, substituting `test-fast` for `test` at step 3. + +- [ ] **Step 3: Run it** + +```bash +make check +``` + +Expected: all five steps pass. + +- [ ] **Step 4: Commit** + +```bash +git add Makefile +git commit -m "build: make check runs the Rust router's fmt/clippy/test gate too" +``` + +--- + +### Task 13: Real GPU session -- PAID, needs explicit go-ahead + +**Files:** +- Create (pod-local, never committed): none beyond what Task 9 already wrote. +- Produces: raw evidence under `docs/findings/phase-7/` (Grafana screenshot, metrics export, cost record) that Task 14 writes up. + +**Interfaces:** +- Consumes: everything from Tasks 1-12, on a clean checkout of `phase-7-productionization`. +- Produces: `POD_ID`, `POD_SSH`, `RATE` (the live $/hr) for Task 14's cost writeup; the Grafana screenshot and the manually-captured screen recording, saved locally. + +- [ ] **Step 1: Confirm the live price and get the go-ahead** + +Query the live GPU price and availability (RunPod MCP `list-gpu-types` / `get-capacity`, or the console) for an L40-class card, RunPod Secure or Community Cloud per current stock. Tell the user: the GPU id, the hourly rate, the $5 cap, and that this session is expected to take well under an hour. **Do not create the pod until the user says yes.** Record the rate as `RATE`. + +- [ ] **Step 2: Create the pod** + +```bash +uv run python -m scripts.gpu.provision create --name dispatch-phase-7 \ + --gpu-type "" --image "" \ + --cloud SECURE --disk-gb 60 +uv run python -m scripts.gpu.provision wait --pod-id +``` + +Note `POD_ID` and `POD_SSH`. Start a wall-clock note (`date -u`). + +- [ ] **Step 3: Ship the repo and build the environment** + +Follow the established pattern from every prior phase's runbook (clone/ship the repo onto the pod, `uv sync --all-extras --dev`, download the model into a persistent mount, `HF_HOME` set accordingly). Confirm live which of the previously-documented `transformers` remote-code breaks (Phase 0/1/3/4/5a/5b) still reproduce at the pod's installed `transformers` version, and apply the same fixes if so. + +- [ ] **Step 4: Re-verify correctness on this GPU** + +```bash +.venv/bin/python -m pytest -m gpu tests/unit/test_phase7_kernel_responder.py -v +``` + +Expected: `test_kernel_responder_matches_same_session_stock_greedy_text` passes. If it doesn't, stop and debug before continuing -- correctness before the demo, same rule as every prior phase. + +- [ ] **Step 5: Start the real model server** + +```bash +.venv/bin/python scripts/run_model_server.py --responder kernel --moe-kernel naive --port 50051 & +``` + +- [ ] **Step 6: Open the SSH tunnel from the dev machine** + +In a terminal on the dev machine (not the pod): + +```bash +ssh -N -L 50051:localhost:50051 +``` + +- [ ] **Step 7: Bring up the K8s demo against the real tunnel** + +```bash +./scripts/run_kind_demo.sh +``` + +The `ExternalName` Service (`k8s/model-server-external.yaml`) is unchanged from Task 11's rehearsal -- it still points at `host.docker.internal`, which now resolves through the SSH tunnel to the real GPU-backed model server instead of the local stub. + +- [ ] **Step 8: Start the manual screen recording** + +Before firing the demo request, start an OS-level screen recording covering both the terminal (for the `curl` command and its output) and the browser window (for Grafana). On macOS: Cmd+Shift+5, select "Record Selected Portion" or the full screen, start recording. Keep it short -- 15-30 seconds is enough to show the request, the response, and the dashboard updating. + +- [ ] **Step 9: Fire the real demo request** + +```bash +curl -s -X POST localhost:8080/generate \ + -H 'content-type: application/json' \ + -d '{"prompt":"The quick brown fox jumps over the lazy dog.","max_new_tokens":16}' +``` + +Expected: real generated text (not the stub's canned tokens) and real `ttft_ms`/`inter_token_latencies_ms` values from actual GPU inference. + +- [ ] **Step 10: Capture the Grafana screenshot via browser automation** + +Wait a few seconds for Prometheus to scrape (5s interval), then use browser automation (Playwright or Chrome DevTools MCP) to navigate to `http://localhost:3000/d/dispatch-router-demo`, wait for the panels to render with non-empty data, and save a screenshot to `docs/findings/phase-7/2026-09-20-phase-7-grafana-dashboard.png`. + +- [ ] **Step 11: Stop the screen recording and save it** + +Stop the OS-level recording from Step 8. Save it as `docs/findings/phase-7/2026-09-20-phase-7-demo-recording.mov` (or convert to a short `.gif` if the file is large -- keep it well under a few MB for the README embed in Task 14). + +- [ ] **Step 12: Export the raw metrics text and record cost** + +```bash +curl -s localhost:8080/metrics > /tmp/phase7-metrics.txt +``` + +Copy `/tmp/phase7-metrics.txt` to `docs/findings/phase-7/2026-09-20-phase-7-metrics.txt` on the dev machine. Query RunPod's billing API for this pod's actual measured cost (Phase 6's established method, not rate x duration). + +- [ ] **Step 13: Tear down** + +```bash +kind delete cluster --name dispatch-demo +``` + +Kill the SSH tunnel (Step 6) and the model server (Step 5). Stop and terminate the pod: + +```bash +uv run python -m scripts.gpu.provision stop --pod-id +uv run python -m scripts.gpu.provision terminate --pod-id +``` + +Verify termination independently (pod lookup returns not-found or `TERMINATED`), per this repo's standing convention. + +--- + +### Task 14: Findings, docs, and the PR + +**Files:** +- Create: `docs/findings/phase-7/2026-09-20-phase-7-productionization-run.md`, `docs/runbooks/phase-7-productionization.md` +- Modify: `docs/STATUS.md`, `README.md`, `CLAUDE.md` + +**Interfaces:** +- Consumes: Task 13's evidence (screenshot, recording, metrics export, cost record). +- Produces: the phase's final write-up and the PR. + +- [ ] **Step 1: Write the findings doc** + +Create `docs/findings/phase-7/2026-09-20-phase-7-productionization-run.md` covering: what was built, the real demo request and its measured TTFT/inter-token-latency values, the Grafana screenshot (embedded), a link to the screen recording, the measured cost against the $5 cap, and any deviations from the plan discovered while executing Tasks 1-13 (the `host.docker.internal` finding from Task 1, any Rust API-surface adjustments from Task 2's Step 5 note, anything else) -- following this repo's established findings-doc style (see `docs/findings/phase-6/` for the pattern). + +- [ ] **Step 2: Write the runbook** + +Create `docs/runbooks/phase-7-productionization.md` from what actually worked in Task 13's session (not a re-statement of the plan -- the runbook records the real commands, the real pod id, the real timings), following the pattern of `docs/runbooks/phase-6-final-benchmark.md`. + +- [ ] **Step 3: Update `docs/STATUS.md`** + +Add a "Phase 7 progress" completion entry (extending the section Task 1 started): all tasks done, `make check` green throughout (both gates), the measured cost, and a note that this closes the system design's phase table (SS7) entirely -- Phase 0 through 7 now all complete. + +- [ ] **Step 4: Refresh README and CLAUDE.md** + +Update `README.md` with Phase 7's summary and embed the screen recording (or a GIF converted from it). Update `CLAUDE.md`'s "Current status" section with a Phase 7 entry matching the style of Phases 0-6's entries there, and note explicitly that this completes the system design's 8-phase plan (SS7). + +- [ ] **Step 5: Final `make check` and commit** + +```bash +make check +git add docs/findings/phase-7 docs/runbooks/phase-7-productionization.md docs/STATUS.md README.md CLAUDE.md +git commit -m "docs: record Phase 7 productionization run, close out the system design's phase table" +``` + +- [ ] **Step 6: Open the PR** + +Ask the user before pushing or opening the PR, per this repo's standing convention. On approval: + +```bash +git push -u origin phase-7-productionization +gh pr create --title "Phase 7: productionization (Rust router, Docker, K8s, observability)" --body "$(cat <<'EOF' +## Summary +- Rust router (tonic gRPC client, axum HTTP, Prometheus metrics) in front of a Python model server, per ADR-0003's TGI-shaped split +- Real deepseek-ai/deepseek-moe-16b-base inference (Phase 6's naive bf16 kernel, Phase 4's ColocatedWorker) demoed once through a local kind cluster, SSH-tunneled to a rented GPU +- Grafana dashboard showing real TTFT/inter-token-latency/queue-depth from the one real demo request +- Closes the system design's 8-phase plan (SS7) -- Phase 0 through 7 all complete + +## Test plan +- [ ] `make check` green (Rust and Python gates) +- [ ] `gpu`-marked correctness test passed on the real rented GPU (Task 13, Step 4) +- [ ] Real demo request produced real generated text and real metrics (Task 13, Step 9) +- [ ] Grafana screenshot and screen recording captured as evidence +EOF +)" +``` + diff --git a/docs/runbooks/phase-7-productionization.md b/docs/runbooks/phase-7-productionization.md new file mode 100644 index 0000000..2157f45 --- /dev/null +++ b/docs/runbooks/phase-7-productionization.md @@ -0,0 +1,153 @@ +# Runbook: Phase 7 productionization (rented GPU + local kind demo) + +One combined session -- real model server, SSH tunnel, `kind` demo, +Grafana screenshot, screen recording -- per +`docs/plans/2026-09-20-phase-7-productionization-plan.md` Task 13. Budget +cap: $5, a ceiling not a target. Actual cost: **$6.8563** (RunPod's +billing API), exceeded and disclosed mid-session -- see the findings doc +for why. This runbook records what actually worked on 2026-09-21, not the +plan's first-guess commands. + +1. **L40 sold out during pod creation; A40 was the live substitute.** + Quoted at $0.82/hr at session start via `list-gpu-types`/`get-capacity`; + by the time `create-pod` ran, stock was gone. A40 ($0.49/hr, 46068MiB) + was in stock and is what actually ran this session -- check live stock + immediately before creating, not just at the start of the + conversation. + +2. **Direct SSH, same pattern as Phase 6's runbook**: + + ssh -o ConnectTimeout=15 -o ServerAliveInterval=5 -o ServerAliveCountMax=3 \ + -i ~/.ssh/dispatch_runpod_ed25519 -p 22094 root@63.141.33.106 '' + + The `ServerAliveInterval`/`ServerAliveCountMax` pair matters here more + than in prior phases -- this session's SSH connection dropped + mid-command at least twice (`Timeout, server ... not responding`, + `Connection reset`) with no other symptom. Retrying the exact same + command after a fresh connectivity check (`ssh ... echo PING_OK`) + always worked; there was no underlying pod problem. + +3. **Start the model server as a module, not a script**: + + tmux new-session -d -s modelserver -x 200 -y 50 \ + 'export HF_HOME=/workspace/hf_cache; .venv/bin/python -m scripts.run_model_server --responder kernel --port 50051 > /workspace/modelserver.log 2>&1' + + `python scripts/run_model_server.py` puts `scripts/` itself on + `sys.path[0]`, not the repo root, so the lazy + `from scripts.gpu.phase7_kernel_responder import ...` import inside + `build_responder()` raises `ModuleNotFoundError: No module named + 'scripts'`. `-m scripts.run_model_server` (run from + `/workspace/dispatch`) puts the repo root on `sys.path[0]` instead and + resolves cleanly. Same fix needed for the correctness-test invocation + if using a wrapper script rather than pytest directly (pytest itself + is unaffected -- it inserts the repo root on its own). + +4. **Same `transformers` remote-code shims as every prior phase's + runbook**, applied via a pod-local wrapper (never committed) that + monkeypatches before importing `scripts.run_model_server.main`: + + import transformers.utils.import_utils as _iu + _iu.is_torch_fx_available = lambda: False + + from transformers.cache_utils import DynamicCache + DynamicCache.get_usable_length = lambda self, *a, **kw: self.get_seq_length(0) + + @classmethod + def _from_legacy_cache(cls, past_key_values=None): + if past_key_values is None: + return cls() + if isinstance(past_key_values, cls): + return past_key_values + raise NotImplementedError("pod-local shim: only None/DynamicCache supported") + DynamicCache.from_legacy_cache = _from_legacy_cache + DynamicCache.to_legacy_cache = lambda self: self + + All four still reproduced at this pod's installed `transformers` + version, exactly as documented in Phase 5b's runbook. + +5. **If the gpu-marked correctness test fails identically after a fix + ships, suspect stale bytecode before suspecting the fix.** + `/workspace` on this pod is `mfs#ca-mtl-1.runpod.net`, a FUSE network + filesystem; a checksum-verified-identical file still reproduced the + exact pre-fix failure on a second run, and took *longer* (1422s vs. + 1247s) -- consistent with Python recompiling and re-caching bytecode + under filesystem-level contention rather than trusting a possibly + stale `.pyc`. Fix: + + find /workspace/dispatch -iname '__pycache__' -exec rm -rf {} + + .venv/bin/python -B _patch_and_test.py -m gpu tests/unit/test_phase7_kernel_responder.py -v + + `-B` disables bytecode caching outright -- slower, but removes the + variable entirely rather than hoping the manual clear was enough. + +6. **If real generated text comes back with no spaces between words + (or literal `Ġ`/`Ċ` characters), check the tokenizer's own wiring + before touching the serving code.** This is not the same bug as + per-token-vs-whole-sequence decode (which *was* a real, separate, + already-fixed issue in `KernelResponder.generate()` before this + session). Diagnose in order: + + python -c " + from transformers import AutoTokenizer + tok = AutoTokenizer.from_pretrained('deepseek-ai/deepseek-moe-16b-base', trust_remote_code=True) + print(tok.tokenize('hello world')) # word boundaries lost already? + print(tok.backend_tokenizer.pre_tokenizer) # Metaspace (expects SentencePiece '▁')? + import json, glob + vocab = json.load(open(glob.glob('/workspace/hf_cache/hub/models--deepseek-ai--deepseek-moe-16b-base/snapshots/*/tokenizer.json')[0]))['model']['vocab'] + print(sum(1 for k in vocab if '▁' in k), sum(1 for k in vocab if 'Ġ' in k)) # which marker does the vocab actually use? + " + + If the vocab is majority `Ġ`-marked (byte-level BPE) but the + pre-tokenizer/decoder expect `▁` (SentencePiece), rewire both: + + from tokenizers import pre_tokenizers, decoders + tok.backend_tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=True) + tok.backend_tokenizer.decoder = decoders.ByteLevel() + + Verify with a round trip (`tok(tok.decode(ids))["input_ids"] == ids`) + before trusting it against the real model. This fix is committed + (`scripts/gpu/phase7_kernel_responder.py`'s `fix_tokenizer_byte_level`, + called once in `build_kernel_responder`) since it corrects the + tokenizer's own shipped configuration, not a version-specific + environment mismatch -- unlike the `transformers` shims in step 4, + there's no reason to re-derive it on the next GPU session for this + model. + +7. **SSH tunnel from the dev machine, `kind` demo unmodified from the + stub rehearsal**: + + ssh -f -N -i ~/.ssh/dispatch_runpod_ed25519 -p 22094 -L 50051:localhost:50051 root@ + ./scripts/run_kind_demo.sh + + `k8s/model-server-external.yaml`'s `ExternalName` Service needed zero + changes -- `host.docker.internal` resolves through the tunnel exactly + as it resolved to the local stub in Task 11's rehearsal. + +8. **Fire a few warm-up requests before capturing evidence.** The Grafana + dashboard's panels are far more legible with 5-10 data points than + with 1; fired several varied prompts before the screenshot and the + on-camera recording, all counted honestly in the final + `dispatch_router_requests_total`. + +9. **PII lives in unexpected corners of a screen recording, not just the + terminal prompt.** A `Cmd+Tab` window-switch mid-recording produced a + warped/rotated transition frame (a static redaction box can't track + it) and, moments later, an app-switcher tooltip that briefly showed a + VS Code workspace name containing the same username the terminal + prompt already leaked. Frame-by-frame review (`ffmpeg -vf fps=N` at + decreasing intervals to bound each transient element's exact start/end + time) found both; a short solid-black cut over the transition window + was simpler and more reliable than trying to track warped geometry. + +10. **Pull evidence before tearing down anything.** Order that avoided + re-provisioning: copy pod-side logs off first (`ssh ... cat + /workspace/*.log > local-file`), export the router's `/metrics` via a + fresh `kubectl port-forward` (needed *before* `kind delete cluster`, + not after), *then* delete the `kind` cluster, *then* kill the SSH + tunnel and the model-server `tmux` session, *then* terminate the pod + via the RunPod MCP `delete-pod`, and verify with `get-pod` returning + `404`. + +11. **Real cost, from RunPod's billing API, not rate × duration**: query + `list-pod-billing` with the pod id after termination, not before -- + the last partial hour only settles once the pod actually stops. diff --git a/k8s/grafana.yaml b/k8s/grafana.yaml new file mode 100644 index 0000000..a5249ce --- /dev/null +++ b/k8s/grafana.yaml @@ -0,0 +1,131 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-datasource + namespace: dispatch-demo +data: + datasource.yaml: | + apiVersion: 1 + datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-dashboard-provider + namespace: dispatch-demo +data: + dashboards.yaml: | + apiVersion: 1 + providers: + - name: dispatch + type: file + options: + path: /var/lib/grafana/dashboards +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: grafana-dashboard-dispatch + namespace: dispatch-demo +data: + dispatch-router.json: | + { + "title": "Dispatch Router - Phase 7 Demo", + "uid": "dispatch-router-demo", + "schemaVersion": 39, + "refresh": "5s", + "time": { "from": "now-5m", "to": "now" }, + "panels": [ + { + "id": 1, + "title": "Requests total", + "type": "stat", + "gridPos": { "h": 6, "w": 6, "x": 0, "y": 0 }, + "targets": [{ "expr": "dispatch_router_requests_total", "refId": "A" }] + }, + { + "id": 2, + "title": "Queue depth", + "type": "timeseries", + "gridPos": { "h": 8, "w": 9, "x": 6, "y": 0 }, + "targets": [{ "expr": "dispatch_router_queue_depth", "refId": "A" }] + }, + { + "id": 3, + "title": "Mean time to first token (s)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 9, "x": 15, "y": 0 }, + "targets": [ + { + "expr": "dispatch_router_ttft_seconds_sum / dispatch_router_ttft_seconds_count", + "refId": "A" + } + ] + } + ] + } +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: grafana + namespace: dispatch-demo +spec: + replicas: 1 + selector: + matchLabels: + app: grafana + template: + metadata: + labels: + app: grafana + spec: + containers: + - name: grafana + image: grafana/grafana:13.2.2 + ports: + - containerPort: 3000 + env: + # Anonymous admin access is fine here: this cluster is local- + # only, demoed once, and torn down (design doc SS6/SS7, + # "no standing public deployment") -- it avoids a login step + # in the automated screenshot capture (Task 13/14) for a + # cluster that never leaves the dev machine. + - name: GF_AUTH_ANONYMOUS_ENABLED + value: "true" + - name: GF_AUTH_ANONYMOUS_ORG_ROLE + value: "Admin" + volumeMounts: + - name: datasource + mountPath: /etc/grafana/provisioning/datasources + - name: dashboard-provider + mountPath: /etc/grafana/provisioning/dashboards + - name: dashboard-dispatch + mountPath: /var/lib/grafana/dashboards + volumes: + - name: datasource + configMap: + name: grafana-datasource + - name: dashboard-provider + configMap: + name: grafana-dashboard-provider + - name: dashboard-dispatch + configMap: + name: grafana-dashboard-dispatch +--- +apiVersion: v1 +kind: Service +metadata: + name: grafana + namespace: dispatch-demo +spec: + selector: + app: grafana + ports: + - port: 3000 + targetPort: 3000 diff --git a/k8s/model-server-external.yaml b/k8s/model-server-external.yaml new file mode 100644 index 0000000..5bc4f69 --- /dev/null +++ b/k8s/model-server-external.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Service +metadata: + name: model-server + namespace: dispatch-demo +spec: + type: ExternalName + externalName: host.docker.internal + ports: + - port: 50051 diff --git a/k8s/namespace.yaml b/k8s/namespace.yaml new file mode 100644 index 0000000..fbea892 --- /dev/null +++ b/k8s/namespace.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: dispatch-demo diff --git a/k8s/prometheus.yaml b/k8s/prometheus.yaml new file mode 100644 index 0000000..caf2e89 --- /dev/null +++ b/k8s/prometheus.yaml @@ -0,0 +1,55 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + namespace: dispatch-demo +data: + prometheus.yml: | + global: + scrape_interval: 5s + scrape_configs: + - job_name: dispatch-router + static_configs: + - targets: ["router:8080"] + metrics_path: /metrics +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus + namespace: dispatch-demo +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + containers: + - name: prometheus + image: prom/prometheus:v3.13.3 + args: ["--config.file=/etc/prometheus/prometheus.yml"] + ports: + - containerPort: 9090 + volumeMounts: + - name: config + mountPath: /etc/prometheus + volumes: + - name: config + configMap: + name: prometheus-config +--- +apiVersion: v1 +kind: Service +metadata: + name: prometheus + namespace: dispatch-demo +spec: + selector: + app: prometheus + ports: + - port: 9090 + targetPort: 9090 diff --git a/k8s/router-deployment.yaml b/k8s/router-deployment.yaml new file mode 100644 index 0000000..805ffbb --- /dev/null +++ b/k8s/router-deployment.yaml @@ -0,0 +1,64 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: router-config + namespace: dispatch-demo +data: + MODEL_SERVER_ENDPOINT: "http://model-server.dispatch-demo.svc.cluster.local:50051" + ROUTER_LISTEN_ADDR: "0.0.0.0:8080" + ROUTER_QUEUE_CAPACITY: "1" +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: router + namespace: dispatch-demo +spec: + replicas: 1 + selector: + matchLabels: + app: dispatch-router + template: + metadata: + labels: + app: dispatch-router + spec: + containers: + - name: router + image: dispatch-router:demo + imagePullPolicy: IfNotPresent + ports: + - containerPort: 8080 + envFrom: + - configMapRef: + name: router-config + # main.rs retries connecting to the model server for up to 30s + # (30 attempts x 1s) before giving up -- these probes must + # tolerate that whole window, not just a quick local startup. + livenessProbe: + httpGet: + path: /healthz + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 8 + readinessProbe: + httpGet: + path: /readyz + port: 8080 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 8 +--- +apiVersion: v1 +kind: Service +metadata: + name: router + namespace: dispatch-demo +spec: + selector: + app: dispatch-router + ports: + - name: http + port: 8080 + targetPort: 8080 diff --git a/proto/dispatch.proto b/proto/dispatch.proto new file mode 100644 index 0000000..a829ef9 --- /dev/null +++ b/proto/dispatch.proto @@ -0,0 +1,28 @@ +syntax = "proto3"; + +package dispatch.v1; + +// The gRPC contract between the Rust router and the Python model server +// (docs/design/2026-09-20-phase-7-productionization.md SS3-4). Compiled +// by both sides -- tonic-build/prost-build (Rust, router/build.rs) and +// grpcio-tools (Python, scripts/gen_proto.py) -- each vendors its own +// protoc, so no system protoc install is required. +service ModelServer { + // Streams generated text token-by-token. The final message in the + // stream always has is_final=true and empty text; every message + // before it has is_final=false and carries exactly one token's text. + rpc Generate(GenerateRequest) returns (stream GenerateResponse); +} + +message GenerateRequest { + string request_id = 1; + string prompt = 2; + uint32 max_new_tokens = 3; +} + +message GenerateResponse { + string request_id = 1; + string text = 2; + bool is_final = 3; + double t_emit_unix = 4; +} diff --git a/pyproject.toml b/pyproject.toml index 47b881d..00f9982 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,6 +17,14 @@ dependencies = [ "safetensors>=0.8.0", # Linux-only: triton publishes manylinux wheels and nothing else. "triton>=3.8.0; sys_platform == 'linux'", + "grpcio>=1.84.0", + # The generated dispatch_pb2.py imports google.protobuf directly at + # runtime -- previously only available transitively via the dev-only + # grpcio-tools, so a --no-dev install (the model server's Docker + # image) had grpcio but not protobuf. Found live: a docker-compose + # run of the built image raised ModuleNotFoundError: No module named + # 'google' on startup. + "protobuf>=7.36.2", ] # Runtime dependencies (torch, triton, the serving stack) get added as each @@ -34,12 +42,19 @@ dev = [ "pytest-cov>=7.1.0", "ruff>=0.16.7", "types-requests>=2.33.0.20260906", + "grpcio-tools>=1.84.0", + "types-grpcio>=1.83.0.20260730", ] [tool.ruff] line-length = 100 target-version = "py312" src = ["src", "tests"] +extend-exclude = [ + "src/dispatch/proto/dispatch_pb2.py", + "src/dispatch/proto/dispatch_pb2.pyi", + "src/dispatch/proto/dispatch_pb2_grpc.py", +] [tool.ruff.lint] select = ["E", "F", "I", "N", "UP", "B", "SIM", "PL", "RUF", "C90"] @@ -72,6 +87,12 @@ python_version = "3.12" strict = true warn_unreachable = true mypy_path = "src" +# Generated protobuf/grpc code, excluded from direct checking -- the +# dispatch.proto.* override below (follow_imports="skip") only changes how +# this module is treated when *imported*, not when mypy walks `src` and +# hits it directly as a root target; found live when the override alone +# still left 10 errors inside these exact two files. +exclude = ["^src/dispatch/proto/dispatch_pb2(_grpc)?\\.pyi?$"] # Triton isn't installed at all on macOS. Skipping it makes mypy treat it as # Any everywhere, so the Mac and Linux CI give the same answer. @@ -96,6 +117,20 @@ module = ["vllm", "vllm.*", "sglang", "sglang.*"] ignore_missing_imports = true follow_imports = "skip" +# Generated protobuf/grpc code -- dispatch_pb2.py's message classes have a +# .pyi (gen_proto.py's --pyi_out), but dispatch_pb2_grpc.py's service/stub/ +# servicer classes have no official pyi generator and stay untyped, which +# under --strict makes every call into them a "call to untyped function" +# error at the caller. follow_imports="skip" (not ignore_errors, which only +# suppresses errors *inside* the module, not "untyped call" errors at +# *importers*) makes mypy treat the whole module as Any, same boundary as +# triton/deep_ep/vllm/sglang above -- found live when ignore_errors alone +# still left 14 real errors in files that import this module. +[[tool.mypy.overrides]] +module = ["dispatch.proto.*"] +ignore_missing_imports = true +follow_imports = "skip" + [tool.coverage.run] source = ["src/dispatch"] diff --git a/router/Cargo.lock b/router/Cargo.lock new file mode 100644 index 0000000..5b83c9c --- /dev/null +++ b/router/Cargo.lock @@ -0,0 +1,1215 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "aho-corasick" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba" +dependencies = [ + "memchr", +] + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "async-trait" +version = "0.1.92" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "axum" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" +dependencies = [ + "axum-core", + "bytes", + "form_urlencoded", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "itoa", + "matchit", + "memchr", + "mime", + "percent-encoding", + "pin-project-lite", + "serde_core", + "serde_json", + "serde_path_to_error", + "serde_urlencoded", + "sync_wrapper", + "tokio", + "tower", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "axum-core" +version = "0.5.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "http-body-util", + "mime", + "pin-project-lite", + "sync_wrapper", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "bitflags" +version = "2.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600" + +[[package]] +name = "dispatch-router" +version = "0.1.0" +dependencies = [ + "axum", + "futures-core", + "http-body-util", + "prometheus", + "prost", + "protoc-bin-vendored", + "serde", + "serde_json", + "tokio", + "tokio-stream", + "tonic", + "tonic-prost", + "tonic-prost-build", + "tower", +] + +[[package]] +name = "either" +version = "1.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34" + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "fixedbitset" +version = "0.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" + +[[package]] +name = "fnv" +version = "1.0.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures-channel" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4" +dependencies = [ + "futures-core", +] + +[[package]] +name = "futures-core" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e" + +[[package]] +name = "futures-sink" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d" + +[[package]] +name = "futures-task" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd" + +[[package]] +name = "futures-util" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc" +dependencies = [ + "futures-core", + "futures-task", + "pin-project-lite", + "slab", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "h2" +version = "0.4.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http", + "indexmap", + "slab", + "tokio", + "tokio-util", + "tracing", +] + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23169fe34a5fbcdd3f3862e78fb9b6fccd5f02a6dc6f732547005d45631ce71c" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "httpdate" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" + +[[package]] +name = "hyper" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "27b501faa50e7a26c3d3560ca625132f4078a17771f4810baf70475ae48cbe43" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "h2", + "http", + "http-body", + "httparse", + "httpdate", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-timeout" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" +dependencies = [ + "hyper", + "hyper-util", + "pin-project-lite", + "tokio", + "tower-service", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "libc", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "indexmap" +version = "2.14.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", +] + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" + +[[package]] +name = "matchit" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mio" +version = "1.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b18443e9c262bfe8fa82f51666e2642c53393f7e5c27b3e1aeab922cff5b9d8" +dependencies = [ + "libc", + "wasi", + "windows-sys", +] + +[[package]] +name = "multimap" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-link", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "petgraph" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" +dependencies = [ + "fixedbitset", + "hashbrown 0.15.5", + "indexmap", +] + +[[package]] +name = "pin-project" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn 2.0.119", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "prometheus" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3ca5326d8d0b950a9acd87e6a3f94745394f62e4dae1b1ee22b2bc0c394af43a" +dependencies = [ + "cfg-if", + "fnv", + "lazy_static", + "memchr", + "parking_lot", + "protobuf", + "thiserror 2.0.20", +] + +[[package]] +name = "prost" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1" +dependencies = [ + "bytes", + "prost-derive", +] + +[[package]] +name = "prost-build" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" +dependencies = [ + "heck", + "itertools", + "log", + "multimap", + "petgraph", + "prettyplease", + "prost", + "prost-types", + "pulldown-cmark", + "pulldown-cmark-to-cmark", + "regex", + "syn 2.0.119", + "tempfile", +] + +[[package]] +name = "prost-derive" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "prost-types" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f94967dc7688f3054c7fac87473ffae4cc4c3904800e2d9f5b857246d8963b0a" +dependencies = [ + "prost", +] + +[[package]] +name = "protobuf" +version = "3.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d65a1d4ddae7d8b5de68153b48f6aa3bba8cb002b243dbdbc55a5afbc98f99f4" +dependencies = [ + "once_cell", + "protobuf-support", + "thiserror 1.0.69", +] + +[[package]] +name = "protobuf-support" +version = "3.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e36c2f31e0a47f9280fb347ef5e461ffcd2c52dd520d8e216b52f93b0b0d7d6" +dependencies = [ + "thiserror 1.0.69", +] + +[[package]] +name = "protoc-bin-vendored" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d1c381df33c98266b5f08186583660090a4ffa0889e76c7e9a5e175f645a67fa" +dependencies = [ + "protoc-bin-vendored-linux-aarch_64", + "protoc-bin-vendored-linux-ppcle_64", + "protoc-bin-vendored-linux-s390_64", + "protoc-bin-vendored-linux-x86_32", + "protoc-bin-vendored-linux-x86_64", + "protoc-bin-vendored-macos-aarch_64", + "protoc-bin-vendored-macos-x86_64", + "protoc-bin-vendored-win32", +] + +[[package]] +name = "protoc-bin-vendored-linux-aarch_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c350df4d49b5b9e3ca79f7e646fde2377b199e13cfa87320308397e1f37e1a4c" + +[[package]] +name = "protoc-bin-vendored-linux-ppcle_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a55a63e6c7244f19b5c6393f025017eb5d793fd5467823a099740a7a4222440c" + +[[package]] +name = "protoc-bin-vendored-linux-s390_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1dba5565db4288e935d5330a07c264a4ee8e4a5b4a4e6f4e83fad824cc32f3b0" + +[[package]] +name = "protoc-bin-vendored-linux-x86_32" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8854774b24ee28b7868cd71dccaae8e02a2365e67a4a87a6cd11ee6cdbdf9cf5" + +[[package]] +name = "protoc-bin-vendored-linux-x86_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b38b07546580df720fa464ce124c4b03630a6fb83e05c336fea2a241df7e5d78" + +[[package]] +name = "protoc-bin-vendored-macos-aarch_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89278a9926ce312e51f1d999fee8825d324d603213344a9a706daa009f1d8092" + +[[package]] +name = "protoc-bin-vendored-macos-x86_64" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81745feda7ccfb9471d7a4de888f0652e806d5795b61480605d4943176299756" + +[[package]] +name = "protoc-bin-vendored-win32" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95067976aca6421a523e491fce939a3e65249bac4b977adee0ee9771568e8aa3" + +[[package]] +name = "pulldown-cmark" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9f068eba8e7071c5f9511831b44f32c740d5adf574e990f946ddb53db2f314e" +dependencies = [ + "bitflags", + "memchr", + "unicase", +] + +[[package]] +name = "pulldown-cmark-to-cmark" +version = "22.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ab1ad36992cead65f02aa399a373a42730922f1525d988172634fdefdecb8a60" +dependencies = [ + "pulldown-cmark", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags", +] + +[[package]] +name = "regex" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "rustix" +version = "1.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_path_to_error" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457" +dependencies = [ + "itoa", + "serde", + "serde_core", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba467056f1b547ed52077911161fc86985becbc60e8e1857c8a144dab0def891" + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom", + "once_cell", + "rustix", + "windows-sys", +] + +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl 1.0.69", +] + +[[package]] +name = "thiserror" +version = "2.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" +dependencies = [ + "thiserror-impl 2.0.20", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "libc", + "mio", + "pin-project-lite", + "socket2", + "tokio-macros", + "windows-sys", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.6", +] + +[[package]] +name = "tokio-stream" +version = "0.1.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3d06f0b082ba57c26b79407372e57cf2a1e28124f78e9479fe80322cf53420b" +dependencies = [ + "futures-core", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tokio-util" +version = "0.7.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "494815d09bf52b5548659851081238f0ca39ff638363907596da739561c62c52" +dependencies = [ + "bytes", + "futures-core", + "futures-sink", + "libc", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "tonic" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef" +dependencies = [ + "async-trait", + "axum", + "base64", + "bytes", + "h2", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-timeout", + "hyper-util", + "percent-encoding", + "pin-project", + "socket2", + "sync_wrapper", + "tokio", + "tokio-stream", + "tower", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tonic-build" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c68f61875ac5293cf72e6c8cf0158086428c82c37229e98c840878f1706b0322" +dependencies = [ + "prettyplease", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tonic-prost" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50849f68853be452acf590cde0b146665b8d507b3b8af17261df47e02c209ea0" +dependencies = [ + "bytes", + "prost", + "tonic", +] + +[[package]] +name = "tonic-prost-build" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "654e5643eff75d7f8c99197ce1440ed19a3474eada74c12bbac488b2cafdae27" +dependencies = [ + "prettyplease", + "proc-macro2", + "prost-build", + "prost-types", + "quote", + "syn 2.0.119", + "tempfile", + "tonic-build", +] + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "indexmap", + "pin-project-lite", + "slab", + "sync_wrapper", + "tokio", + "tokio-util", + "tower-layer", + "tower-service", + "tracing", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "log", + "pin-project-lite", + "tracing-attributes", + "tracing-core", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "unicase" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" + +[[package]] +name = "unicode-ident" +version = "1.0.26" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954" + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/router/Cargo.toml b/router/Cargo.toml new file mode 100644 index 0000000..0668d96 --- /dev/null +++ b/router/Cargo.toml @@ -0,0 +1,24 @@ +[package] +name = "dispatch-router" +version = "0.1.0" +edition = "2021" + +[dependencies] +tonic = "0.14.6" +prost = "0.14.4" +tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "sync", "time"] } +axum = "0.8.9" +serde = { version = "1.0.229", features = ["derive"] } +serde_json = "1.0.151" +prometheus = "0.14.0" +tonic-prost = "0.14.6" + +[build-dependencies] +protoc-bin-vendored = "3.2.0" +tonic-prost-build = "0.14.6" + +[dev-dependencies] +tower = { version = "0.5.3", features = ["util"] } +http-body-util = "0.1.5" +tokio-stream = "0.1.19" +futures-core = "0.3.34" diff --git a/router/build.rs b/router/build.rs new file mode 100644 index 0000000..baff7af --- /dev/null +++ b/router/build.rs @@ -0,0 +1,11 @@ +fn main() -> Result<(), Box> { + let protoc_path = protoc_bin_vendored::protoc_bin_path()?; + std::env::set_var("PROTOC", protoc_path); + // tonic 0.14 split prost integration out of tonic-build into + // tonic-prost-build (build-time codegen) + tonic-prost (the runtime + // ProstCodec the generated code references) -- tonic_build::compile_protos + // no longer exists. Found live by reading tonic-build 0.14.6's actual + // source after the old API call failed to compile. + tonic_prost_build::compile_protos("../proto/dispatch.proto")?; + Ok(()) +} diff --git a/router/src/app.rs b/router/src/app.rs new file mode 100644 index 0000000..a8092ad --- /dev/null +++ b/router/src/app.rs @@ -0,0 +1,183 @@ +use std::sync::Arc; + +use axum::extract::State; +use axum::response::IntoResponse; +use axum::routing::{get, post}; +use axum::{Json, Router}; +use serde::{Deserialize, Serialize}; +use tokio::sync::Mutex; + +use crate::client::ModelServerClient; +use crate::metrics::RouterMetrics; +use crate::queue::AdmissionQueue; + +#[derive(Clone)] +pub struct AppState { + pub queue: Arc, + pub client: Arc>, + pub metrics: Arc, +} + +#[derive(Debug, Deserialize)] +pub struct GenerateHttpRequest { + pub prompt: String, + pub max_new_tokens: u32, +} + +#[derive(Debug, Serialize, Deserialize)] +pub struct GenerateHttpResponse { + pub text: String, + pub ttft_ms: f64, + pub inter_token_latencies_ms: Vec, +} + +pub fn build_app(state: AppState) -> Router { + Router::new() + .route("/generate", post(generate)) + .route("/healthz", get(healthz)) + .route("/readyz", get(readyz)) + .route("/metrics", get(metrics_handler)) + .with_state(state) +} + +async fn generate( + State(state): State, + Json(req): Json, +) -> Result, axum::http::StatusCode> { + let _ticket = state.queue.admit().await; + state.metrics.queue_depth.set(state.queue.depth() as i64); + state.metrics.requests_total.inc(); + + let mut client = state.client.lock().await; + let request_id = request_id(); + let result = client + .generate(&request_id, &req.prompt, req.max_new_tokens, &state.metrics) + .await + .map_err(|_| axum::http::StatusCode::BAD_GATEWAY)?; + + Ok(Json(GenerateHttpResponse { + text: result.full_text, + ttft_ms: result.ttft.as_secs_f64() * 1000.0, + inter_token_latencies_ms: result + .inter_token_latencies + .iter() + .map(|d| d.as_secs_f64() * 1000.0) + .collect(), + })) +} + +async fn healthz() -> &'static str { + "ok" +} + +/// Startup blocks on a successful connection to the model server (see +/// main.rs) -- by the time this process is serving HTTP at all, it is +/// also ready, so readyz and healthz report the same thing here. Kept +/// as a separate endpoint because K8s conventionally wires liveness and +/// readiness probes to different paths. +async fn readyz() -> &'static str { + "ok" +} + +async fn metrics_handler(State(state): State) -> impl IntoResponse { + state.metrics.encode() +} + +fn request_id() -> String { + let nanos = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system clock is after the epoch") + .as_nanos(); + format!("{nanos:x}") +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use http_body_util::BodyExt; + use tokio::sync::Mutex; + use tower::ServiceExt; + + use super::*; + use crate::client::ModelServerClient; + use crate::metrics::RouterMetrics; + use crate::queue::AdmissionQueue; + use crate::test_support::spawn_mock_model_server; + + async fn test_state(chunks: Vec<&'static str>) -> AppState { + let endpoint = spawn_mock_model_server(chunks).await; + let client = ModelServerClient::connect(endpoint).await.expect("connect"); + AppState { + queue: Arc::new(AdmissionQueue::new(1)), + client: Arc::new(Mutex::new(client)), + metrics: Arc::new(RouterMetrics::new()), + } + } + + #[tokio::test] + async fn generate_endpoint_returns_full_text_and_timing() { + let app = build_app(test_state(vec!["Hello", " world"]).await); + + let request = axum::http::Request::builder() + .method("POST") + .uri("/generate") + .header("content-type", "application/json") + .body(axum::body::Body::from( + r#"{"prompt":"hi","max_new_tokens":2}"#, + )) + .expect("build request"); + + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK); + + let body = response + .into_body() + .collect() + .await + .expect("read body") + .to_bytes(); + let parsed: GenerateHttpResponse = + serde_json::from_slice(&body).expect("valid json response"); + assert_eq!(parsed.text, "Hello world"); + assert!(parsed.ttft_ms >= 0.0); + } + + #[tokio::test] + async fn healthz_and_readyz_return_ok() { + for path in ["/healthz", "/readyz"] { + let app = build_app(test_state(vec!["ok"]).await); + let request = axum::http::Request::builder() + .uri(path) + .body(axum::body::Body::empty()) + .expect("build request"); + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!( + response.status(), + axum::http::StatusCode::OK, + "path: {path}" + ); + } + } + + #[tokio::test] + async fn metrics_endpoint_exposes_prometheus_text_format() { + let app = build_app(test_state(vec!["ok"]).await); + + let request = axum::http::Request::builder() + .uri("/metrics") + .body(axum::body::Body::empty()) + .expect("build request"); + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK); + + let body = response + .into_body() + .collect() + .await + .expect("read body") + .to_bytes(); + let text = String::from_utf8(body.to_vec()).expect("utf8"); + assert!(text.contains("dispatch_router_queue_depth")); + } +} diff --git a/router/src/client.rs b/router/src/client.rs new file mode 100644 index 0000000..19eaca5 --- /dev/null +++ b/router/src/client.rs @@ -0,0 +1,103 @@ +use std::time::{Duration, Instant}; + +use tonic::transport::Channel; +use tonic::Request; + +use crate::metrics::RouterMetrics; +use crate::pb::dispatch_v1::model_server_client::ModelServerClient as GrpcClient; +use crate::pb::dispatch_v1::GenerateRequest; + +pub struct ModelServerClient { + inner: GrpcClient, +} + +#[derive(Debug, Clone)] +pub struct StreamedGeneration { + pub full_text: String, + pub ttft: Duration, + pub inter_token_latencies: Vec, +} + +impl ModelServerClient { + pub async fn connect(endpoint: String) -> Result { + let inner = GrpcClient::connect(endpoint).await?; + Ok(Self { inner }) + } + + pub async fn generate( + &mut self, + request_id: &str, + prompt: &str, + max_new_tokens: u32, + metrics: &RouterMetrics, + ) -> Result { + let start = Instant::now(); + let mut stream = self + .inner + .generate(Request::new(GenerateRequest { + request_id: request_id.to_string(), + prompt: prompt.to_string(), + max_new_tokens, + })) + .await? + .into_inner(); + + let mut full_text = String::new(); + let mut ttft: Option = None; + let mut inter_token_latencies = Vec::new(); + let mut last_token_at = start; + + while let Some(chunk) = stream.message().await? { + if chunk.is_final { + break; + } + let now = Instant::now(); + match ttft { + None => { + let elapsed = now.duration_since(start); + metrics.ttft_seconds.observe(elapsed.as_secs_f64()); + ttft = Some(elapsed); + } + Some(_) => { + let gap = now.duration_since(last_token_at); + metrics + .inter_token_latency_seconds + .observe(gap.as_secs_f64()); + inter_token_latencies.push(gap); + } + } + last_token_at = now; + full_text.push_str(&chunk.text); + } + + Ok(StreamedGeneration { + full_text, + ttft: ttft.unwrap_or_default(), + inter_token_latencies, + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::test_support::spawn_mock_model_server; + + #[tokio::test] + async fn generate_accumulates_text_and_records_latencies() { + let endpoint = spawn_mock_model_server(vec!["Hello", " world", "!"]).await; + let mut client = ModelServerClient::connect(endpoint).await.expect("connect"); + let metrics = RouterMetrics::new(); + + let result = client + .generate("req-1", "hi", 3, &metrics) + .await + .expect("generate succeeds"); + + assert_eq!(result.full_text, "Hello world!"); + assert_eq!(result.inter_token_latencies.len(), 2); + assert!(metrics + .encode() + .contains("dispatch_router_ttft_seconds_count 1")); + } +} diff --git a/router/src/lib.rs b/router/src/lib.rs new file mode 100644 index 0000000..10fc839 --- /dev/null +++ b/router/src/lib.rs @@ -0,0 +1,8 @@ +pub mod app; +pub mod client; +pub mod metrics; +pub mod pb; +pub mod queue; + +#[cfg(test)] +pub mod test_support; diff --git a/router/src/main.rs b/router/src/main.rs new file mode 100644 index 0000000..19811cc --- /dev/null +++ b/router/src/main.rs @@ -0,0 +1,62 @@ +use std::env; +use std::sync::Arc; +use std::time::Duration; + +use tokio::sync::Mutex; + +use dispatch_router::app::{build_app, AppState}; +use dispatch_router::client::ModelServerClient; +use dispatch_router::metrics::RouterMetrics; +use dispatch_router::queue::AdmissionQueue; + +const CONNECT_RETRY_ATTEMPTS: u32 = 30; +const CONNECT_RETRY_DELAY: Duration = Duration::from_secs(1); + +/// Container/pod startup order guarantees the model server's *container* +/// has started, not that it has finished syncing its venv and bound its +/// port -- found live: a docker-compose run had the router exit +/// immediately on a single failed connect attempt while the model server +/// was still starting. Retries for up to CONNECT_RETRY_ATTEMPTS * +/// CONNECT_RETRY_DELAY before giving up for good. +async fn connect_with_retry(endpoint: &str) -> Result { + let mut last_err = None; + for attempt in 1..=CONNECT_RETRY_ATTEMPTS { + match ModelServerClient::connect(endpoint.to_string()).await { + Ok(client) => return Ok(client), + Err(err) => { + println!( + "model server not ready yet (attempt {attempt}/{CONNECT_RETRY_ATTEMPTS}): {err}" + ); + last_err = Some(err); + tokio::time::sleep(CONNECT_RETRY_DELAY).await; + } + } + } + Err(last_err.expect("loop runs at least once")) +} + +#[tokio::main] +async fn main() -> Result<(), Box> { + let model_server_endpoint = + env::var("MODEL_SERVER_ENDPOINT").unwrap_or_else(|_| "http://127.0.0.1:50051".to_string()); + let listen_addr = env::var("ROUTER_LISTEN_ADDR").unwrap_or_else(|_| "0.0.0.0:8080".to_string()); + let queue_capacity: usize = env::var("ROUTER_QUEUE_CAPACITY") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(1); + + println!("connecting to model server at {model_server_endpoint}"); + let client = connect_with_retry(&model_server_endpoint).await?; + + let state = AppState { + queue: Arc::new(AdmissionQueue::new(queue_capacity)), + client: Arc::new(Mutex::new(client)), + metrics: Arc::new(RouterMetrics::new()), + }; + + let app = build_app(state); + let listener = tokio::net::TcpListener::bind(&listen_addr).await?; + println!("dispatch-router listening on {listen_addr}"); + axum::serve(listener, app).await?; + Ok(()) +} diff --git a/router/src/metrics.rs b/router/src/metrics.rs new file mode 100644 index 0000000..27561ed --- /dev/null +++ b/router/src/metrics.rs @@ -0,0 +1,95 @@ +use prometheus::{ + Encoder, Histogram, HistogramOpts, IntCounter, IntGauge, Opts, Registry, TextEncoder, +}; + +pub struct RouterMetrics { + registry: Registry, + pub requests_total: IntCounter, + pub ttft_seconds: Histogram, + pub inter_token_latency_seconds: Histogram, + pub queue_depth: IntGauge, +} + +impl RouterMetrics { + pub fn new() -> Self { + let registry = Registry::new(); + + let requests_total = IntCounter::with_opts(Opts::new( + "dispatch_router_requests_total", + "Total requests admitted by the router", + )) + .expect("valid metric opts"); + registry + .register(Box::new(requests_total.clone())) + .expect("first registration of this metric"); + + let ttft_seconds = Histogram::with_opts(HistogramOpts::new( + "dispatch_router_ttft_seconds", + "Time to first token, seconds", + )) + .expect("valid metric opts"); + registry + .register(Box::new(ttft_seconds.clone())) + .expect("first registration of this metric"); + + let inter_token_latency_seconds = Histogram::with_opts(HistogramOpts::new( + "dispatch_router_inter_token_latency_seconds", + "Gap between consecutive tokens, seconds", + )) + .expect("valid metric opts"); + registry + .register(Box::new(inter_token_latency_seconds.clone())) + .expect("first registration of this metric"); + + let queue_depth = IntGauge::with_opts(Opts::new( + "dispatch_router_queue_depth", + "Requests currently waiting for the single-flight model server", + )) + .expect("valid metric opts"); + registry + .register(Box::new(queue_depth.clone())) + .expect("first registration of this metric"); + + Self { + registry, + requests_total, + ttft_seconds, + inter_token_latency_seconds, + queue_depth, + } + } + + pub fn encode(&self) -> String { + let metric_families = self.registry.gather(); + let mut buffer = Vec::new(); + TextEncoder::new() + .encode(&metric_families, &mut buffer) + .expect("prometheus text encoding never fails for valid metric families"); + String::from_utf8(buffer).expect("prometheus TextEncoder always emits valid utf-8") + } +} + +impl Default for RouterMetrics { + fn default() -> Self { + Self::new() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn encode_reports_recorded_values() { + let metrics = RouterMetrics::new(); + metrics.requests_total.inc(); + metrics.ttft_seconds.observe(0.25); + metrics.queue_depth.set(3); + + let text = metrics.encode(); + + assert!(text.contains("dispatch_router_requests_total 1")); + assert!(text.contains("dispatch_router_queue_depth 3")); + assert!(text.contains("dispatch_router_ttft_seconds_sum 0.25")); + } +} diff --git a/router/src/pb.rs b/router/src/pb.rs new file mode 100644 index 0000000..7c7dafb --- /dev/null +++ b/router/src/pb.rs @@ -0,0 +1,25 @@ +pub mod dispatch_v1 { + tonic::include_proto!("dispatch.v1"); +} + +#[cfg(test)] +mod tests { + use prost::Message; + + use super::dispatch_v1::GenerateRequest; + + #[test] + fn generate_request_round_trips_through_prost_encoding() { + let original = GenerateRequest { + request_id: "r1".to_string(), + prompt: "hello".to_string(), + max_new_tokens: 16, + }; + + let mut buf = Vec::new(); + original.encode(&mut buf).expect("encode"); + let decoded = GenerateRequest::decode(buf.as_slice()).expect("decode"); + + assert_eq!(decoded, original); + } +} diff --git a/router/src/queue.rs b/router/src/queue.rs new file mode 100644 index 0000000..e830af6 --- /dev/null +++ b/router/src/queue.rs @@ -0,0 +1,87 @@ +use std::sync::atomic::{AtomicUsize, Ordering}; + +use tokio::sync::{Semaphore, SemaphorePermit}; + +pub struct AdmissionQueue { + semaphore: Semaphore, + waiting: AtomicUsize, +} + +pub struct AdmissionTicket<'a> { + _permit: SemaphorePermit<'a>, +} + +impl AdmissionQueue { + pub fn new(capacity: usize) -> Self { + Self { + semaphore: Semaphore::new(capacity), + waiting: AtomicUsize::new(0), + } + } + + pub fn depth(&self) -> usize { + self.waiting.load(Ordering::SeqCst) + } + + pub async fn admit(&self) -> AdmissionTicket<'_> { + self.waiting.fetch_add(1, Ordering::SeqCst); + let permit = self + .semaphore + .acquire() + .await + .expect("semaphore is never closed"); + self.waiting.fetch_sub(1, Ordering::SeqCst); + AdmissionTicket { _permit: permit } + } +} + +#[cfg(test)] +mod tests { + use std::sync::atomic::{AtomicUsize as StdAtomicUsize, Ordering as StdOrdering}; + use std::sync::Arc; + + use super::*; + + #[tokio::test] + async fn admit_serializes_single_capacity_access() { + let queue = Arc::new(AdmissionQueue::new(1)); + let concurrent = Arc::new(StdAtomicUsize::new(0)); + let max_concurrent = Arc::new(StdAtomicUsize::new(0)); + + let mut handles = Vec::new(); + for _ in 0..5 { + let queue = queue.clone(); + let concurrent = concurrent.clone(); + let max_concurrent = max_concurrent.clone(); + handles.push(tokio::spawn(async move { + let _ticket = queue.admit().await; + let now = concurrent.fetch_add(1, StdOrdering::SeqCst) + 1; + max_concurrent.fetch_max(now, StdOrdering::SeqCst); + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + concurrent.fetch_sub(1, StdOrdering::SeqCst); + })); + } + for handle in handles { + handle.await.expect("task did not panic"); + } + + assert_eq!(max_concurrent.load(StdOrdering::SeqCst), 1); + } + + #[tokio::test] + async fn depth_reflects_waiting_tasks() { + let queue = Arc::new(AdmissionQueue::new(1)); + let held = queue.admit().await; + + let queue2 = queue.clone(); + let waiter = tokio::spawn(async move { + let _ticket = queue2.admit().await; + }); + + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + assert_eq!(queue.depth(), 1); + + drop(held); + waiter.await.expect("task did not panic"); + } +} diff --git a/router/src/test_support.rs b/router/src/test_support.rs new file mode 100644 index 0000000..67fd5e7 --- /dev/null +++ b/router/src/test_support.rs @@ -0,0 +1,70 @@ +use std::pin::Pin; + +use tokio::net::TcpListener; +use tokio::sync::mpsc; +use tokio_stream::wrappers::{ReceiverStream, TcpListenerStream}; +use tonic::{Request, Response, Status}; + +use crate::pb::dispatch_v1::model_server_server::{ModelServer, ModelServerServer}; +use crate::pb::dispatch_v1::{GenerateRequest, GenerateResponse}; + +struct MockModelServer { + chunks: Vec<&'static str>, +} + +#[tonic::async_trait] +impl ModelServer for MockModelServer { + type GenerateStream = + Pin> + Send>>; + + async fn generate( + &self, + request: Request, + ) -> Result, Status> { + let request_id = request.into_inner().request_id; + let chunks = self.chunks.clone(); + let (tx, rx) = mpsc::channel(8); + tokio::spawn(async move { + for text in chunks { + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + let _ = tx + .send(Ok(GenerateResponse { + request_id: request_id.clone(), + text: text.to_string(), + is_final: false, + t_emit_unix: 0.0, + })) + .await; + } + let _ = tx + .send(Ok(GenerateResponse { + request_id, + text: String::new(), + is_final: true, + t_emit_unix: 0.0, + })) + .await; + }); + Ok(Response::new(Box::pin(ReceiverStream::new(rx)))) + } +} + +/// Starts an in-process, real gRPC server implementing ModelServer that +/// streams `chunks` back as separate non-final tokens followed by one +/// final empty message, then returns its `http://host:port` endpoint. +/// Test-only (router/src/lib.rs gates this module behind `#[cfg(test)]`). +pub async fn spawn_mock_model_server(chunks: Vec<&'static str>) -> String { + let listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind ephemeral port"); + let addr = listener.local_addr().expect("local addr"); + tokio::spawn(async move { + tonic::transport::Server::builder() + .add_service(ModelServerServer::new(MockModelServer { chunks })) + .serve_with_incoming(TcpListenerStream::new(listener)) + .await + .expect("mock server exited unexpectedly"); + }); + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + format!("http://{addr}") +} diff --git a/router/tests/cross_language_integration.rs b/router/tests/cross_language_integration.rs new file mode 100644 index 0000000..0aaf9a5 --- /dev/null +++ b/router/tests/cross_language_integration.rs @@ -0,0 +1,131 @@ +use std::os::unix::process::CommandExt; +use std::path::PathBuf; +use std::process::{Child, Command, Stdio}; +use std::sync::Arc; +use std::time::Duration; + +use tokio::sync::Mutex; + +use dispatch_router::app::{build_app, AppState}; +use dispatch_router::client::ModelServerClient; +use dispatch_router::metrics::RouterMetrics; +use dispatch_router::queue::AdmissionQueue; + +const TEST_PORT: u16 = 50099; + +struct StubServerGuard(Child); + +impl Drop for StubServerGuard { + fn drop(&mut self) { + // `uv run python ...` forks rather than exec'ing into the real + // interpreter on this machine -- found live: killing only the + // immediate `uv` child left the actual model_server.py process + // running, orphaned and reparented to launchd, still listening + // on TEST_PORT after the test had already passed. spawn_stub_server + // puts this child in its own process group (process_group(0)), so + // killing the *group* (negative pid) reaches uv, python, and + // anything else uv forked, not just the one pid this Child handle + // names. + let pid = self.0.id(); + let _ = Command::new("kill") + .args(["-9", &format!("-{pid}")]) + .status(); + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn repo_root() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .parent() + .expect("router/ has a parent directory") + .to_path_buf() +} + +fn spawn_stub_server() -> StubServerGuard { + // stderr must NOT be Stdio::inherit(): that hands this process's own + // stderr fd to `uv run` and everything it execs. If this test's own + // output is itself piped somewhere (`cargo test ... | tail`, as in a + // manual invocation), the inherited fd is the pipe's write end -- + // and if this test process is then killed before its Drop guard runs + // (SIGKILL, or the harness reclaiming a timed-out foreground command), + // the orphaned Python subprocess keeps that fd open forever, so the + // pipe reader never sees EOF and hangs indefinitely. Found live: a + // manually piped `cargo test ... | tail -60` run hung for 27+ minutes + // after this test's own process was SIGKILLed, with the model server + // still running as an orphan reparented to launchd. Stdio::piped() + // (captured, never inherited) closes that hole. + let child = Command::new("uv") + .args([ + "run", + "python", + "scripts/run_model_server.py", + "--responder", + "stub", + "--port", + &TEST_PORT.to_string(), + ]) + .current_dir(repo_root()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .process_group(0) + .spawn() + .expect("failed to spawn scripts/run_model_server.py -- is uv on PATH?"); + StubServerGuard(child) +} + +async fn wait_for_server_ready(endpoint: &str) { + for _ in 0..50 { + if ModelServerClient::connect(endpoint.to_string()) + .await + .is_ok() + { + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + panic!("stub model server never became reachable at {endpoint}"); +} + +#[tokio::test] +async fn router_streams_real_python_stub_server_responses_over_http() { + let _guard = spawn_stub_server(); + let endpoint = format!("http://127.0.0.1:{TEST_PORT}"); + wait_for_server_ready(&endpoint).await; + + let client = ModelServerClient::connect(endpoint) + .await + .expect("connect to the real Python stub server"); + let state = AppState { + queue: Arc::new(AdmissionQueue::new(1)), + client: Arc::new(Mutex::new(client)), + metrics: Arc::new(RouterMetrics::new()), + }; + let app = build_app(state); + + let request = axum::http::Request::builder() + .method("POST") + .uri("/generate") + .header("content-type", "application/json") + .body(axum::body::Body::from( + r#"{"prompt":"hi","max_new_tokens":3}"#, + )) + .expect("build request"); + + use tower::ServiceExt; + let response = app.oneshot(request).await.expect("router did not panic"); + assert_eq!(response.status(), axum::http::StatusCode::OK); + + use http_body_util::BodyExt; + let body = response + .into_body() + .collect() + .await + .expect("read body") + .to_bytes(); + let parsed: dispatch_router::app::GenerateHttpResponse = + serde_json::from_slice(&body).expect("valid json response"); + + // StubResponder's default tokens, from model_server.py. + assert_eq!(parsed.text, "The quick brown"); +} diff --git a/scripts/gen_proto.py b/scripts/gen_proto.py new file mode 100644 index 0000000..ad91070 --- /dev/null +++ b/scripts/gen_proto.py @@ -0,0 +1,61 @@ +"""Regenerates the Python gRPC stubs from proto/dispatch.proto. Two known +grpc_tools limitations are worked around here: it emits a bare +`import dispatch_pb2` in the generated _grpc.py file, which breaks once +the file lives inside the dispatch.proto package (fixed by rewriting that +one import to a relative one after generation); and --python_out alone +generates message classes via runtime reflection, invisible to mypy -- +--pyi_out generates the accompanying .pyi so GenerateRequest/ +GenerateResponse are statically typed (dispatch_pb2_grpc's +service/stub/servicer classes still aren't -- no official pyi generator +covers those, which is why pyproject.toml's mypy override for +dispatch.proto.* uses follow_imports="skip" rather than relying on stubs +alone). +""" + +from __future__ import annotations + +import re +import subprocess +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +PROTO_DIR = REPO_ROOT / "proto" +OUT_DIR = REPO_ROOT / "src" / "dispatch" / "proto" + + +def main() -> None: + OUT_DIR.mkdir(parents=True, exist_ok=True) + (OUT_DIR / "__init__.py").touch(exist_ok=True) + subprocess.run( + [ + sys.executable, + "-m", + "grpc_tools.protoc", + f"-I{PROTO_DIR}", + f"--python_out={OUT_DIR}", + f"--pyi_out={OUT_DIR}", + f"--grpc_python_out={OUT_DIR}", + str(PROTO_DIR / "dispatch.proto"), + ], + check=True, + cwd=REPO_ROOT, + ) + grpc_file = OUT_DIR / "dispatch_pb2_grpc.py" + text = grpc_file.read_text() + patched = re.sub( + r"^import dispatch_pb2 as dispatch__pb2$", + "from . import dispatch_pb2 as dispatch__pb2", + text, + flags=re.MULTILINE, + ) + if patched == text: + raise RuntimeError( + "expected bare 'import dispatch_pb2' line not found -- " + "grpc_tools output format changed since this script was written" + ) + grpc_file.write_text(patched) + + +if __name__ == "__main__": + main() diff --git a/scripts/gpu/phase7_kernel_responder.py b/scripts/gpu/phase7_kernel_responder.py new file mode 100644 index 0000000..d849a29 --- /dev/null +++ b/scripts/gpu/phase7_kernel_responder.py @@ -0,0 +1,164 @@ +"""GPU-only real inference for Phase 7's model server demo: wraps the +real deepseek-ai/deepseek-moe-16b-base model, Phase 6's naive bf16 +kernel, and Phase 4's ColocatedWorker into model_server.py's Responder +protocol. Imported lazily by scripts/run_model_server.py only when +--responder kernel is passed -- never imported by CPU tests or CI. + +Single-flight only: ColocatedWorker's continuous-batching machinery is +reused, but this responder enforces at most one in-flight stream at a +time (a lock, not a queue) -- Phase 7's design explicitly scopes out +concurrent multi-stream serving (design doc SS6, non-goals). +""" + +from __future__ import annotations + +import threading +import time +from collections.abc import Iterator +from typing import Any + +import torch +from tokenizers import decoders, pre_tokenizers +from transformers import DynamicCache, PreTrainedTokenizerBase + +from dispatch.benchmark.harness import load_model +from dispatch.kernels.backends import resolve_backend +from dispatch.kernels.integration import fix_rope_inv_freq, patch_moe_infer +from dispatch.serving.colocated import ColocatedWorker +from dispatch.serving.disaggregated import DecodeFn, PrefillFn, Request +from dispatch.serving.model_server import ServerBusyError, TokenEvent + +MODEL_NAME = "deepseek-ai/deepseek-moe-16b-base" +DEMO_REQUEST_ID = "demo" + + +# Same pre-v5 DynamicCache shim Phase 3/4's own GPU scripts use -- +# DeepSeek's remote-code modeling file still calls the removed +# get_usable_length even against this repo's transformers>=5.17.0 floor. +def _get_usable_length( + self: DynamicCache, new_seq_length: int | None = None, layer_idx: int = 0 +) -> int: + return self.get_seq_length(layer_idx) + + +DynamicCache.get_usable_length = _get_usable_length # type: ignore[attr-defined] + + +def make_prefill_fn(model: torch.nn.Module, device: str) -> PrefillFn: + def prefill_fn(input_ids: torch.Tensor, attention_mask: torch.Tensor) -> Any: + with torch.no_grad(): + outputs = model( + input_ids=input_ids.to(device), + attention_mask=attention_mask.to(device), + use_cache=True, + ) + first_tokens = outputs.logits[:, -1, :].argmax(dim=-1) + cache = DynamicCache.from_legacy_cache(outputs.past_key_values) # type: ignore[attr-defined] + return first_tokens.cpu(), cache + + return prefill_fn + + +def make_decode_fn(model: torch.nn.Module, device: str) -> DecodeFn: + def decode_fn( + next_input_ids: torch.Tensor, + cache: DynamicCache, + attention_mask: torch.Tensor, + position_ids: torch.Tensor, + ) -> Any: + with torch.no_grad(): + outputs = model( + input_ids=next_input_ids.to(device), + past_key_values=cache.to_legacy_cache(), # type: ignore[attr-defined] + attention_mask=attention_mask.to(device), + position_ids=position_ids.to(device), + use_cache=True, + ) + next_tokens = outputs.logits[:, -1, :].argmax(dim=-1) + updated_cache = DynamicCache.from_legacy_cache(outputs.past_key_values) # type: ignore[attr-defined] + return next_tokens.cpu(), updated_cache + + return decode_fn + + +class KernelResponder: + def __init__(self, model: torch.nn.Module, tokenizer: Any, *, device: str = "cuda") -> None: + self.tokenizer = tokenizer + self._device = device + self._prefill_fn = make_prefill_fn(model, device) + self._decode_fn = make_decode_fn(model, device) + self._lock = threading.Lock() + + def generate(self, prompt: str, max_new_tokens: int) -> Iterator[TokenEvent]: + if not self._lock.acquire(blocking=False): + raise ServerBusyError("model server is already streaming one request") + try: + input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids + worker = ColocatedWorker(self._prefill_fn, self._decode_fn, batch_size=1) + worker.submit( + Request( + request_id=DEMO_REQUEST_ID, + prompt_ids=input_ids, + max_new_tokens=max_new_tokens, + eos_token_id=self.tokenizer.eos_token_id, + ) + ) + emitted_text = "" + while True: + completed = worker.step() + active = worker.snapshot_active_tokens() + token_ids = active.get(DEMO_REQUEST_ID) + if token_ids is None and completed: + token_ids = completed[0].generated_token_ids + if token_ids is not None: + # Decoding the whole sequence and diffing against the + # previous decode -- not each new token id in + # isolation -- because a byte-level BPE tokenizer's + # decode() only converts its internal space marker + # ('Ġ') into a real space when it has surrounding + # context. Found live on the real GPU session: per- + # token decode produced 'TheĠquickĠbrown' + # instead of 'The quick brown'. + full_text = self.tokenizer.decode(token_ids) + new_text = full_text[len(emitted_text) :] + if new_text: + yield TokenEvent(text=new_text, is_final=False, t_emit=time.perf_counter()) + emitted_text = full_text + if completed: + yield TokenEvent(text="", is_final=True, t_emit=time.perf_counter()) + return + finally: + self._lock.release() + + +# deepseek-ai/deepseek-moe-16b-base ships a tokenizer.json whose vocab is +# majority GPT-2-style byte-level BPE (47,723 of 100,000 entries carry the +# 'Ġ' space marker) but is wired to a SentencePiece Metaspace +# pre-tokenizer/decoder pair (expects '▁', which appears in zero vocab +# entries). Found live on this GPU session: encode() glues words together +# with no space markers at all ("The quick brown fox" -> a token stream +# indistinguishable from "Thequickbrownfox"), and decode() then either +# drops spaces silently or leaks a literal 'Ġ'/'Ċ' glyph, depending on +# which vocab entries the greedy path happens to hit. This is a defect in +# the model's own shipped tokenizer file, not in transformers or dispatch +# -- Phases 0-6 never noticed because they only ever compared logits/token +# ids, never decoded text for a human to read. Rewiring both the +# pre-tokenizer and decoder to ByteLevel (matching what the vocab actually +# is) round-trips exactly: encode(decode(ids)) == ids, verified against +# several multi-word prompts. +def fix_tokenizer_byte_level(tokenizer: PreTrainedTokenizerBase) -> None: + backend = tokenizer.backend_tokenizer + backend.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=True) + backend.decoder = decoders.ByteLevel() + + +def build_kernel_responder(*, moe_kernel: str = "naive") -> KernelResponder: + model, tokenizer = load_model( + MODEL_NAME, device="cuda", dtype=torch.bfloat16, trust_remote_code=True + ) + fix_tokenizer_byte_level(tokenizer) + fix_rope_inv_freq(model) + patched = patch_moe_infer(model, resolve_backend(moe_kernel)) + if patched == 0: + raise RuntimeError(f"patch_moe_infer patched zero layers for backend {moe_kernel!r}") + return KernelResponder(model, tokenizer, device="cuda") diff --git a/scripts/run_kind_demo.sh b/scripts/run_kind_demo.sh new file mode 100755 index 0000000..3f98fec --- /dev/null +++ b/scripts/run_kind_demo.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Brings up the K8s side of Phase 7's demo: a kind cluster with the +# router, Prometheus, and Grafana. Assumes something is already +# listening on the dev machine's port 50051 for the model-server +# ExternalName Service to reach -- the stub responder for this task's +# rehearsal (`uv run python scripts/run_model_server.py --responder stub +# --port 50051`, in another terminal), or the real SSH tunnel for +# Task 13's paid session. This script's own job is K8s bring-up only. +set -euo pipefail + +CLUSTER_NAME="dispatch-demo" +NAMESPACE="dispatch-demo" + +echo "[1/6] Creating kind cluster ${CLUSTER_NAME} (if not already up)" +if ! kind get clusters | grep -qx "${CLUSTER_NAME}"; then + kind create cluster --name "${CLUSTER_NAME}" +fi +kubectl config use-context "kind-${CLUSTER_NAME}" + +echo "[2/6] Building and loading images into the kind cluster" +docker build -t dispatch-router:demo -f docker/router.Dockerfile . +docker build -t dispatch-model-server:demo -f docker/model-server.Dockerfile . +kind load docker-image dispatch-router:demo --name "${CLUSTER_NAME}" +kind load docker-image dispatch-model-server:demo --name "${CLUSTER_NAME}" + +echo "[3/6] Applying manifests" +kubectl apply -f k8s/namespace.yaml +kubectl apply -f k8s/router-deployment.yaml +kubectl apply -f k8s/model-server-external.yaml +kubectl apply -f k8s/prometheus.yaml +kubectl apply -f k8s/grafana.yaml + +echo "[4/6] Waiting for rollouts" +kubectl -n "${NAMESPACE}" rollout status deployment/router --timeout=120s +kubectl -n "${NAMESPACE}" rollout status deployment/prometheus --timeout=120s +kubectl -n "${NAMESPACE}" rollout status deployment/grafana --timeout=120s + +echo "[5/6] Port-forwarding router (8080) and Grafana (3000) to localhost" +kubectl -n "${NAMESPACE}" port-forward svc/router 8080:8080 & +ROUTER_PF_PID=$! +kubectl -n "${NAMESPACE}" port-forward svc/grafana 3000:3000 & +GRAFANA_PF_PID=$! +trap 'kill ${ROUTER_PF_PID} ${GRAFANA_PF_PID} 2>/dev/null || true' EXIT +sleep 3 + +echo "[6/6] Ready." +echo " Demo request: curl -s -X POST localhost:8080/generate -H 'content-type: application/json' -d '{\"prompt\":\"The quick brown fox\",\"max_new_tokens\":16}'" +echo " Grafana: http://localhost:3000 (anonymous admin)" +echo "Press Ctrl-C to stop the port-forwards. Tear down the cluster separately with:" +echo " kind delete cluster --name ${CLUSTER_NAME}" +wait diff --git a/scripts/run_model_server.py b/scripts/run_model_server.py new file mode 100644 index 0000000..e13e693 --- /dev/null +++ b/scripts/run_model_server.py @@ -0,0 +1,46 @@ +"""CLI entrypoint for Phase 7's model server. `--responder stub` (the +default) needs nothing but this repo's normal dependencies and is what +tests, the cross-language router integration test, and docker-compose +local verification all run. `--responder kernel` additionally imports +scripts/gpu/phase7_kernel_responder.py (torch, the real model, Phase 6's +naive kernel) -- lazily, only when selected, so this module stays +importable on a machine with neither GPU nor torch installed. +""" + +from __future__ import annotations + +import argparse + +from dispatch.serving.model_server import Responder, StubResponder, serve + + +def build_responder(args: argparse.Namespace) -> Responder: + if args.responder == "stub": + return StubResponder() + from scripts.gpu.phase7_kernel_responder import build_kernel_responder # noqa: PLC0415 + + return build_kernel_responder(moe_kernel=args.moe_kernel) + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--responder", choices=["stub", "kernel"], default="stub") + parser.add_argument("--port", type=int, default=50051) + parser.add_argument( + "--moe-kernel", + default="naive", + help="Only used with --responder kernel; passed to resolve_backend.", + ) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> None: + args = parse_args(argv) + responder = build_responder(args) + server, bound_port = serve(responder, port=args.port) + print(f"model server listening on port {bound_port}, responder={args.responder}") + server.wait_for_termination() + + +if __name__ == "__main__": + main() diff --git a/src/dispatch/proto/__init__.py b/src/dispatch/proto/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/dispatch/proto/dispatch_pb2.py b/src/dispatch/proto/dispatch_pb2.py new file mode 100644 index 0000000..098088b --- /dev/null +++ b/src/dispatch/proto/dispatch_pb2.py @@ -0,0 +1,40 @@ +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# NO CHECKED-IN PROTOBUF GENCODE +# source: dispatch.proto +# Protobuf Python Version: 7.35.1 +"""Generated protocol buffer code.""" +from google.protobuf import descriptor as _descriptor +from google.protobuf import descriptor_pool as _descriptor_pool +from google.protobuf import runtime_version as _runtime_version +from google.protobuf import symbol_database as _symbol_database +from google.protobuf.internal import builder as _builder +_runtime_version.ValidateProtobufRuntimeVersion( + _runtime_version.Domain.PUBLIC, + 7, + 35, + 1, + '', + 'dispatch.proto' +) +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + + + +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0e\x64ispatch.proto\x12\x0b\x64ispatch.v1\"M\n\x0fGenerateRequest\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12\x0e\n\x06prompt\x18\x02 \x01(\t\x12\x16\n\x0emax_new_tokens\x18\x03 \x01(\r\"[\n\x10GenerateResponse\x12\x12\n\nrequest_id\x18\x01 \x01(\t\x12\x0c\n\x04text\x18\x02 \x01(\t\x12\x10\n\x08is_final\x18\x03 \x01(\x08\x12\x13\n\x0bt_emit_unix\x18\x04 \x01(\x01\x32X\n\x0bModelServer\x12I\n\x08Generate\x12\x1c.dispatch.v1.GenerateRequest\x1a\x1d.dispatch.v1.GenerateResponse0\x01\x62\x06proto3') + +_globals = globals() +_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) +_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'dispatch_pb2', _globals) +if not _descriptor._USE_C_DESCRIPTORS: + DESCRIPTOR._loaded_options = None + _globals['_GENERATEREQUEST']._serialized_start=31 + _globals['_GENERATEREQUEST']._serialized_end=108 + _globals['_GENERATERESPONSE']._serialized_start=110 + _globals['_GENERATERESPONSE']._serialized_end=201 + _globals['_MODELSERVER']._serialized_start=203 + _globals['_MODELSERVER']._serialized_end=291 +# @@protoc_insertion_point(module_scope) diff --git a/src/dispatch/proto/dispatch_pb2.pyi b/src/dispatch/proto/dispatch_pb2.pyi new file mode 100644 index 0000000..a3d2dc1 --- /dev/null +++ b/src/dispatch/proto/dispatch_pb2.pyi @@ -0,0 +1,27 @@ +from google.protobuf import descriptor as _descriptor +from google.protobuf import message as _message +from typing import ClassVar as _ClassVar, Optional as _Optional + +DESCRIPTOR: _descriptor.FileDescriptor + +class GenerateRequest(_message.Message): + __slots__ = ("request_id", "prompt", "max_new_tokens") + REQUEST_ID_FIELD_NUMBER: _ClassVar[int] + PROMPT_FIELD_NUMBER: _ClassVar[int] + MAX_NEW_TOKENS_FIELD_NUMBER: _ClassVar[int] + request_id: str + prompt: str + max_new_tokens: int + def __init__(self, request_id: _Optional[str] = ..., prompt: _Optional[str] = ..., max_new_tokens: _Optional[int] = ...) -> None: ... + +class GenerateResponse(_message.Message): + __slots__ = ("request_id", "text", "is_final", "t_emit_unix") + REQUEST_ID_FIELD_NUMBER: _ClassVar[int] + TEXT_FIELD_NUMBER: _ClassVar[int] + IS_FINAL_FIELD_NUMBER: _ClassVar[int] + T_EMIT_UNIX_FIELD_NUMBER: _ClassVar[int] + request_id: str + text: str + is_final: bool + t_emit_unix: float + def __init__(self, request_id: _Optional[str] = ..., text: _Optional[str] = ..., is_final: _Optional[bool] = ..., t_emit_unix: _Optional[float] = ...) -> None: ... diff --git a/src/dispatch/proto/dispatch_pb2_grpc.py b/src/dispatch/proto/dispatch_pb2_grpc.py new file mode 100644 index 0000000..5ce0a0f --- /dev/null +++ b/src/dispatch/proto/dispatch_pb2_grpc.py @@ -0,0 +1,115 @@ +# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! +"""Client and server classes corresponding to protobuf-defined services.""" +import grpc +import warnings + +from . import dispatch_pb2 as dispatch__pb2 + +GRPC_GENERATED_VERSION = '1.84.0' +GRPC_VERSION = grpc.__version__ +_version_not_supported = False + +try: + from grpc._utilities import first_version_is_lower + _version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION) +except ImportError: + _version_not_supported = True + +if _version_not_supported: + raise RuntimeError( + f'The grpc package installed is at version {GRPC_VERSION},' + + ' but the generated code in dispatch_pb2_grpc.py depends on' + + f' grpcio>={GRPC_GENERATED_VERSION}.' + + f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}' + + f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.' + ) + + +class ModelServerStub: + """The gRPC contract between the Rust router and the Python model server + (docs/design/2026-09-20-phase-7-productionization.md SS3-4). Compiled + by both sides -- tonic-build/prost-build (Rust, router/build.rs) and + grpcio-tools (Python, scripts/gen_proto.py) -- each vendors its own + protoc, so no system protoc install is required. + """ + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.Generate = channel.unary_stream( + '/dispatch.v1.ModelServer/Generate', + request_serializer=dispatch__pb2.GenerateRequest.SerializeToString, + response_deserializer=dispatch__pb2.GenerateResponse.FromString, + _registered_method=True) + + +class ModelServerServicer: + """The gRPC contract between the Rust router and the Python model server + (docs/design/2026-09-20-phase-7-productionization.md SS3-4). Compiled + by both sides -- tonic-build/prost-build (Rust, router/build.rs) and + grpcio-tools (Python, scripts/gen_proto.py) -- each vendors its own + protoc, so no system protoc install is required. + """ + + def Generate(self, request, context): + """Streams generated text token-by-token. The final message in the + stream always has is_final=true and empty text; every message + before it has is_final=false and carries exactly one token's text. + """ + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_ModelServerServicer_to_server(servicer, server): + rpc_method_handlers = { + 'Generate': grpc.unary_stream_rpc_method_handler( + servicer.Generate, + request_deserializer=dispatch__pb2.GenerateRequest.FromString, + response_serializer=dispatch__pb2.GenerateResponse.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'dispatch.v1.ModelServer', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + server.add_registered_method_handlers('dispatch.v1.ModelServer', rpc_method_handlers) + + + # This class is part of an EXPERIMENTAL API. +class ModelServer: + """The gRPC contract between the Rust router and the Python model server + (docs/design/2026-09-20-phase-7-productionization.md SS3-4). Compiled + by both sides -- tonic-build/prost-build (Rust, router/build.rs) and + grpcio-tools (Python, scripts/gen_proto.py) -- each vendors its own + protoc, so no system protoc install is required. + """ + + @staticmethod + def Generate(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_stream( + request, + target, + '/dispatch.v1.ModelServer/Generate', + dispatch__pb2.GenerateRequest.SerializeToString, + dispatch__pb2.GenerateResponse.FromString, + options, + channel_credentials, + insecure, + call_credentials, + compression, + wait_for_ready, + timeout, + metadata, + _registered_method=True) diff --git a/src/dispatch/serving/colocated.py b/src/dispatch/serving/colocated.py index d3ae025..7cfc64e 100644 --- a/src/dispatch/serving/colocated.py +++ b/src/dispatch/serving/colocated.py @@ -61,3 +61,14 @@ def step(self) -> list[RequestResult]: self._active, self._decode_fn, self._clock_fn ) return already_done + completed + + def snapshot_active_tokens(self) -> dict[str, list[int]]: + """Every currently-active request's full generated-token-id list + so far, keyed by request_id -- returns copies, never internal + references. Additive to Phase 4's step()/submit(): step() + already returns *completed* RequestResults; this exposes the + still-in-flight ones too, which Phase 7's streaming model server + needs to emit each token as soon as it's produced rather than + only at completion. + """ + return {r.request_id: list(r.generated_token_ids) for r in self._active} diff --git a/src/dispatch/serving/model_server.py b/src/dispatch/serving/model_server.py new file mode 100644 index 0000000..36c4eb4 --- /dev/null +++ b/src/dispatch/serving/model_server.py @@ -0,0 +1,90 @@ +"""gRPC model server for Phase 7's router demo: streams token-by-token +generation over the shared proto/dispatch.proto contract. Two Responders +share one Servicer so the wire contract is identical whether responses +come from StubResponder (canned, no model, used by tests/dev/docker- +compose) or the real KernelResponder (scripts/gpu/phase7_kernel_responder.py, +GPU-only, used only for the real demo session) -- this module itself +never imports torch, so it stays importable and testable anywhere. +""" + +from __future__ import annotations + +import time +from collections.abc import Iterator +from concurrent import futures +from dataclasses import dataclass +from typing import Protocol + +import grpc + +from dispatch.proto import dispatch_pb2, dispatch_pb2_grpc + + +class ServerBusyError(RuntimeError): + """Raised by a Responder that can only stream one request at a time + when a second request arrives while the first is still in flight.""" + + +@dataclass(frozen=True) +class TokenEvent: + text: str + is_final: bool + t_emit: float + + +class Responder(Protocol): + def generate(self, prompt: str, max_new_tokens: int) -> Iterator[TokenEvent]: ... + + +class StubResponder: + """Canned, deterministic tokens with a fixed per-token delay. No + torch, no model weights -- used by unit tests, the cross-language + router integration test (Task 8), and docker-compose local + verification, all without a GPU. + """ + + def __init__( + self, + *, + tokens: tuple[str, ...] = ("The", " quick", " brown", " fox", " jumps"), + delay_seconds: float = 0.01, + ) -> None: + self._tokens = tokens + self._delay_seconds = delay_seconds + + def generate(self, prompt: str, max_new_tokens: int) -> Iterator[TokenEvent]: + del prompt # canned output does not depend on the prompt + for text in self._tokens[:max_new_tokens]: + time.sleep(self._delay_seconds) + yield TokenEvent(text=text, is_final=False, t_emit=time.perf_counter()) + yield TokenEvent(text="", is_final=True, t_emit=time.perf_counter()) + + +class ModelServerServicer(dispatch_pb2_grpc.ModelServerServicer): # type: ignore[misc] + # Generated base class is Any-typed (follow_imports="skip", pyproject.toml) + # -- strict mode's disallow_subclassing_any is deliberately overridden + # here, the one place this module actually needs the generated servicer. + def __init__(self, responder: Responder) -> None: + self._responder = responder + + def Generate( # noqa: N802 -- overrides a grpc_tools-generated method name + self, request: dispatch_pb2.GenerateRequest, context: grpc.ServicerContext + ) -> Iterator[dispatch_pb2.GenerateResponse]: + try: + for event in self._responder.generate(request.prompt, request.max_new_tokens): + yield dispatch_pb2.GenerateResponse( + request_id=request.request_id, + text=event.text, + is_final=event.is_final, + t_emit_unix=event.t_emit, + ) + except ServerBusyError as exc: + context.abort(grpc.StatusCode.RESOURCE_EXHAUSTED, str(exc)) + + +def serve(responder: Responder, *, port: int = 0) -> tuple[grpc.Server, int]: + server = grpc.server(futures.ThreadPoolExecutor(max_workers=4)) + dispatch_pb2_grpc.add_ModelServerServicer_to_server(ModelServerServicer(responder), server) + bound_port = server.add_insecure_port(f"[::]:{port}") + server.start() + return server, bound_port diff --git a/tests/unit/test_colocated.py b/tests/unit/test_colocated.py index 69d0fd2..af66b75 100644 --- a/tests/unit/test_colocated.py +++ b/tests/unit/test_colocated.py @@ -93,3 +93,15 @@ def test_colocated_worker_respects_batch_size_across_prefill_and_decode() -> Non b_done = worker.step() # "b" now gets its prefill turn, already done at max_new_tokens=1 assert [r.request_id for r in b_done] == ["b"] + + +def test_snapshot_active_tokens_reports_in_flight_generated_ids_so_far() -> None: + worker = ColocatedWorker(_fake_prefill_fn, _fake_decode_fn, batch_size=4) + worker.submit(Request("a", torch.tensor([[1, 2, 3]]), max_new_tokens=3)) + + worker.step() # prefilled and decoded once: 2 of 3 tokens so far, still active + + assert worker.snapshot_active_tokens() == {"a": [100, 101]} + + worker.step() # completes; no longer active + assert worker.snapshot_active_tokens() == {} diff --git a/tests/unit/test_model_server.py b/tests/unit/test_model_server.py new file mode 100644 index 0000000..cee4d51 --- /dev/null +++ b/tests/unit/test_model_server.py @@ -0,0 +1,54 @@ +"""No GPU, no torch -- StubResponder and the servicer wiring only.""" + +from __future__ import annotations + +import grpc +import pytest + +from dispatch.proto import dispatch_pb2, dispatch_pb2_grpc +from dispatch.serving.model_server import ServerBusyError, StubResponder, serve + + +def test_stub_responder_yields_requested_token_count_then_final() -> None: + responder = StubResponder(tokens=("a", "b", "c", "d"), delay_seconds=0.0) + + events = list(responder.generate("ignored", max_new_tokens=2)) + + assert [e.text for e in events] == ["a", "b", ""] + assert [e.is_final for e in events] == [False, False, True] + + +def test_serve_streams_stub_responses_over_real_grpc() -> None: + server, port = serve(StubResponder(tokens=("hi",), delay_seconds=0.0), port=0) + try: + channel = grpc.insecure_channel(f"localhost:{port}") + stub = dispatch_pb2_grpc.ModelServerStub(channel) + request = dispatch_pb2.GenerateRequest(request_id="r1", prompt="hello", max_new_tokens=1) + + responses = list(stub.Generate(request)) + + assert [r.text for r in responses] == ["hi", ""] + assert [r.is_final for r in responses] == [False, True] + assert all(r.request_id == "r1" for r in responses) + finally: + server.stop(grace=None) + + +def test_generate_aborts_with_resource_exhausted_on_server_busy_error() -> None: + class _BusyResponder: + def generate(self, prompt: str, max_new_tokens: int): # type: ignore[no-untyped-def] + raise ServerBusyError("only one stream at a time") + yield # type: ignore[unreachable] # pragma: no cover -- makes this a generator function + + server, port = serve(_BusyResponder(), port=0) + try: + channel = grpc.insecure_channel(f"localhost:{port}") + stub = dispatch_pb2_grpc.ModelServerStub(channel) + request = dispatch_pb2.GenerateRequest(request_id="r1", prompt="x", max_new_tokens=1) + + with pytest.raises(grpc.RpcError) as exc_info: + list(stub.Generate(request)) + + assert exc_info.value.code() == grpc.StatusCode.RESOURCE_EXHAUSTED + finally: + server.stop(grace=None) diff --git a/tests/unit/test_phase7_kernel_responder.py b/tests/unit/test_phase7_kernel_responder.py new file mode 100644 index 0000000..b18c066 --- /dev/null +++ b/tests/unit/test_phase7_kernel_responder.py @@ -0,0 +1,53 @@ +"""Real end-to-end model-server correctness on a real GPU: KernelResponder's +streamed token sequence matches a same-session stock-model greedy +generation on the identical prompt, exactly -- Phase 4's own bar for its +correctness gate, applied here to the streaming responder. Excluded from +CI -- no GPU runner there. +""" + +from __future__ import annotations + +import pytest +import torch +from scripts.gpu.phase7_kernel_responder import MODEL_NAME, build_kernel_responder + +from dispatch.benchmark.harness import load_model +from dispatch.kernels.integration import fix_rope_inv_freq + +pytestmark = pytest.mark.gpu +pytest.importorskip("triton", reason="triton ships Linux wheels only") +if not torch.cuda.is_available(): + pytest.skip("needs a CUDA device", allow_module_level=True) + +PROMPT = "The quick brown fox jumps over the lazy dog." +MAX_NEW_TOKENS = 8 + + +def _stock_greedy_text(prompt: str, max_new_tokens: int) -> str: + model, tokenizer = load_model( + MODEL_NAME, device="cuda", dtype=torch.bfloat16, trust_remote_code=True + ) + fix_rope_inv_freq(model) + model.eval() # type: ignore[no-untyped-call] + input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda") + generated: list[int] = [] + past = None + next_input = input_ids + with torch.no_grad(): + for _ in range(max_new_tokens): + outputs = model(input_ids=next_input, past_key_values=past, use_cache=True) + past = outputs.past_key_values + next_token = outputs.logits[:, -1, :].argmax(dim=-1, keepdim=True) + generated.append(int(next_token.item())) + next_input = next_token + return str(tokenizer.decode(generated)) + + +def test_kernel_responder_matches_same_session_stock_greedy_text() -> None: + expected_text = _stock_greedy_text(PROMPT, MAX_NEW_TOKENS) + + responder = build_kernel_responder(moe_kernel="naive") + events = list(responder.generate(PROMPT, MAX_NEW_TOKENS)) + generated_text = "".join(e.text for e in events if not e.is_final) + + assert generated_text == expected_text diff --git a/tests/unit/test_run_model_server.py b/tests/unit/test_run_model_server.py new file mode 100644 index 0000000..f5247f0 --- /dev/null +++ b/tests/unit/test_run_model_server.py @@ -0,0 +1,24 @@ +"""CPU-only: proves --responder stub needs nothing but this repo's normal +dependencies. --responder kernel is exercised only on a real GPU +(Task 9's gpu-marked test), never here. +""" + +from __future__ import annotations + +from scripts.run_model_server import build_responder, parse_args + + +def test_default_responder_is_stub() -> None: + args = parse_args([]) + + assert args.responder == "stub" + assert args.port == 50051 + + +def test_build_responder_stub_needs_no_gpu_imports() -> None: + args = parse_args(["--responder", "stub"]) + + responder = build_responder(args) + + events = list(responder.generate("hi", max_new_tokens=1)) + assert events[-1].is_final is True diff --git a/uv.lock b/uv.lock index d1dfd74..b9a3cc1 100644 --- a/uv.lock +++ b/uv.lock @@ -448,7 +448,9 @@ version = "0.1.0" source = { editable = "." } dependencies = [ { name = "accelerate" }, + { name = "grpcio" }, { name = "huggingface-hub" }, + { name = "protobuf" }, { name = "requests" }, { name = "safetensors" }, { name = "torch" }, @@ -458,17 +460,21 @@ dependencies = [ [package.dev-dependencies] dev = [ + { name = "grpcio-tools" }, { name = "mypy" }, { name = "pytest" }, { name = "pytest-cov" }, { name = "ruff" }, + { name = "types-grpcio" }, { name = "types-requests" }, ] [package.metadata] requires-dist = [ { name = "accelerate", specifier = ">=1.15.0" }, + { name = "grpcio", specifier = ">=1.84.0" }, { name = "huggingface-hub", specifier = ">=1.31.0" }, + { name = "protobuf", specifier = ">=7.36.2" }, { name = "requests", specifier = ">=2.34.2" }, { name = "safetensors", specifier = ">=0.8.0" }, { name = "torch", specifier = ">=2.14.0" }, @@ -478,10 +484,12 @@ requires-dist = [ [package.metadata.requires-dev] dev = [ + { name = "grpcio-tools", specifier = ">=1.84.0" }, { name = "mypy", specifier = ">=2.3.1" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "pytest-cov", specifier = ">=7.1.0" }, { name = "ruff", specifier = ">=0.16.7" }, + { name = "types-grpcio", specifier = ">=1.83.0.20260730" }, { name = "types-requests", specifier = ">=2.33.0.20260906" }, ] @@ -503,6 +511,110 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, ] +[[package]] +name = "grpcio" +version = "1.84.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3f/4f/4435c0aae54657258d9cfcba78598f3d9e5fe4c82ff18d78558567b90faf/grpcio-1.84.0.tar.gz", hash = "sha256:19aaf172fc2edbefccce3f6e92c5150975dbe56c45744e9e87cf72ebdf85bfbe", size = 13493876, upload-time = "2026-09-14T06:59:33.291Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0a/c1/4c9a2e0e6b0aaf02781404cad2f79211f989f2c827cf672a4a48d1604d3e/grpcio-1.84.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:b5c6f20d657ae09ae4e30d9d3a21edd13f1219d58cc6f999b9d1bb63be9c1baa", size = 6415756, upload-time = "2026-09-14T06:57:39.345Z" }, + { url = "https://files.pythonhosted.org/packages/b1/57/131e7007bdee9acb77a8dbe8a16fa9fef75f88c1695242d8ee0993ac2d3d/grpcio-1.84.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:406583b4e8fb2282ebd392e12b963e601c1f82e07125a8c2cb5b144e7e024796", size = 12339195, upload-time = "2026-09-14T06:57:42.373Z" }, + { url = "https://files.pythonhosted.org/packages/db/d1/a7b7cda98fcab9b3d2916204a872d87371158a7a34e41768f524584fb64d/grpcio-1.84.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fbdbcd06986ede3ce584083b1dc2afe6808e8943e5cf50ad11183c03aceda25a", size = 6984468, upload-time = "2026-09-14T06:57:45.035Z" }, + { url = "https://files.pythonhosted.org/packages/19/81/c5be83e3ac9416f73c4c51fe1ea9c41a0c42fc3509e3505faa46f5046abe/grpcio-1.84.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:23e6e8e8a75cff88e0a793bfd3becea03a13e2763ae90c1ff573bc19ca5b429a", size = 7749432, upload-time = "2026-09-14T06:57:47.395Z" }, + { url = "https://files.pythonhosted.org/packages/a0/bf/258cd7c0a7ed92745dc93c31666d462d05b702807a689744bd49fb833bde/grpcio-1.84.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b44f0a0fc7bc6677d38cc80bca1a32814ce6c8f200fb8b3c1a61c9d77eaefbf3", size = 7156115, upload-time = "2026-09-14T06:57:49.657Z" }, + { url = "https://files.pythonhosted.org/packages/2b/4b/7f829418dbfcf91b875e55e2973f1059a95decb4f081313416317ef04ec1/grpcio-1.84.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:210e4c32f907045eb8158273e60c6ab69a3947697df6245dbda381f26c59485b", size = 7708010, upload-time = "2026-09-14T06:57:52.496Z" }, + { url = "https://files.pythonhosted.org/packages/34/f0/9932e2fec6a04205f8bf3f8f4d2020479dcdac88feb6f93822ed31bf0eba/grpcio-1.84.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:a71d24f40b0cc6798feaa978c7411dc1135b7018e9fc0442db611c139bf58344", size = 8759980, upload-time = "2026-09-14T06:57:55.312Z" }, + { url = "https://files.pythonhosted.org/packages/2c/5c/b67407c6dbc480dfc0715f6eccdb1061e7c88d85f9a330a241d357a538c5/grpcio-1.84.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f6c972474ce691aca74e58d17625450cef153dc4760364cadeb167983ea6d589", size = 8124904, upload-time = "2026-09-14T06:57:58.569Z" }, + { url = "https://files.pythonhosted.org/packages/02/37/2bfdae2df8dfcfc0df619b628e0c7153ce703adae827243f44720322ccc1/grpcio-1.84.0-cp312-cp312-win32.whl", hash = "sha256:0d532ade4486dad9b302ffa4d4683d67561051c26d17c4023322845e9fa10140", size = 4478915, upload-time = "2026-09-14T06:58:00.714Z" }, + { url = "https://files.pythonhosted.org/packages/85/2c/309268b7b39f6deb2342f634841e105623a0b67982e8b10ec516782ff1c6/grpcio-1.84.0-cp312-cp312-win_amd64.whl", hash = "sha256:49717e857899f4136d7657bf5aded61ac479110a075438290923a4d86af7cd02", size = 5253534, upload-time = "2026-09-14T06:58:03.336Z" }, + { url = "https://files.pythonhosted.org/packages/5d/51/40f99701adb01d4e5316a2aaf13838da1a24d5c879cd8c95156d7c364454/grpcio-1.84.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:209414080da8c20af94df1395b635da52dd57b5edc9e917e1deca0dc1c4bb55e", size = 6427619, upload-time = "2026-09-14T06:58:06.025Z" }, + { url = "https://files.pythonhosted.org/packages/c5/4b/ed8e22a1237e6b2be6ef4f221d074a5b0e0dd8a0da8c944c04aea731f0eb/grpcio-1.84.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:e41c3993eee896c617dbd8a505085d28b6e84a0445ed9a1f40f95808473cf678", size = 12336549, upload-time = "2026-09-14T06:58:08.583Z" }, + { url = "https://files.pythonhosted.org/packages/d3/50/00165b05cd73f45996748ea67ce9e55d08936f2fea94a7fd8541cc2d0e54/grpcio-1.84.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fff5ef3fe1bba7d6147e5f19e01e5e122ac2c076486887ddcb8d42e663400fbe", size = 6989458, upload-time = "2026-09-14T06:58:11.884Z" }, + { url = "https://files.pythonhosted.org/packages/26/38/d0486230e684d916f97429a53041db88410e662a38f2a8d09e2d90375840/grpcio-1.84.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b8c62888c3e49debf37ad9773e3c02f77b0c1e811f8fb0962f2b6c3bbab5b97a", size = 7757778, upload-time = "2026-09-14T06:58:14.849Z" }, + { url = "https://files.pythonhosted.org/packages/da/56/548a643decb059ca244499c675ae2c13a15f523ba94592c2774bd80a13c1/grpcio-1.84.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:986e9751d416d7a6eaa2fecdac38da63153d63a4b340ba7d624889c490451500", size = 7159572, upload-time = "2026-09-14T06:58:17.87Z" }, + { url = "https://files.pythonhosted.org/packages/db/f5/42caac81a79ec680f1f7a8eaf7ca90d2f93936ce0c3a073141ba96757f77/grpcio-1.84.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5933a052946873d01a42119a05420d669bdca436aeba2d1851988ccb12b421c0", size = 7710547, upload-time = "2026-09-14T06:58:20.607Z" }, + { url = "https://files.pythonhosted.org/packages/57/a4/828ad990b2410fee0a55cc73aa1bf98eb5b911c54847374ef4f24b9e877b/grpcio-1.84.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:e094dd21f077af8194923fc263cad872eaa1802bb0156fd7e5ae18e99cd86715", size = 8761519, upload-time = "2026-09-14T06:58:23.875Z" }, + { url = "https://files.pythonhosted.org/packages/d5/a5/1f91af098919eaf5d80d5a61126ad9fae074e5190c25a3014ce1d8d0d890/grpcio-1.84.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:08735e3d08d24ab3132cf87e2e5dea8746cabcc7d676c2b0b7362f195feef9d9", size = 8121424, upload-time = "2026-09-14T06:58:27.006Z" }, + { url = "https://files.pythonhosted.org/packages/8c/8f/77fd4a7a913b636785479922349c4cb98d94d05d15652e556b3ca0df6663/grpcio-1.84.0-cp313-cp313-win32.whl", hash = "sha256:70bb4ce8be0c5606bec259cbd7152374470396413b7863a658a08c849e6b29ff", size = 4477974, upload-time = "2026-09-14T06:58:29.528Z" }, + { url = "https://files.pythonhosted.org/packages/d0/9a/1fa59ddbfc8898e5518d1447e46f771f387f0ed6132ad531395338e51a5c/grpcio-1.84.0-cp313-cp313-win_amd64.whl", hash = "sha256:b61692f0069b3eee2fc8a3a1b7f6c044df9e03fede6ce69b3ca832e1c39f26c5", size = 5255326, upload-time = "2026-09-14T06:58:31.781Z" }, + { url = "https://files.pythonhosted.org/packages/26/6f/e25ca89ca5b0b7b95464c907a5c21a77c0ac8c4ee1dca164c4dd8f153ddb/grpcio-1.84.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:026d757df86c5b7a41de8200b9a2cda454aaa5004cb0c7e3374c66eb82f61499", size = 6428207, upload-time = "2026-09-14T06:58:34.401Z" }, + { url = "https://files.pythonhosted.org/packages/cd/b4/6b76b429f3f9b901cdbc306c81364d708bc957f847a05cbd1046cd2d05d8/grpcio-1.84.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:3de427b05f244ba2c2a9bdc67e7a6731c8340811524ecc4435466549f8af1d17", size = 12342420, upload-time = "2026-09-14T06:58:37.416Z" }, + { url = "https://files.pythonhosted.org/packages/af/64/ac86d638ba7f73bee0dccb608ba551d4f63adf75151f00d2c43e46d3979e/grpcio-1.84.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e90e3bdf7b5eac005fef631adae9cafde16f922def207b80a7c46b253c18ad20", size = 6998396, upload-time = "2026-09-14T06:58:40.535Z" }, + { url = "https://files.pythonhosted.org/packages/4a/65/fa12e9ec9d7ebf8cc3e81428fa9e1ca0d30d22d546ce2baa4c64bc917cbc/grpcio-1.84.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:e88d304f094f4937bc27ec6a435e218a084168f11ec630c8d5d39b431d08d81d", size = 7757538, upload-time = "2026-09-14T06:58:43.297Z" }, + { url = "https://files.pythonhosted.org/packages/21/d7/94240c7fae121ff1f116dcf04a3b7ee0216a06832c704310363f72638d4c/grpcio-1.84.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:57dc36a5ab0e676f5f6e171de2917fd0aef73f32a9aaf23956bfe19997a30bd1", size = 7161480, upload-time = "2026-09-14T06:58:45.939Z" }, + { url = "https://files.pythonhosted.org/packages/23/c9/7033e95d4b344969818b09185721c7608b47fc2498d97b5e4eec4995dbf3/grpcio-1.84.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:5deda5b4bf62769eb98c119cca43d40e1231e34846b19db5cdea821d446a2253", size = 7720191, upload-time = "2026-09-14T06:58:48.308Z" }, + { url = "https://files.pythonhosted.org/packages/95/22/b45df2deba81d55069076859480bae7109c9eec02bce5515c799530cc2aa/grpcio-1.84.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:9bab4cf571653a8afffb83ce21aa27b51dfe629b526b7b6adec35491fe1fc2ea", size = 8762792, upload-time = "2026-09-14T06:58:51.068Z" }, + { url = "https://files.pythonhosted.org/packages/de/c4/3e1c3d6155c16b8737cc31d5b477d6cf1fc7cdd10d58320cf0ec9b446f42/grpcio-1.84.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c5559b492007dc09b4de9b95dab05f0b5e53547aad230cf07e46c7dd017a3be5", size = 8123299, upload-time = "2026-09-14T06:58:54.332Z" }, + { url = "https://files.pythonhosted.org/packages/56/fe/f4864de5b815e5ba18858771f99381a398fac14117f89ef5291ed43d3c4e/grpcio-1.84.0-cp314-cp314-win32.whl", hash = "sha256:2c024da73b296f040b8360e60bd73a659b230093684a438da0e1260f34cc724e", size = 4562560, upload-time = "2026-09-14T06:58:56.894Z" }, + { url = "https://files.pythonhosted.org/packages/44/03/640811d4d8c84f5e603995c5a9bab725223aa472cad9ca4286c3bbf1c3e3/grpcio-1.84.0-cp314-cp314-win_amd64.whl", hash = "sha256:800b7e00d92553313c0463c200087930aa78678ec1d528193aeb50906f55989b", size = 5394092, upload-time = "2026-09-14T06:58:59.61Z" }, + { url = "https://files.pythonhosted.org/packages/4a/1a/9e3d2c9f005f680f03308fa894b1db91d4ab3f0fe65ff630c69561e91e95/grpcio-1.84.0-cp315-cp315-linux_armv7l.whl", hash = "sha256:47ecf0d9b81d981f07b61bd89eced9d2582f5eaacc3aaa36ad27f81aef70a27f", size = 6428252, upload-time = "2026-09-14T06:59:02.597Z" }, + { url = "https://files.pythonhosted.org/packages/77/34/0bc9f52ebf091311651eeab3a452fb557985604a3088cb5406f4d6df85d3/grpcio-1.84.0-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:61386101ecaa096b694d0dd278caf99a56aeec78440cc17e918eef0b50f2d567", size = 12359488, upload-time = "2026-09-14T06:59:05.646Z" }, + { url = "https://files.pythonhosted.org/packages/93/0e/c31052712f241cb6ecae9c226fabd519b7f8c64a7a40bac27e9ca0405b78/grpcio-1.84.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f6d178ba6dc8e82976c184b65fddde172d054c17237993a3e083efe4f134d55b", size = 7019339, upload-time = "2026-09-14T06:59:08.76Z" }, + { url = "https://files.pythonhosted.org/packages/55/b9/b9b33ea4f1eb4cad28833cade604febf357385b5ebb0c9c7562d020e167a/grpcio-1.84.0-cp315-cp315-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:15bb76489e337fc492685c9758e2fd4d4ab516b901ad830dc5a91987decf00be", size = 7107974, upload-time = "2026-09-14T06:59:11.568Z" }, + { url = "https://files.pythonhosted.org/packages/0e/9e/799d4c45db91bbdcd8c54b3982932dbcf3d059f7ce67dca3e8540faa1ece/grpcio-1.84.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:82da34ae4f639c73ac46e521e00c0a49bf86f717b9fb1f405f133e98731e38dc", size = 7200036, upload-time = "2026-09-14T06:59:14.401Z" }, + { url = "https://files.pythonhosted.org/packages/45/dc/dcfdd13ada41aff9098f0c2c6f260eb7debbc88b84b7e5fcbd085165427d/grpcio-1.84.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:9b73836ba0e16fcbb57c31cf6cbc2907c8d8c790b83679df454b74bd15e0be04", size = 7742281, upload-time = "2026-09-14T06:59:17.348Z" }, + { url = "https://files.pythonhosted.org/packages/55/31/75eab2ec77b80804bc5e21cec99b57598e726fca6484cd3e8920a97639d5/grpcio-1.84.0-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:42959bd50dd660ffc3f2a9bec15a6da4f9aaa0dda555d59ff2d2e80b908456a8", size = 8113629, upload-time = "2026-09-14T06:59:20.584Z" }, + { url = "https://files.pythonhosted.org/packages/34/f0/fdcf6bdc1df9ca11679a1187bef8e6b81df31a2baae69497e17344f05ea3/grpcio-1.84.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:659728f20fc7a0933ed7b1945435e31014b97ab8a5a7edcbaa70da4794aeb191", size = 8152972, upload-time = "2026-09-14T06:59:24.523Z" }, + { url = "https://files.pythonhosted.org/packages/5c/cf/6720e720bfa80fcb1ace873f66724eb3c8b03bba2fa078a30c12cab3212e/grpcio-1.84.0-cp315-cp315-win32.whl", hash = "sha256:edb6f87fc60ff438557291501b3e16c7a77c3b01a52d782cf276dccc7c5dd89c", size = 4561981, upload-time = "2026-09-14T06:59:27.275Z" }, + { url = "https://files.pythonhosted.org/packages/7f/b9/69d8a709df225bc2e06e028e9465166b174c24b3da07cc72d9a5ddc63194/grpcio-1.84.0-cp315-cp315-win_amd64.whl", hash = "sha256:4119efa6519871719ad81f33bc95ab87857dcb1c5801f30a6e592f2c41164169", size = 5394757, upload-time = "2026-09-14T06:59:30.118Z" }, +] + +[[package]] +name = "grpcio-tools" +version = "1.84.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "grpcio" }, + { name = "protobuf" }, + { name = "setuptools" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/cd/db/a5dba38d7ff7711d1ad05f2b43751bd0e9f234fae7c89df1846a9de034f9/grpcio_tools-1.84.0.tar.gz", hash = "sha256:210ac5ac9803569490ec33574b7e995bc087815b00d9b777e0134cab5ed9a379", size = 6401750, upload-time = "2026-09-14T07:03:45.725Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/55/f9/ad0fc599f687227569fb0f37695ad0b66b6c515ca4b938df863f1efb1109/grpcio_tools-1.84.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:bcc3b6f41e02d77e519e6e4f114f7ab5a22815acd9b6a3c8965417df36938de3", size = 2653285, upload-time = "2026-09-14T07:01:55.305Z" }, + { url = "https://files.pythonhosted.org/packages/c0/14/cc6e137a4fea4cf35b115c22e99be391b446ac002cd8742e8d97ecff5fd4/grpcio_tools-1.84.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:9315344bed77b08c155672ab2d77513e1e8a20fd2744ab0bb72647e22bc900ac", size = 5965932, upload-time = "2026-09-14T07:01:58.431Z" }, + { url = "https://files.pythonhosted.org/packages/85/7f/602dffdd92363c6b28959502eb97badcf28176c501e3f8a26d30d6893d31/grpcio_tools-1.84.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a83ccb3f47f841d92f04fd35ed1f8b094e2a92c33ae41f26732e970e0361ff14", size = 2705432, upload-time = "2026-09-14T07:02:00.882Z" }, + { url = "https://files.pythonhosted.org/packages/55/1d/7a7bfd74fcf34d96995a05f03f9c01305f64982c6e53296363fe2d8f9911/grpcio_tools-1.84.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:9b0f4aa6fd1a72e2048742016da37cfa773394ee2b96daef6e6ef38a02eabea6", size = 3033412, upload-time = "2026-09-14T07:02:03.246Z" }, + { url = "https://files.pythonhosted.org/packages/74/18/d9fa4b43c4974094a9e720e97fc90fcba1908f8c806b36eefabf6c148d88/grpcio_tools-1.84.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc1708de3ba6cc46eff02de1aea865442418167794ca963a2cf6ea2013930146", size = 2774501, upload-time = "2026-09-14T07:02:05.329Z" }, + { url = "https://files.pythonhosted.org/packages/2b/23/0458fb717829008b4c2160e37b2e4d6d7c31ad8c8b2ee411819f682a929e/grpcio_tools-1.84.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:45e38ee36a131ad1123b2741e045656419551efe52837a4702e09daee339fbcf", size = 3229874, upload-time = "2026-09-14T07:02:08.973Z" }, + { url = "https://files.pythonhosted.org/packages/af/5a/e955b667d7fb2a8746f62342a108008798a333d3824b3d28b0c5d8034613/grpcio_tools-1.84.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:a87275c13a9e6027d164494483027b9b08a8b9f6176108cb27d4bc13c8cbc720", size = 3803163, upload-time = "2026-09-14T07:02:12.006Z" }, + { url = "https://files.pythonhosted.org/packages/08/fe/be66ea5d0962793156cc6e29da40e80a4f6ec12371564e6ba7237f8427e6/grpcio_tools-1.84.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:602453d5a04f74ead2077064249fc462bd4a93a1d23cb9f2c89cd254ac170ab7", size = 3461818, upload-time = "2026-09-14T07:02:14.099Z" }, + { url = "https://files.pythonhosted.org/packages/6e/32/9ecf7738075039348096a4cd739f0e994bbd28ef288f5bde68891739f22c/grpcio_tools-1.84.0-cp312-cp312-win32.whl", hash = "sha256:13a7e252569e3d2b3496fad5439a5f02f9f8d3454e66c5177fb3127342635c57", size = 1022489, upload-time = "2026-09-14T07:02:18.338Z" }, + { url = "https://files.pythonhosted.org/packages/e7/2c/599ce7a1d0c9c232bde27d327d605d72f07437e0e665ca8153d499557367/grpcio_tools-1.84.0-cp312-cp312-win_amd64.whl", hash = "sha256:848338ebb0f1bccaf15c09d3905a2bda5907101ca2c88ed411c4ae707615f8ac", size = 1192285, upload-time = "2026-09-14T07:02:20.492Z" }, + { url = "https://files.pythonhosted.org/packages/c1/58/688e987d701d3673e6013adeb7c3d40f5886a79bc5fe3d2c462f13447cf3/grpcio_tools-1.84.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:7a34eee4038b8a92c4d2bd56ff6a68b7debb0e80fdd9a1f2dc77895525da2bc3", size = 2652846, upload-time = "2026-09-14T07:02:23.065Z" }, + { url = "https://files.pythonhosted.org/packages/26/e9/fbc0a4d4234e7b7622bbeae07da6c4620a0baf2a9764f1d0728a12f0d2fd/grpcio_tools-1.84.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:131cc59f5612cc6d2b7f83adea3177eb85a0b52196007c555e7d80bb1d0b2b97", size = 5963562, upload-time = "2026-09-14T07:02:25.956Z" }, + { url = "https://files.pythonhosted.org/packages/66/55/41dc86dd98fb5a8b620e47d2343030243bf80411d8da372ee1554c75b667/grpcio_tools-1.84.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b8c6d43a94a22a4a4630b112366fb01bebd4e3e2ac7519ee171268c8804f4e05", size = 2705097, upload-time = "2026-09-14T07:02:28.344Z" }, + { url = "https://files.pythonhosted.org/packages/58/45/7a7c5e80a122990d61b0ff6603d3aaf1952de1faf4b2d2e7a6445d5bd894/grpcio_tools-1.84.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0130a41b311c5352fa7ab1e21da63b59db0af6559205d41c384099fc4c59f0be", size = 3033063, upload-time = "2026-09-14T07:02:30.636Z" }, + { url = "https://files.pythonhosted.org/packages/fd/a8/169ee6a6eb3225892c6be243452161b016f3bbd93fc555857dbb70d5939c/grpcio_tools-1.84.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fad2e65eed6e98ca01046bd8a47e446c7f760af89f2a958a497d61da203f9fd8", size = 2773655, upload-time = "2026-09-14T07:02:33.092Z" }, + { url = "https://files.pythonhosted.org/packages/ef/c1/41f67a4ce7c221810d515f7c384e2efbe3b547209b6686c018e9a4e54375/grpcio_tools-1.84.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:770f7c400339350e47abca5a874e5ddce9dfb313e9994af79af52482862bcb38", size = 3229788, upload-time = "2026-09-14T07:02:35.493Z" }, + { url = "https://files.pythonhosted.org/packages/bf/64/dcdfb115bfc0fa1c2659fa48c8b743cdb45d74c93d5e1a490665152ce068/grpcio_tools-1.84.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ee0609bc149bfe0b0e974ad2c3b30facc8fde16cb6c142ebaf9b5c3ffb428f97", size = 3802534, upload-time = "2026-09-14T07:02:38.172Z" }, + { url = "https://files.pythonhosted.org/packages/cd/47/2f36195b5c59cab8b8388c0e9d1383dfbc48a816db033789bd96692ab3d5/grpcio_tools-1.84.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6cf4be6baf5f932950ec24c8ccd9284b6fa46e5b275aeb4789d7fe50a8452611", size = 3461034, upload-time = "2026-09-14T07:02:40.764Z" }, + { url = "https://files.pythonhosted.org/packages/07/99/2956e79ccf82f225fe4e03835ff72bd673a95cd0e1541259b3964ffc2214/grpcio_tools-1.84.0-cp313-cp313-win32.whl", hash = "sha256:bd034763ecf817c3e97a9aeb1e5c5006389c06644973595af5f3a32a1e738c37", size = 1022115, upload-time = "2026-09-14T07:02:43.1Z" }, + { url = "https://files.pythonhosted.org/packages/71/01/89a1d1f00801f892142ed31f28a2f325ce682b02bc294abf2ea8e951d205/grpcio_tools-1.84.0-cp313-cp313-win_amd64.whl", hash = "sha256:2d1e701bd77282618e76898b7dfe05094201c679ccb0b321fff2985d0fdb2e3f", size = 1191934, upload-time = "2026-09-14T07:02:45.829Z" }, + { url = "https://files.pythonhosted.org/packages/9d/e4/a6b28ea267d0eb19d36543561bd38deb33cafa663c1b9aa78f83a3cb044f/grpcio_tools-1.84.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:b648d986c5465ea6b2df5457401bf5f27619394c889e688d6e67a453beb0db1e", size = 2652843, upload-time = "2026-09-14T07:02:48.292Z" }, + { url = "https://files.pythonhosted.org/packages/8f/64/3aa9f40934937acc73d18e241735ecf98e91e8522fb50308aa625238ee54/grpcio_tools-1.84.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:ce0d963308f1954c8265828b8aa0d37cdfebb068727fa34c700e992a64a0bdc6", size = 5963459, upload-time = "2026-09-14T07:02:50.822Z" }, + { url = "https://files.pythonhosted.org/packages/87/6b/008c3e31e7681cacdc61ec0428f569f8ea959d48ed355c80ad9c748c743e/grpcio_tools-1.84.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a425e85bd95eb107f8a51baf717e265c4c37e1d0a31d57d7da6c7a1e302ffa5e", size = 2705302, upload-time = "2026-09-14T07:02:53.058Z" }, + { url = "https://files.pythonhosted.org/packages/16/11/d8e17542a79c5a2645353e7cfe53b36a96bc3bf61d6502514eff997eac8c/grpcio_tools-1.84.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:259d3064dfced0b5439e26379f02a14cc696107fc488cb61bd0ebad09c76fa44", size = 3033047, upload-time = "2026-09-14T07:02:55.785Z" }, + { url = "https://files.pythonhosted.org/packages/50/91/1bc18ec13f07fb77e1327cbbaeabbeb5dcf1cfe17fe8def9f4cd86fec4b4/grpcio_tools-1.84.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:38b2819f6a04cb98158815f7d12cc14fd62659ce65c30b5c8c9f0efcf898cb6e", size = 2773834, upload-time = "2026-09-14T07:02:58.455Z" }, + { url = "https://files.pythonhosted.org/packages/fa/74/c6557d8928422a18d3a3e8522b67914d861026759243df28740cb1c515ce/grpcio_tools-1.84.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e4afaf1820c5a0c105538acf956cd2187b46b5438eb34f9b8e6257b674e97a67", size = 3229790, upload-time = "2026-09-14T07:03:01.779Z" }, + { url = "https://files.pythonhosted.org/packages/6a/63/02ba2c2866b6a18bd4d256aeccecfbc22908356cae9a659ba3e06fd0ee0e/grpcio_tools-1.84.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:1c76a4cee1dd0e4dfcd31e1024a69303296efdd1757173248d8c59bb64a1372b", size = 3802611, upload-time = "2026-09-14T07:03:04.581Z" }, + { url = "https://files.pythonhosted.org/packages/80/95/350e3329b12e8fe775dda19c3f5c6e260af0f9bbb1165d11b7fb10a2d89f/grpcio_tools-1.84.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c47c6f708e2bf94503e31c578c904890225fd894a3f25f907fd3832f22b1c793", size = 3461325, upload-time = "2026-09-14T07:03:08.521Z" }, + { url = "https://files.pythonhosted.org/packages/fe/22/8463063fed0ead6c901f547d1cc7bd052391ef8eff4d1aa2e0f58f4a383c/grpcio_tools-1.84.0-cp314-cp314-win32.whl", hash = "sha256:daa0e3dff4feedbdfebc9192f2d714d5e37621b2466ba07070d002d1081a6e6f", size = 1045048, upload-time = "2026-09-14T07:03:10.723Z" }, + { url = "https://files.pythonhosted.org/packages/8c/41/4f7dd965ef2d72bc55ef208d0eadd73509aa7d4957a6eea5059a256ea01d/grpcio_tools-1.84.0-cp314-cp314-win_amd64.whl", hash = "sha256:a64a86d7e32d6ff57e0d4c6d01bac1ccd5718d01c5a5042b14ad63a00bd365f7", size = 1224203, upload-time = "2026-09-14T07:03:13.512Z" }, + { url = "https://files.pythonhosted.org/packages/d0/d0/6cb3f84a4994a60929c2da21f5129cb04b675e814b3b03e4bbfc45512a2e/grpcio_tools-1.84.0-cp315-cp315-linux_armv7l.whl", hash = "sha256:ed27e0c12e687a4b15f6352e98eb794a296bdcc75fc26fbd5e2d1d6844c0bb5a", size = 2652855, upload-time = "2026-09-14T07:03:17.043Z" }, + { url = "https://files.pythonhosted.org/packages/79/bf/0525cdfd7eb41feed328c3e839da12d56be9a23d87c74e15e1b16c3b2b5a/grpcio_tools-1.84.0-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:a30b3259bbcd7aa1377e8cf5e39b30962f88ead94bd1a25c30ab8819c1163a0d", size = 5971465, upload-time = "2026-09-14T07:03:20.057Z" }, + { url = "https://files.pythonhosted.org/packages/55/e3/ef3de0d88b69a022198ce1d8ff27df970f9bbf5f0f3d65576879054718e0/grpcio_tools-1.84.0-cp315-cp315-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b268a8cc6a0ffde0388371fee57942585060e89a3904eec1a2c00765707a26b3", size = 2707417, upload-time = "2026-09-14T07:03:22.826Z" }, + { url = "https://files.pythonhosted.org/packages/0a/c4/f9c204a32145191bbcb39d487e2abcb9ed96c95e7d9e56267a1b5d378a61/grpcio_tools-1.84.0-cp315-cp315-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:feab5e59a8cbba38190196a29b878bedf3ea8e1baf94fe26fb21ca9a5b06e17c", size = 2904340, upload-time = "2026-09-14T07:03:25.8Z" }, + { url = "https://files.pythonhosted.org/packages/b7/54/a6d0aa5fc695e98c40442d9da819e2c178e3ee170182ff34922da512294b/grpcio_tools-1.84.0-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:be960444736ff4363aad257847e0b6de8798a818b7760253b043f1c2141b522b", size = 2756591, upload-time = "2026-09-14T07:03:28.57Z" }, + { url = "https://files.pythonhosted.org/packages/ca/40/343a5fb15e9b702df619f34302829ae3f02306c6e22ce98a2634e5ee51d7/grpcio_tools-1.84.0-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:5c772fff61c94a526869fbbdc1cf5d40047170c0d591609685170130de031e63", size = 3231636, upload-time = "2026-09-14T07:03:31.285Z" }, + { url = "https://files.pythonhosted.org/packages/e1/1f/32445fe2f52f3e3fa1c83131f1c051d025baa1218e38fa51d7d3dab70248/grpcio_tools-1.84.0-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:769ae9073f09b2dd322b4de5e5fffa7c2f38340cf779a21486c45946ab2b2991", size = 3684445, upload-time = "2026-09-14T07:03:34.124Z" }, + { url = "https://files.pythonhosted.org/packages/1f/49/10314e948033f1f2c5a4d68edcade8795c5fab7cb4a132ae78d3cc98e300/grpcio_tools-1.84.0-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:cee293333fc9efaa1e75d8baf0150d79007874c7bb364cffebf35346836038e8", size = 3435303, upload-time = "2026-09-14T07:03:37.42Z" }, + { url = "https://files.pythonhosted.org/packages/3a/69/69e92ff54e590236eff01ca72e9722ff31baac828c4bffcfac1f9a5e850d/grpcio_tools-1.84.0-cp315-cp315-win32.whl", hash = "sha256:65a2ae3836ffb7b035e341a6dcc81e3d8b090b0df173851715f44cdd89723b1e", size = 1045039, upload-time = "2026-09-14T07:03:39.731Z" }, + { url = "https://files.pythonhosted.org/packages/b4/8e/12b84ac60171f8401d31bf64e9cdf3e041653e3e5c79ce50cac7b2b8fa5d/grpcio_tools-1.84.0-cp315-cp315-win_amd64.whl", hash = "sha256:f28ffc8f0d2831a81239cee6b038ee3254bd7ac884fe69cc99b4ee83ff1fc1a5", size = 1224255, upload-time = "2026-09-14T07:03:42.561Z" }, +] + [[package]] name = "h11" version = "0.16.0" @@ -1132,6 +1244,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] +[[package]] +name = "protobuf" +version = "7.36.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d9/89/5b8517baa72f84a67b8a307ba953c91057af618bf40bf676f3c03551f8f0/protobuf-7.36.2.tar.gz", hash = "sha256:497d0463ff3316681da6c0b9e8d06cb465d61abce00b613ab42226175644d1bb", size = 512737, upload-time = "2026-09-17T20:07:59.326Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/72/98342feb672507c8f3a69e34b4fa8961f608edba5c1a48a6f47156d92cb5/protobuf-7.36.2-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:cbc70b17ee27e28894c7fee8bb04be1abead49e936bc70eb60052531eee2079e", size = 456039, upload-time = "2026-09-17T20:07:51.542Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ea/91fdf7c2b8bbd49cde056f00a9df6773532987e1c00fe2830b895af95c7e/protobuf-7.36.2-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:e11e1f0180583a2af89db6a2ecd9e8dc40aa6d2988ca175bfd0e6d12ea72d74e", size = 344219, upload-time = "2026-09-17T20:07:52.914Z" }, + { url = "https://files.pythonhosted.org/packages/17/ab/5fd5f8ece73fad885c5a09aa849b32d70472f954ba3a92d3bb5974ea953b/protobuf-7.36.2-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:f4fee11ec330d238b34a05c9b675f693c20415d1c5bd7d5320cc2f8a798eb9cf", size = 357223, upload-time = "2026-09-17T20:07:53.985Z" }, + { url = "https://files.pythonhosted.org/packages/db/f3/3996583dd2906297a637af12114deddf7658af6e683fedb83be061983fb5/protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:89f23aa53c24553a2416fd4fd1ec06f74fa42b14b546d8883128813f775bbfd2", size = 343223, upload-time = "2026-09-17T20:07:54.931Z" }, + { url = "https://files.pythonhosted.org/packages/fc/1b/dcc64f358fcb51811b58ae40b3d28f820725f116d86487cc20bd4b130701/protobuf-7.36.2-cp310-abi3-win32.whl", hash = "sha256:912c1221170e16c08d1f086762f563dd61ff83c18b5fa6652952dfaded66f728", size = 442998, upload-time = "2026-09-17T20:07:55.826Z" }, + { url = "https://files.pythonhosted.org/packages/8a/55/b77bda4e5e5f5971fb51b07663694690e9afdb9402136c16a522bd621cad/protobuf-7.36.2-cp310-abi3-win_amd64.whl", hash = "sha256:a300819d441e078a5608c0d3c709796bb548136058fda017ae51d425b44fd353", size = 456514, upload-time = "2026-09-17T20:07:57.188Z" }, + { url = "https://files.pythonhosted.org/packages/e4/04/d52c7016b04b6c5108f26691f9d33ec82a9b65d041f1a9c771137693d618/protobuf-7.36.2-py3-none-any.whl", hash = "sha256:bdb3a345d48db958e6ce1f18e508beb0cc981d64f24088427549c866cd039f1e", size = 179806, upload-time = "2026-09-17T20:07:58.211Z" }, +] + [[package]] name = "psutil" version = "7.2.2" @@ -1584,6 +1711,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/dc/bf/205d0004930ede8f542fb58f601526fccf4ae7626075ca1e6c4de5d3d652/typer-0.27.2-py3-none-any.whl", hash = "sha256:b3a5fc4342d5fc8fda8fc3010b1cf117e9249aab7fae800c2eff62fd3842d97d", size = 123130, upload-time = "2026-08-28T10:26:53.752Z" }, ] +[[package]] +name = "types-grpcio" +version = "1.83.0.20260730" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/70/b7/65a3ebc728ddcdfa3ac6b8872956c14193e5b95e9b43ae5d988d29f41202/types_grpcio-1.83.0.20260730.tar.gz", hash = "sha256:e2b0532526a563dbb39aa6ee39fe0961488116ac1afec3065607532c93299b64", size = 15455, upload-time = "2026-07-30T04:44:31.863Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d3/1f/6bd33dd7cfb782669c25e669abe0058d9dc4e1c25f34b8e6b97845c531ca/types_grpcio-1.83.0.20260730-py3-none-any.whl", hash = "sha256:400e0517d088e90ecfe302a96f09728cb9c72508d835fa0793fce552cd4e1477", size = 15917, upload-time = "2026-07-30T04:44:31.027Z" }, +] + [[package]] name = "types-requests" version = "2.33.0.20260906"