Repository navigation
Phase 7: productionization (Rust router, Docker, K8s, observability) - #9
Merged
Merged
Conversation
Fills in the system design's Phase 7 sketch (design doc SS8, ADR-0003) with concrete scope decided in brainstorming: real model on a real rented GPU behind the router, a local kind cluster (not cloud) for the K8s demo, SSH-tunnel connectivity matching Phase 5b/6's existing pattern, plain K8s manifests over Helm, and a $5 GPU cap.
Discussed explicitly: a standing public GPU-backed endpoint was considered and declined (real ongoing cost, contradicts this project's own cost-discipline rules). A short screen recording of the live demo is the recruiter-facing artifact instead, embedded in the README refresh at phase completion.
14 tasks: shared gRPC contract, a combined lib+bin Rust router crate (admission queue, Prometheus metrics, tonic client, axum HTTP surface), a Python model server behind a Responder protocol (StubResponder for everything but the paid session, KernelResponder wrapping Phase 4's ColocatedWorker + Phase 6's naive kernel for it), Docker, K8s manifests verified via a local kind rehearsal before the one real GPU session, and a Rust-aware make check. All package/crate versions checked live against pypi.org/crates.io/Docker Hub/GitHub releases, not guessed.
tonic 0.14 split prost integration out of tonic-build into tonic-prost-build (codegen) + tonic-prost (the runtime ProstCodec) -- tonic_build::compile_protos no longer exists, found live when the plan's originally-written build.rs failed to compile. Switched to tonic_prost_build::compile_protos; tonic::async_trait (used by later tasks' mock servers) is unaffected and still re-exported.
…cer, CLI) Three real mypy fixes beyond the plan's original code: generated pb2 message classes need --pyi_out to be visible to mypy at all; follow_imports="skip" (not ignore_errors) is the override that actually suppresses "untyped call" errors at importers, matching the existing triton/deep_ep/vllm/sglang pattern; and that override needs a matching top-level exclude since mypy walks generated files directly as root targets regardless of per-module overrides for imports.
Found and fixed a real process-lifecycle bug: uv run python forks rather than exec's on this machine, so killing only the immediate child left the actual model server running as an orphan reparented to launchd. Fixed with process_group(0) plus a negative-pid group kill in Drop, and Stdio::piped() instead of inherit() to close the fd-inheritance hole that let the orphan hang an unrelated piped cargo test invocation for 27+ minutes.
…e 6's kernel snapshot_active_tokens() is a small, additive change to Phase 4's ColocatedWorker (returns copies of each active request's generated token ids so far, never internal references) -- step() already returns completed RequestResults; this exposes in-flight ones too, which streaming needs to emit tokens as they're produced. The gpu-marked correctness test compares against a same-session stock generation rather than a stored Phase 6 reference file, matching this repo's actual established gate pattern more closely than the design doc's shorthand wording.
…ompose Three real build/runtime bugs found only by actually running the built images: missing README.md (uv sync's build step needs it), protobuf missing at runtime (only transitively available via the dev-only grpcio-tools), and uv run without --no-dev reinstalling dev tools into the container on every start. A fourth surfaced once the stack ran: the router raced the model server's startup and exited on the first failed connect -- fixed with a bounded retry loop plus restart: on-failure as defense in depth.
…he stub Full rehearsal, no GPU: kind cluster up, both images built and loaded, manifests applied, both rollouts succeeded, a real request through the router pod -> ExternalName Service -> host.docker.internal reached the local stub server and returned real text, Prometheus confirmed it was actually scraping the router, Grafana came up healthy, clean teardown. Router liveness/readiness probes needed periodSeconds/failureThreshold tolerant of the retry loop's up-to-30s startup window (Task 10), found before the rehearsal rather than by it. Also fixes a `## Next step` header in docs/STATUS.md that got dropped in an earlier task's edit -- the surrounding paragraph survived but the heading line itself was lost; restored to match main.
…gn's phase table Task 13: real paid GPU session (RunPod A40, $6.8563 of a $5 cap, disclosed) proving the router/model-server/kind stack against real GPU inference over an SSH tunnel. Found and fixed two real bugs: a sys.path issue running the model server as a script instead of a module, and a genuine defect in the model's shipped tokenizer (wired to the wrong pre-tokenizer/decoder pair for its actual byte-level BPE vocab) that silently dropped every space in generated text -- fixed in scripts/gpu/phase7_kernel_responder.py, not a pod-local shim. Task 14: findings doc, runbook, STATUS.md/README/CLAUDE.md refreshed. This closes the system design's 8-phase plan, Phase 0 through 7.
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
make checkgreen (Rust and Python gates)gpu-marked correctness test passed on the real rented GPU (Task 13, Step 4)