fix(router): seed clean peers under cache-owner pressure

[DavidBellamy]

Description

Problem

Cache-aware routing can concentrate a shared-prefix workload on one cached owner while healthy workers retain capacity. In the observed Kimi K3 incident, one public worker held 38 running and 64-102 waiting requests while almost every peer was idle. Aggregate engine-feedback admission then rejected new work before worker selection, so the fleet could not seed a clean peer and recover.

Solution

Add a default-off, partition-allowlisted distribution-headroom path that remains subordinate to fair-share scheduling. A request may seed one exact clean worker only after claiming the redeemed capacity credit's one-shot scheduler admission proof. Selection is bound to the worker identity and revision, reverified before dispatch, capped at one in-flight seed per partition, and fails closed on stale or incomplete telemetry.

Cache-aware routing continues to prefer an authoritative owner with headroom. Only when every eligible owner is full may it select one clean peer. Provisional ownership is serialized by prefix and is never published as committed ownership before a successful non-streaming backend response. Failures quarantine matching descendants for the configured cooldown; success enters a cooldown that prevents another concurrent expansion.

This PR stacks on the merged work-conserving capacity-credit contract from #60 and the worker-activation capacity fix from #57.

This first release is limited to the regular gRPC worker-selection pipeline and assumes one active SMG process. HTTP routing, PD/EPD routing, streaming requests, and backend fanout do not use distribution seeding and retain their existing behavior or fail closed.

Closes #61

Changes

• derive exact per-worker issuable headroom from complete, fresh rank telemetry without increasing scheduler capacity
• add default-off partition allowlist and hard seed cap validation
• claim and consume the move-only scheduler admission proof before any routing or lease mutation
• bind distribution leases to exact worker identity, generation, revision, model, and partition
• serialize ordinary spills and distribution seeds through a bounded prefix-protection state machine
• hand ordinary predispatch reservations to worker load accounting without a slot gap
• add terminal success commit and failure quarantine behavior for non-streaming generation
• reject streaming and fanout requests from the seed path while leaving ordinary routing unchanged
• expose Python launcher arguments and validation for the new controls

Test Plan

Reproduce the recovery scenario with two cache-aware workers: make worker A the authoritative prefix owner and fill it to its feedback limit while worker B remains clean and has headroom. Redeem one fair-share capacity credit for the exact request/model/partition, then verify that one request seeds B, a second seed is rejected at the cap, and ordinary requests cannot join B until terminal success commits ownership. Repeat with incomplete/stale telemetry, a replaced worker revision, a replayed proof, streaming/fanout input, backend failure after an early KV event, and concurrent ordinary-spill/seed acquisition; each must fail closed or preserve one-winner serialization.

Validation run on exact PR head:

cargo test --offline -p smg --lib
test result: ok. 1537 passed; 0 failed; 5 ignored

cargo +nightly fmt --all -- --check
silent success

cargo clippy --offline -p smg --lib -- -D warnings \
  -A clippy::allow-attributes -A clippy::ignored-unit-patterns
Finished successfully. The two allows cover pre-existing workspace lints outside this diff.

cargo clippy --offline -p smg --all-targets -- -D warnings \
  -A clippy::allow-attributes -A clippy::ignored-unit-patterns \
  -A clippy::needless-raw-string-hashes -A clippy::disallowed-methods \
  -A clippy::absolute-paths
Finished successfully. Every allowed finding is in an unchanged file on prod.

make python-dev JOBS=8
Built and editable-installed smg 1.9.0 successfully.

pytest -q bindings/python/tests
251 passed; 4 skipped; 88.23% coverage

pytest -q bindings/python/tests/test_arg_parser.py
37 passed; 1 skipped

ruff check e2e_test/ bindings/python/ scripts/
All checks passed

mypy bindings/python/ --config-file mypy.ini
Success: no issues found in 16 source files

The broader cargo test --offline --workspace gate reaches eight pre-existing compile errors in unchanged protocol-builder tests (ChatChoice, ChatCompletionMessage, ChatMessageDelta, and ChatStreamChoice initializers omit the existing other field). This PR changes no protocol or workspace manifest file. The changed SMG library and Python binding suites above are green. Full-tree Ruff formatting likewise reports one unchanged prod issue in test_router_config.py; both Python files changed by this PR pass Ruff formatting individually.

GitHub CodeQL for Actions, Go, Python, and Rust, plus title/commit-message validation, pass on exact head aa51d482c9ca8af8c09967f98378260545c214cc.

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings is blocked by documented pre-existing prod lints and the local OpenCV dependency; the changed package passes strict lib Clippy and all-target Clippy with only unchanged-prod lint categories allowed
• Documentation not required because the feature is default-off and launcher help/config validation describe every new argument
• Optional: Please join us on Slack #sig-smg to discuss, review, and merge PRs