fix(cache): include scheduler occupancy in owner pressure

[DavidBellamy]

Description

Problem

Cache-aware engine-load routing currently defines pressure as the larger of KV token usage and utilization. A Kimi K3 engine can therefore remain a suitable cached owner while its scheduler is full and has a large waiting queue, so cache affinity keeps feeding that owner while other healthy engines stay idle.

On the observed M2 workload, hot owners reached the 37-request cap with up to 116 waiting requests while most of the 50-engine pool was idle. This PR only changes routing pressure. It does not change adaptive-admission ordering or enable the feature in production.

Solution

Include scheduler occupancy in per-worker pressure when every healthy candidate reports a positive max-running cap for every DP rank:

pressure = max(token_usage, utilization, (running + waiting) / max_running)

The existing 90% high watermark, 10-point owner slack, replication ceiling, cooldown, and provisional-owner coalescing remain unchanged. If any candidate has missing, partial, or stale cap telemetry, the entire decision falls back to the existing backend pressure. This avoids false spills from incomplete multi-DP caps and from SGLang Prometheus fallback counts that may be duplicated across TP ranks.

Refs #11.

Changes

• Add scheduler occupancy to the cache-aware engine pressure plan.
• Sum complete request counts and caps across DP ranks.
• Fail open fleet-wide to backend pressure when scheduler caps are incomplete.
• Add regression coverage for below/above-watermark behavior, multi-DP aggregation, partial telemetry, TP-duplicated fallback counts, existing suitable owners, and saturated spill candidates.

Test Plan

Local verification:

• cargo +nightly fmt --all -- --check
• git diff --check
• cargo test -p smg policies::cache_aware --lib (53 passed)
• cargo test -p smg --lib (1,477 passed, 5 ignored)
• Changed-package clippy with unrelated Rust 1.94 baseline lints allowed

The repository-wide cargo clippy --all-targets --all-features -- -D warnings remains blocked on the unchanged base by existing Rust 1.94 lint failures and this Mac's missing pkg-config/OpenCV dependency. Repository-wide cargo test also fails compiling existing openai-protocol builder tests that omit the other field. Neither failure touches this diff.

Before production use, run an isolated M2 routing canary on the same TP16 engines and growing-prefix trace:

1. Compare engine-load routing disabled versus enabled with adaptive admission in shadow mode.
2. Promote only the passing arm to admission enforcement.
3. Require at least 3 of 4 engines active, bounded prefix-owner replication, full load/cap telemetry coverage, no increase in terminal errors or retractions, and no more than a 10 percentage-point cache-reuse loss.
4. Use successful output tokens per node and completed rollouts per hour as the primary gates. The target is at least 2x over the current owner-trapped baseline.

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes (blocked by unchanged base and local OpenCV tooling, detailed above)
• (Optional) Documentation updated
• (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

[DavidBellamy]

Canary update from the isolated two-engine Kimi K3 run:

• A valid production-SMG control completed at an exact 76-slot scheduler capacity: 228/228 turns, 228,000 output tokens, 104.969 output tok/s/node, zero terminal failures, 408s, retractions, OOMs, or restarts.
• Routing stayed 100:0 for the full trace. The cached owner handled every selection while the second healthy engine stayed idle.
• Control cache reuse was 98.957%. Admission generated 50,322 retried 429s, exactly 47,999 running_limit plus 2,323 engine_waiting decisions.
• The patched owner-pressure arm was not measured. Its gateway startup exposed a separate worker lifecycle race: both 38-slot workers were healthy and ready, but aggregate capacity remained 38 instead of 76. Strict pre-traffic gates aborted the arm.

The capacity race is tracked in #56 and fixed with deterministic regressions in draft PR #57. This PR should remain draft until a combined canary image includes #57 and the same 76-way A/B completes. The balanced-engine estimate is promising, but there is no treatment throughput claim yet.

[DavidBellamy]

Canary correction (2026-08-07): the first isolated M1 A/B did not execute this change. The control and nominal treatment images contained the same compiled smg_rs.abi3.so (sha256:0f93632c7a0f905c4b52edc83383c4949fd69ba063ac6309cb7d936de6455f23). A shared CARGO_TARGET_DIR reused the control Rust artifact while wheel metadata changed, so distinct wheel/image hashes were misleading.

The later legacy /get_load fallback hypothesis was also disproved: the canary logs show successful rich /v1/loads polling, and the pinned SGLang build populates max_running_requests. I reverted that unrelated fallback in 11c6a2f6; the eight focused scheduler-pressure tests pass.

I am rebuilding the original treatment with a source-specific Cargo target and will require the embedded .so digest to differ from control both before installation and after sqsh export. Until that passes and the A/B is rerun, the cancelled job is packaging-negative evidence only, not a behavioral validation of this PR.

[DavidBellamy]

Corrected isolated M1 canary result (job 1191549)

The earlier A/B artifact was invalid because its nominal treatment and control images loaded the same compiled smg_rs binary. I rebuilt the treatment with a unique Cargo target and required wheel, installed-container, exported-image, and fresh-container hashes to agree before launch.

Runtime provenance in this run:

• treatment: this PR plus fix(worker): emit activation status events #57, loaded smg_rs SHA-256 8b4ae47918b8ec304673bcd2aadff55fe9ec52ca0ae205f8a07b2fc72ef6fa02
• control: fix(worker): emit activation status events #57 only, loaded smg_rs SHA-256 0f93632c7a0f905c4b52edc83383c4949fd69ba063ac6309cb7d936de6455f23
• both runtime-loaded hashes matched their embedded provenance and differed from each other

Workload and isolation:

• two independent TP16/EP16 Kimi K3 engines, two H200 nodes per engine, plus one isolated CPU/router node
• 38 running-request slots per engine, 76 trajectories offered concurrently
• 76 growing-prefix trajectories x 3 turns x 1,000 output tokens
• 16,078,104 submitted prompt tokens / 228,000 output tokens, ratio 70.518:1
• both arms were forced to the same pre-release state: cached owner 35 running, peer 0 running, both 0 waiting, then the remaining 41 trajectories were released
• adaptive admission remained shadow-only, so this isolates cache-aware routing pressure

Matched result:

metric	#57 control	#55 + #57 treatment	change
successful output tok/s, fleet	438.090	730.382	+66.72%
successful output tok/s per 4 serving nodes	109.523	182.595	1.667x
successful output tok/s per all 5 allocated nodes	87.618	146.076	1.667x
whole-arm wall time	520.441 s	312.166 s	-40.02%
completed trajectories/hour	525.708	876.458	+66.72%
latency p50 / p95	162.462 / 170.627 s	85.184 / 115.135 s	lower
fleet running max	38	73	+35
generation share	0% / 100%	51.32% / 48.68%	balanced
cached prompt fraction	98.29%	98.58%	+0.29 pp

Both arms completed 228/228 turns with zero terminal failures, 429 retries, retractions, or CUDA OOMs. The treatment made 10 bounded owner_pressure_spill decisions, 207 cached-owner holds, and 11 conservative no_safe_spill_hold decisions. Control made 228/228 cached-owner holds, reached 38 running plus up to 40 queued on one engine, and left the peer completely idle.

The fail-closed analyzer intentionally exited nonzero on three caveats, so I am keeping this PR in draft and am not recommending production activation yet:

1. Exact output-token hashes matched for 156/228 turns (52/76 trajectories) and differed for 72/228 turns under temperature 0. Prompt-token counts and the common prefix were identical. The deployed SGLang profile does not enable batch-invariant deterministic inference, so changed batch shapes can change greedy output, but this still needs a fixed-trace or quality-parity canary before rollout.
2. The analyzer classified the 11 no_safe_spill_hold decisions as an unsafe-branch failure. The code path is conservative: when no unpressured replica exists, it preserves affinity on the least-pressured owner rather than creating another owner. It produced no retractions here, but admission enforcement should be tested with it.
3. Each engine reached near-maximum concurrency (38 and 35), fleet max was 73, and simultaneous high concurrency persisted for 17 aligned samples. One engine's nonzero-running p50 was 24 rather than the analyzer's 30 threshold; the other was 31.

The complete analyzer is at /lustrefs/users/david.bellamy/k3-owner-pressure-routing/runs/1191549/analysis.json, SHA-256 8aab1f496308fcb0b0929ddba1564f54951a4ff46233933a141f2cce306e2cf0. All five nodes returned idle and production Comet was not changed.

Next gates before merge/activation: code-owner review, fixed-trace or task-quality parity, then a multi-engine canary with admission enforcement rather than shadow mode.

[DavidBellamy]

Enforced-admission follow-up (M1 job 1191552)

The 182.595 successful output tok/s/node shadow result above is real, but this PR is not sufficient for the current production admission path by itself.

Source audit shows that HTTP adaptive admission runs before route_typed_request_once and worker selection. In enforce mode, tracker.should_reject() returns 429 immediately, so cache-aware selection and owner_pressure_spill never run for that attempt.

The five-node M1 follow-up reproduced the exact live state in two consecutive one-second gateway samples using the provenance-verified treatment binary:

• cached owner: 38 running, 36 waiting, hard cap 38
• peer: 0 running, 0 waiting, hard cap 38
• load coverage: 1.0
• max-running coverage: 1.0
• aggregate hard cap: 76
• aggregate mean/max token usage: 0.06 / 0.12
• waiting allowance: 0

Snapshot artifacts:

• /lustrefs/users/david.bellamy/k3-enforced-admission-repro/runs/1191552/owner-pressure-treatment-hot-owner-confirmation-1.prom, SHA-256 0f803b4ac7c49af3a90383fa92a5b70e573baa68a488827aa491ae43521107df
• /lustrefs/users/david.bellamy/k3-enforced-admission-repro/runs/1191552/owner-pressure-treatment-hot-owner-confirmation-2.prom, SHA-256 eeecda300e4f1437b7e48e0ee6ef158499ff386234db911aa8cbb3a0dea1bbba

At that state the current EngineFeedback decision is deterministically engine_waiting: token pressure is below threshold, waiting is 36 > 0, and that branch precedes running_limit. Enforced HTTP handling then returns 429 before smg_worker_selection_total or any cache-policy decision can increment. This explains the production loop in which an idle peer cannot acquire its first prefix owner.

I am deliberately classifying this as source-plus-live-state evidence, not an endpoint 429 measurement. The one-shot probe wrapper rejected a zero-retry CLI argument and the fail-closed cleanup stopped the gateway before a replacement raw probe could be sent. The harness is corrected for a future rerun; no production process was touched, and all five nodes returned idle.

I also prototyped a distribution-headroom admission rule in an unpublished worktree. It bypasses both the waiter gate and a skewed learned knee only with complete per-replica load/cap telemetry, a distinct clean peer, and aggregate hard-cap headroom. Formatting/diff checks and 24 targeted Rust tests pass, including hot-owner/idle-peer, learned-knee, full-saturation, and incomplete-telemetry cases. It is not merge-ready: the admission controller cannot yet prove owner-pressure routing is enabled, and a one-request relaxation can close again as soon as the waiter clears. The safe design needs an explicit routing-policy opt-in plus a bounded one-in-flight seeding lease/cooldown.

This PR should remain draft. Required next gate: couple distribution-aware admission to owner-pressure routing, then rerun the corrected enforced canary before code-owner review and activation.

[DavidBellamy]

Safety hardening checkpoint after the 1191549 result

The 182.595 successful output tok/s per serving node result remains valid for the isolated two-engine warm shared-prefix canary described above. It is not a production-readiness result.

A recovered stack atop the exact PR #59 head now adds three fail-closed hardenings:

• malformed, negative, incomplete, or rank-inconsistent engine pressure telemetry cannot contribute distribution headroom;
• a seeded route publishes cache ownership only after a clean terminal stream with recognized usage and terminal markers, not after the first 2xx frame;
• enabled distribution headroom rejects max_inflight other than 1 and is explicitly labeled isolated-singleton-canary-only.

Local validation on the recovered exact tree:

• cargo +nightly fmt --all -- --check: pass
• cargo test -p smg --lib: 1,554 passed, 5 ignored
• cargo clippy -p smg --lib --tests: pass with pre-existing warnings only
• make python-dev: pass
• Python binding tests: 252 passed, 4 skipped

Production remains NO-GO. The seed permit is process-local, so overlapping blue/green gateways or any other multiple-router topology can each mint one permit against the same workers. The next production prerequisite is a canonical shared per-partition lease acquired before the local permit and held through the seeded request lifetime. The implementation must then be split into independently reviewed capacity, routing-policy, and HTTP integration changes.

No new commit, branch push, image, deployment, or production mutation was made for this checkpoint. The current hardened code is preserved only as an unapplied local patch against b75db35, SHA-256 d36fed8b722944eefa5d1f2be6bfccdba70ff5c465db6785565af483343849fb.

[DavidBellamy]

Safety correction to my previous checkpoint: a subsequent audit found the local distribution-headroom prototype is NO-GO, even for the earlier singleton canary shape. max_inflight=1 bounded only the explicit permit; ordinary cache-aware requests could (1) join the pending seed target and (2) use the pre-existing provisional owner-pressure spill path without that permit. The measured 182.6 successful output tok/s/node remains a valid result for the tested treatment, but it may partly depend on this uncontrolled fanout and therefore does not validate safe code. No production deployment occurred. I am removing pending joins and testing an exclusive additional-owner-expansion guard with exact partition scope, fail-closed refund, and a process-local selection/lease/commit fence. Until that passes a fresh rerun, treat the routing result as diagnostic evidence only. Multiple routers, rolling/blue-green activation, mesh, eviction, and existing policy state remain explicitly unsupported. Fair-share PRs should remain draft/disabled and must not depend on this prototype.

[DavidBellamy]

Hardened-prototype checkpoint (2026-08-08)

The corrected owner-pressure design is now frozen locally but remains unpublished, disabled, and production NO-GO.

• Base: exact SMG PR feat(admission): wire scheduler capacity credits #59 head b75db356b74a3d6e989726b443c8ce75a66c4b99
• Scope: 19 files, 5,808 insertions / 237 deletions
• Frozen patch SHA-256: 765c83fe25f52517b25e475e3f69e06eb37881e340ec873386bf01a63bc3e180
• Exact current Rust library suite: 1,561 passed, 5 ignored, 0 failed
• Focused current tests: distribution guard 5, seed isolation 1, guarded no-internal-retry 1, distribution headroom 7, all passed
• Python bindings: 252 passed, 4 skipped
• Nightly format and git diff --check: passed

The hardened proof boundary is narrow: one fresh singleton SMG process, one exact worker partition, controlled generation traffic, stable worker URLs, cold routing state, mesh disabled, owner-tree eviction disabled, trusted partition header overwritten by Comet, and max_inflight=1. Ordinary traffic may establish/follow the first cold owner but cannot expand an already-owned prefix; only the exact leased route can add a clean-peer owner after terminal backend success. Guarded requests do not retry internally.

A separate --bin smg distribution_headroom target compile did not finish because the local disk filled while writing Rust metadata; it reported no source or assertion failure. A clean executable image build is therefore still required before even the narrow rerun. Multiple routers, blue/green overlap, existing routing state, mesh, eviction, and production remain unsupported because the permit/mutex is process-local.

The fair-share boundary remains unchanged: PR #59 is the generic scheduler-slot lease, Comet draft PR smg-project#596 owns fair-share ordering, and this throughput-side design owns per-worker issuable headroom plus target-constrained selection, verification, and refund/fail-closed. Neither dependency should be activated from this checkpoint.

The 182.595 output tok/s/node result remains diagnostic only until the hardened route is rebuilt and rerun. No production mutation occurred.

[DavidBellamy]

Final exact-source validation checkpoint (2026-08-08)

The hardened prototype remains unpublished as a PR and disabled by default.

• Frozen source: 52d92d62, based on exact PR feat(admission): wire scheduler capacity credits #59 head b75db356.
• Review artifact SHA-256: 9e1b802c7802eb1a5464a9599c316852696b2f4bd917b7eeb2b14bd0be096b1d.
• Workflow-only head: 04b636d6.
• Hosted focused job 93121095840 passed all 25 selected tests on the exact source: 3 non-stream completion-proof tests, 9 streaming proof/error/cancellation tests, 6 exclusive owner-guard tests, and 7 headroom/configuration tests. The overall workflow was then force-cancelled only to release its still-queued internal CPU jobs.
• cargo check -p smg --lib, cargo check -p smg --tests, formatting, whitespace, 252 Python tests, and 38 focused parser tests also pass. The earlier full Rust library run was 1,561 passed / 5 ignored, but predates the final amendments.

Independent review found no fair-share bypass. The special path requires an exact one-shot PR #59 authorization, then independently constrains and revalidates worker URL, generation, revision, telemetry, cache policy, and load transfer before dispatch. Comet PR smg-project#596 therefore remains the upstream fair-share/order owner; this route is subordinate to it.

Production is still NO-GO. In addition to process-local permits and missing backend-KV eviction/tombstones, review found that ordinary cold bootstrap records an owner before backend success, and standard OpenAI streams may omit the usage proof required for owner commit. A narrow canary is only conditionally eligible after a clean image build, with exactly one gateway/no overlap, a prewarmed proved owner, non-stream requests, one partition/model, max_inflight=1, fresh complete telemetry, stable workers, mesh/routing-key override/tree eviction disabled, and no internal retries.

The correct publication path is the documented decomposed stack, not this 6,364-line prototype as one PR. The 182.595 output tok/s/node result remains diagnostic evidence only until the exclusive design is rebuilt and rerun. No production mutation occurred.