[Feature]: make cache-aware routing engine-pressure aware

[DavidBellamy]

Problem Statement

The cache_aware policy measures worker load with a gateway-local active request count. A short request and a very long prompt or reasoning decode each add one, even though their compute and KV-cache pressure differ by orders of magnitude. Cache affinity can therefore route into a hot worker.

Proposed Solution

Feed existing WorkerLoadResponse snapshots into cache_aware routing and add request token-cost reservations. Prefer cache locality only when the cache candidate remains within configurable engine-pressure bounds. Support shadow decisions, telemetry freshness checks, and consistent fallback when engine load is unavailable.

Alternatives Considered

Using power_of_two discards cache affinity. Using only token_usage reacts after engine polling and does not protect against bursts between polls. Moving routing into the caller duplicates gateway behavior.

Feature Area

Routing & Load Balancing

Affected Components

model-gateway, protocols, python-bindings

Use Case

Shared inference replicas receive heterogeneous prompt and output lengths. Routing must account for projected KV occupancy and observed queue or scheduler pressure while retaining prefix-cache benefits when safe.

Priority to You

Critical / Blocking

Contribution

• I am willing to contribute this feature (with guidance)
• I am willing to help test this feature
• I can provide more detailed requirements if needed

Acceptance Criteria

• cache_aware consumes fresh engine load snapshots.
• request reservations cover the interval between selection and engine observation.
• cache affinity is bounded by configurable pressure limits.
• stale or partial telemetry falls back consistently.
• shadow mode records current and proposed selections.
• unit and integration tests cover heterogeneous requests, overload, and stream cleanup.

[DavidBellamy]

The owner-pressure routing work in draft PR #55 reached 182.595 successful output tok/s per serving node at 73/76 running requests in a fixed two-engine M1 A/B, but an enforced-admission follow-up found a separate pre-routing blocker.

HTTP adaptive admission runs before worker selection. M1 job 1191552 captured two consecutive full-coverage samples with the cached owner at 38 running / 36 waiting, its peer at 0 / 0, aggregate hard cap 76, and mean/max token usage 0.06 / 0.12. With a zero waiting allowance, current EngineFeedback deterministically selects engine_waiting and enforced HTTP returns 429 before cache-aware selection can seed the idle peer.

Full evidence, snapshot hashes, and the endpoint-measurement caveat are in #55 (comment).

An unpublished prototype now recognizes complete per-replica distribution headroom and passes 24 targeted tests, including a skewed learned-knee case. It still needs an explicit owner-pressure-routing opt-in and a bounded seeding lease/cooldown before it is safe to publish: a generic bypass could feed a legacy sticky owner, and a one-shot bypass can close again when the first spill clears the waiter.

Proposed acceptance-criteria addition: enforced admission must allow a bounded first-owner seed to a fully observed idle peer while retaining the hard aggregate engine cap, then reject when all replicas are saturated or telemetry is incomplete.