[Performance]: prevent cache affinity from creating hot engines

[DavidBellamy]

Performance issue

Cache-aware routing can concentrate a popular prefix on one authoritative owner even when most healthy engines are idle. In a 51-engine Kimi K3 deployment, one engine held 109 waiting and 38 running requests while 49 engines were idle. Adaptive admission rejected approximately 760 requests per minute and fleet generation throughput collapsed even though distribution-seed selections were occurring.

Expected behavior: cache reuse should reduce prefill work, but a cached owner should lose when its projected queue and decode delay exceeds the cost of recomputing the missing prefix on an idle peer.

Proposed solution

Extend the existing least_load projected-time scorer with authoritative per-worker KV overlap:

projected_seconds =
    waiting_and_reserved_work / throughput
  + active_decode_work / throughput
  + uncached_prefill_tokens / prefill_throughput
  + kv_pressure_barrier

Use exact worker KV store/eviction events only. Preserve atomic decision-time reservations so concurrent sibling requests cannot incast onto the same apparently idle worker. Add a configurable hard escape to the load-only winner, off|shadow|enforce modes, bounded runtime knobs, and conservative load-only fallback for stale or incomplete state. Ordinary routed misses create new prefix owners when authoritative KV events arrive; this first version does not add a background replication controller.

Reproduction profile

• 51 healthy gRPC engines
• One exact cached owner with 109 waiting and 38 running requests
• 49 idle engines
• Direct scalar streaming Chat Completions with a shared long prefix
• Existing cache-aware routing and cap-one distribution seeding

Acceptance criteria

• A deterministic 51-worker regression sends new hot-prefix work to idle peers while the cached owner is overloaded.
• Cache affinity remains preferred when its saved prefill time exceeds the projected load penalty.
• Simultaneous identical requests are spread using atomic decision-time reservations.
• Active decode occupancy contributes to projected cost.
• Missing, stale, incomplete, or invalid telemetry falls back to existing load-only behavior.
• off preserves current least_load behavior.
• shadow records counterfactual choices without changing dispatch.
• enforce uses the projected cache-and-load decision with a hard load escape.
• Existing cache-aware distribution seeding remains orthogonal and disabled for this policy.
• CPU simulation covers hot-prefix bursts, multiple prefixes, KV-event delay/loss, disconnects, and worker restart.

Environment: SMG prod 7c01da7d43b4572749727a9fa4c77524f6a4efbc, native gRPC SGLang workers, bare-metal Slurm deployment.