Bug Description
Enforced least-load cache credit can change nearly every worker choice even when the certified cache saving is small. The selector replaces the legacy least-load base score with a different remaining-decode occupancy estimate, so enforcement can form a concentrated worker queue instead of acting as a bounded cache preference.
Steps to Reproduce
- Configure a model with least_load, gRPC workers, and least_load cache mode enforce.
- Use the production calibration of 200 token/s fallback generation throughput, 8000 token/s prefill throughput, and 2048 mean remaining decode tokens.
- Send sustained scalar streaming Chat Completions traffic with reusable prefixes.
- Compare enforce dispatch with the legacy least-load choice and observe worker queue balance.
Expected Behavior
Cache ownership should influence routing only when the certified prefill saving is greater than the extra load cost. A materially busier cached owner should not beat the ordinary least-load worker. Missing load data for one worker should exclude that worker from cache consideration without disabling cache-aware decisions for the whole fleet.
Actual Behavior
The hybrid workload model dominates the cache credit and changes almost every decision. Under enforcement, those choices alter future cache placement and load, creating feedback that shadow mode cannot fully reproduce. The selector also falls back fleet-wide when any healthy worker lacks a usable load sample.
Component
model-gateway (core routing)
Routing Policy (if applicable)
N/A
Connection Mode
gRPC
Configuration
policy: least_load
least_load_cache_mode: enforce
least_load_default_throughput: 200
least_load_cache_prefill_throughput: 8000
least_load_mean_remaining_decode_tokens: 2048
Logs / Error Output
Environment
SMG prod branch 31cad7d
Production: Linux container under Slurm
Validation: macOS arm64 Rust workspace
Deployment Environment
Other
Streaming Context
Additional Context
Proposed correction:
- Preserve the existing least-load score as the base.
- Prefer a certified cached owner only when its saved prefill time covers its additional least-load cost.
- Keep missing or invalid load handling local to the affected worker.
- Add reason-labelled fallback telemetry.
- Preserve off, shadow, request eligibility, KV certification, and legacy fallback behavior.
Related to #68 and LLM360/comet#651.
Pre-submission Checklist
Bug Description
Enforced least-load cache credit can change nearly every worker choice even when the certified cache saving is small. The selector replaces the legacy least-load base score with a different remaining-decode occupancy estimate, so enforcement can form a concentrated worker queue instead of acting as a bounded cache preference.
Steps to Reproduce
Expected Behavior
Cache ownership should influence routing only when the certified prefill saving is greater than the extra load cost. A materially busier cached owner should not beat the ordinary least-load worker. Missing load data for one worker should exclude that worker from cache consideration without disabling cache-aware decisions for the whole fleet.
Actual Behavior
The hybrid workload model dominates the cache credit and changes almost every decision. Under enforcement, those choices alter future cache placement and load, creating feedback that shadow mode cannot fully reproduce. The selector also falls back fleet-wide when any healthy worker lacks a usable load sample.
Component
model-gateway (core routing)
Routing Policy (if applicable)
N/A
Connection Mode
gRPC
Configuration
Logs / Error Output
Environment
SMG prod branch 31cad7d
Production: Linux container under Slurm
Validation: macOS arm64 Rust workspace
Deployment Environment
Other
Streaming Context
Additional Context
Proposed correction:
Related to #68 and LLM360/comet#651.
Pre-submission Checklist