A discrete-event simulator for request routing across multiple LLM serving instances. Compares four routing strategies under variable arrival rate, prefix sharing, and workload skew.
For full architecture, design decisions, and methodology see design.md.
When serving LLMs across multiple instances, a load balancer must decide where to send each request. Three forces pull in different directions:
Load balance: spread requests evenly to avoid queueing on any one instance.
Cache locality: route same-prefix requests to the same instance to reuse its KV cache. A cache hit saves the cost of re-running the shared prefix.
Queue cost: routing to a cached instance builds that instance's queue, adding wait time that may exceed the cache saving.
round_robin Rotate through instances, ignoring state. least_load Route to instance with shortest queue. prefix_aware Route to instance with prefix cached. Fall back to least-load. hybrid Route to cached instance only if queue depth is below threshold.
All strategies, operational regime (TTFT < 2s):
round_robin: TTFT=148.0ms hit=0.362 imbalance=0.007 least_load: TTFT=148.0ms hit=0.362 imbalance=0.007 prefix_aware: TTFT=204.1ms hit=0.439 imbalance=0.173 hybrid(thr=1): TTFT=140.4ms hit=0.453 imbalance=0.007
Key scenario: zipf=2.0, prefix_shared_frac=0.9 (skewed, high sharing):
round_robin: TTFT=110.9ms hit=0.795 imbalance=0.007 least_load: TTFT=110.9ms hit=0.795 imbalance=0.007 prefix_aware: TTFT=280.3ms hit=0.894 imbalance=0.856 hybrid(thr=1): TTFT=104.7ms hit=0.895 imbalance=0.007
Pure prefix-aware routing increases cache hit rate but degrades TTFT. At high skew and high prefix sharing, prefix_aware achieves hit=0.894 but TTFT=280.3ms versus 110.9ms for round-robin. Concentrating requests on cached instances builds queue faster than cache savings can compensate.
Hybrid routing achieves the best of both policies. hybrid(thr=1) achieves hit=0.895 (matching prefix_aware) with TTFT=104.7ms and imbalance=0.007 (matching round-robin). The queue-depth threshold prevents load concentration while preserving cache locality.
The optimal hybrid threshold was 1 across all tested regimes. This means: route to cached instance only if its queue is empty or has one pending request. Otherwise use least-load. A conservative threshold prevents serialization while still capturing cache benefits.
Round-robin and least-load are equivalent in this simulation. Under homogeneous instances, all strategies carry similar load naturally. Routing intelligence only pays off with skewed prefix-sharing workloads.
request-routing-sim/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── instance.py │ ├── router.py │ ├── simulator.py │ ├── metrics.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs:
results/sweep_summary.csv results/summary.txt plots/ttft_vs_arrival.png plots/hybrid_threshold_sweep.png plots/cache_hit_vs_prefix_frac.png plots/load_imbalance.png plots/optimal_threshold_by_zipf.png
S-LoRA (Stanford 2023): uses scheduling that is aware of both adapter popularity and instance load, similar to the hybrid policy modeled here.
Mooncake (2024): uses prefix-aware routing with explicit cache capacity management, motivating the LRU prefix cache model in this simulator.
This project closes a three-part arc on LLM serving:
llm-inference-scheduler: which request runs inside one instance disaggregated-prefill-decode-sim: how to split phases across nodes request-routing-sim: how to distribute requests across multiple nodes
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026