A discrete-event simulator for multi-LoRA adapter serving on a single GPU. Compares three scheduling strategies under variable VRAM pressure, arrival rate, and adapter popularity distributions.
For full architecture, design decisions, and methodology see design.md.
LoRA adapters are small. The base model is not.
A LLaMA-7B base model consumes ~14GB on a 24GB GPU, leaving ~10GB for adapters. If each adapter is 100MB, you can hold at most 100 adapters resident simultaneously. With 500 adapters in the system, every cache miss requires a CPU-to-GPU swap.
The question is: which scheduling strategy minimizes time-to-first-token?
naive swap Serve each request immediately. Swap in the adapter on cache miss. No batching, no preloading. Maximum simplicity, maximum swap overhead.
hot-set preloading Pre-load the top-K most popular adapters before serving begins. Optimal for skewed workloads at low arrival rates. Loses advantage when pressure is high and arrival rate is high.
batch-by-adapter Collect requests within a time window. Group by adapter. Serve groups together. Amortizes swap cost across co-arriving requests for the same adapter. Requires sufficient request density to be effective.
Hardware: RTX 3090 (24GB VRAM, 10GB adapter budget, 16 GB/s PCIe) Regime: high VRAM pressure (>1x), arrival rate 64 req/s
naive_swap: TTFT=10.50ms swap_rate=0.39 swap%_of_TTFT=29.3% hot_set: TTFT=10.58ms swap_rate=0.40 swap%_of_TTFT=29.3% batch_swap: TTFT= 8.43ms swap_rate=0.14 swap%_of_TTFT=10.9%
Batch-swap reduces TTFT by 20% and swap rate by 64% versus naive serving. Swap overhead accounts for 29% of TTFT in unconstrained naive serving.
TTFT by output length (high pressure, arr=64):
output_len=8 (decode68ms): naive=10.50ms batch=8.43ms hot=10.58ms
output_len=32 (decode272ms): naive=10.51ms batch=8.25ms hot=10.59ms
output_len=128 (decode~1088ms): naive=10.49ms batch=8.09ms hot=10.57ms
TTFT is nearly constant across output lengths because swap happens before the first token regardless of how many tokens will be generated.
Strategy crossover by arrival rate (high pressure):
arr=4: hot_set wins (low QPS, skewed dist) arr=16: batch_swap wins arr=32: batch_swap wins arr=64: batch_swap wins by 2.07-2.40ms
Crossover point: approximately 10-16 req/s.
Swap overhead dominates TTFT for short-output workloads. At high pressure, swap accounts for 29% of TTFT in naive serving. This is output-length invariant: swap happens before the first token.
Batch-swap is the best strategy at high arrival rates under any distribution. At 64 req/s, it reduces TTFT by 20% and swap rate by 64% vs naive.
Hot-set is optimal at low arrival rates with skewed distributions. Under Zipf at arr=4, hot-set halves the swap rate vs naive. At high arrival rates and high pressure, hot-set provides no advantage.
Naive swap never outperforms both alternatives simultaneously. It is dominated across all high-pressure scenarios.
The batch window has diminishing returns beyond 25ms. Most batching gain comes from request density, not window width.
multi-lora-serving-sim/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── vram_manager.py │ ├── strategies.py │ ├── metrics.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs:
results/sweep_summary.csv results/summary.txt plots/latency_vs_arrival.png plots/swap_overhead.png plots/distribution_effect.png plots/batch_window_effect.png plots/vram_capacity_effect.png plots/vram_pressure_effect.png
S-LoRA (Stanford, 2023): production multi-LoRA serving with unified paging and batched adapter computation. This simulator models a simplified version of its adapter scheduling policy in a reproducible, didactic environment.
Punica: efficient batched LoRA kernel computation. Complementary to this work, which focuses on scheduling policy rather than kernel efficiency.
This project closes a four-part series on LLM memory management:
kv-cache-compaction-lab — KV cache memory reclamation paged-attention-sim — fragmentation-free KV allocation kv-cache-quantization-bench — compression without quality loss multi-lora-serving-sim — multi-adapter scheduling under VRAM pressure
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026