Analytical benchmark for speculative decoding at batch sizes 1-64. Finds the gamma crossover where speculative decoding becomes slower than greedy, and shows that batch size does not degrade performance when continuous refill is active.
python simulation throughput llm-inference speculative-decoding continuous-batching draft-model mlsystems batched-inference gamma-tuning
-
Updated
Jul 19, 2026 - Python