===Repository to explore various optimizations in LLMs===
A. Speculative Decoding is a technique usued to accelerate inference in LLMs using smaller LLMs without changing the output probability distribution of the tokens. The low latency of the smaller model combined with accuracy of the larger LLM makes this method very effective. Refer to [1] for more details.
My youtube playlist explaining the paper in extensive detail
In speculative_decoding.py, we have the implementation for the Speculative Decoding Algorithm . Experiments:
-
Alpha (α) calculation
-
Plotting histogram of beta values for various time steps and observing acceptance/rejection
- Exploring the relation between context and beta values
B. I have implemented autoregressive decoding with manual handling of KV Cache in autoregressive_decoding.py. There are two simple experiments with autoregressive decoding:
- Experiment 1: Generating the same set of tokens is much faster with KV Cache.
- Experiment 2: Smaller LLMs are faster than larger LLMs
- Next steps: Implement latent KV Cache and observe speedup
References:
- Leviathan, Y., Kalman, M. & Matias, Y.. (2023). Fast Inference from Transformers via Speculative Decoding. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:19274-19286 Available from https://proceedings.mlr.press/v202/leviathan23a.html.