token-eviction

Here are 2 public repositories matching this topic...

Training-free KV cache compression via E8 lattice quantization; fits longer context in the same VRAM

Compress LLM KV cache by 10-33x with this training-free method requiring one line of code.

compression analysis transformers inference pytorch trace quant attention llama quantization memory-efficient backtest mistral vector-quantization kv-cache llm long-context e8-lattice token-eviction

Add a description, image, and links to the token-eviction topic page so that developers can more easily learn about it.

To associate your repository with the token-eviction topic, visit your repo's landing page and select "manage topics."