paged-kv-cache

A from scratch LLM inference engine build in PyTorch with custom GPT2 transformers, kv cache, paged kv cache, continuous batching and A100 benchmarks

A lightweight, educational LLM inference engine for studying continuous batching, paged KV cache, chunked prefill, and online serving.

A lightweight LLM inference and tool-calling agent system with KV cache experiments.

A learning-oriented single-GPU LLM inference runtime with reproducible RTX 5060 experiments.

Learn AI Infra & DeepSeek inference from scratch. A CPU-only NumPy LLM engine: DeepInfer design, MLA, MoE, paged KV cache, speculative decoding, continuous batching & HTTP serving. English / Chinese.

Production-Grade C++17 AI Serving Engine with Arena Memory Pool, Dynamic Batching, Paged KV-Cache & Dual OpenAI/Ollama Protocols.

ForgeServe is a small, modular LLM inference engine built with Python, PyTorch, and Hugging Face Transformers, designed to make modern inference systems understandable by implementing them step-by-step rather than treating inference as a black box.