gpu-serving

Star

Here are 3 public repositories matching this topic...

coconut-labs / kvwarden

Star

Tenant-fair LLM inference orchestration on a single GPU. No Kubernetes.

python multi-tenancy inference-server vllm llm-inference sglang fair-scheduling gpu-serving

Updated Sep 4, 2026
Python

Akhilesh-Chandewar / pdserve

Star

Prefill/Decode-disaggregated LLM serving library in Go — KV-cache connectors + SLO-aware elastic GPU flipping; published benchmark suite (colocated TPOT p99 inflates 133x under burst)

go benchmarking slo kv-cache vllm llm-inference sglang pd-disaggregation gpu-serving

Updated Sep 17, 2026
Go

aryanputta / cachepilot

Star

Real-time AI memory orchestrator for multi-model GPU serving. PERC: provably optimal KV cache eviction via fractional knapsack. 25-79% eviction cost reduction vs LRU.

optimization multi-model inference-optimization kv-cache llm-inference cache-eviction gpu-serving

Updated Apr 22, 2026
Python

Add this topic to your repo

To associate your repository with the gpu-serving topic, visit your repo's landing page and select "manage topics."

Learn more