Composable and Embeddable Communication Runtime for Distributed AI Services
-
Updated
Jun 5, 2026 - C++
#
disaggregated-serving
Here are 5 public repositories matching this topic...
Benchmark & decision framework for KV cache transfer compression in disaggregated LLM serving — 14+ compressors, 6 models, GPU-calibrated timing, decision flowchart
benchmarking compression transformers inference pytorch quantization mlx kv-cache llm disaggregated-serving
-
Updated
Feb 25, 2026 - Python
A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.
metal gpu-acceleration systems-programming apple-silicon openai-api tensor-parallelism llm-inference speculative-decoding paged-attention continuous-batching prefix-caching disaggregated-serving
-
Updated
May 9, 2026 - Rust
Production-grade Java 25 Virtual Thread inference gateway bridging NVIDIA Triton → Dynamo with Earliest Deadline First (EDF) priority queuing, adaptive batching, and async shadow validation.
redis distributed-systems grpc priority-queues load-balancing model-serving triton-inference-server virtual-threads inference-gateway semantic-caching nvidia-dynamo disaggregated-serving
-
Updated
May 9, 2026 - Java
A fault-tolerant LLM routing system that decouples inference from AWS Bedrock by routing prefill and decode tasks through SQS and ensuring zero-downtime scaling with a graceful drain sidecar.
aws bedrock disaggregation disaggregate disaggregated-serving disaggregated-prefill-decode disaggregated-inference
-
Updated
Mar 24, 2026 - Python
Improve this page
Add a description, image, and links to the disaggregated-serving topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the disaggregated-serving topic, visit your repo's landing page and select "manage topics."