modelsphere

ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.

The models swiss can deploy, and how to serve each one: engine, image, flags, GPUs and tuned variants

Tunes LLM serving configurations overnight on your own GPUs: search spaces, pluggable search policies, benchmarked by LLMBench, winners promoted as merge requests

Helm charts for the ModelSphere LLM inference stack (sglang, vllm, cart, rdma-injector)

Deploy control plane for the ModelSphere sglang and vllm charts, driven by the model catalog

Kubernetes operator that autoscales LLM inference workloads on LLM-specific signals — KV-cache utilization, queue depth, TPM — rather than CPU and memory

Kubernetes operator that keeps routing-layer config (OpenResty / cache-aware-router / monitoring) in sync with discovered model backends

Continuation gateway: when a streaming completion stalls or drops mid-generation, resumes it once from what the client already received, so the client sees one continuous, complete stream

ModelSphere community portal — identity (users/roles/login) + federation gateway to swiss and other backends

Default community health files and organization profile for ModelSphere

HTTP API for reading and changing the SLO of an LLM inference service, backed by LLMSLORequirement CRs

An open benchmark platform for LLM serving endpoints: throughput sweeps, functional acceptance, real-traffic replay, scored leaderboards

Hang-detection sidecar for LLM inference engines (vLLM / sglang)

Search policies for LLM AutoTune: the policy SDK, a reference random search, and a chaos policy

Cache-aware router for LLM inference backends: routes each request to the replica that already holds the longest matching prompt prefix

OpenResty-based session-affinity router for LLM inference backends

Turns SLO requirements and live signals into replica decisions for LLM inference services