LLM AutoTune finds the best serving configuration for a model on your GPUs. A campaign sets a model, a search space of engine settings (sglang, vLLM) and an objective: one metric to improve, plus redlines a config must hold. A policy, a search algorithm packaged as a container, picks the configs to try. The platform launches each config, benchmarks it with LLMBench, and ranks it against the baseline, the configuration production runs today.
- Pluggable search. A policy is any container that implements a small HTTP API (policy contract). Random search is included; without a policy, the platform tries every config in the space.
- Platform-owned measurement. The platform benchmarks every config the same way and re-measures the best before reporting. A policy never grades its own results.
- Real hardware and traffic. Benchmarks run on the GPUs you serve on and can replay captured production traffic. A config that crosses a latency or quality redline is rejected.
- Existing GPUs. Kubernetes GPU nodes or ssh machines, leased to the platform for a nightly window.
- Reproducible results. Every run records its engine command, image, engine version, card type and dataset.
- Restart-safe. All state is in Postgres; restarting the worker mid-run loses nothing.
- Extensible. Plugins add pages, API routes, scheduled steps and search strategies (plugins).
The policy decides what to try. The supervisor starts the policy, launches the configs it requests, and submits each to LLMBench. At the deadline the platform re-measures the policy's best configs itself, and that measurement is the result. Details: platform architecture.
The platform runs on a Kubernetes cluster without GPUs; runs go to GPU
clusters (or ssh machines) added from the UI. Requirements: kubectl,
helm, git, openssl, and docker for building search policies.
git clone https://github.com/modelsphere/llm-autotune
cd llm-autotune
deploy/quickstart.sh # 1. LLM AutoTune + LLMBench on the current context
deploy/gpu-cluster.sh --context <gpu-cluster> # 2. a scoped account on a GPU cluster; writes llm-autotune-runs.kubeconfig- Sign in with the address and password the install prints. On Resources ▸ Add GPU cluster, upload the kubeconfig and register the GPU nodes.
- Create a search space and a campaign. The campaign's Check step pulls the images and mounts the model path on the nodes before anything starts. First campaign walks through it with screenshots.
Installing covers air-gapped clusters, a local setup with two kind clusters, and troubleshooting. After installing covers settings, more GPUs and ingress; Deploying covers the Helm chart and every setting.
deploy/demo.sh --kindCreates a local kind cluster (allow Docker about
8 GB), installs both platforms, and starts a demo campaign on a mock engine
that needs no GPU. The full loop runs; the numbers are not performance data.
deploy/demo.sh down --kind removes it.
| Campaign | One search: a model, a search space, an objective, machines and a schedule. |
| Search space | The engine settings a campaign may change, and their values. |
| Objective | The target metric to rank by, plus redlines a config must hold. |
| Run | One config, launched on a machine and benchmarked. |
| Baseline | The configuration production runs today, measured the same way. |
| Machine | Where runs land: a node of a GPU cluster, or an ssh machine. |
| Policy | A container that decides which config to try next. |
The user guide defines the rest.
backend/ FastAPI API and the orchestrator worker (Python 3.12, uv)
app/
control/ search space, orchestrator, launch drivers
evaluation/ health checks and the benchmark-platform adapter
agent/ read models for LLM-written performance reports
frontend/ Vue 3 + TypeScript UI
deploy/
quickstart.sh installs both platforms; demo.sh installs the no-GPU demo
gpu-cluster.sh prepares a GPU cluster and writes its kubeconfig
helm/ the chart
docker/ images, and a compose file for local development
policies/ submodule: llm-autotune-policies (the SDK and two policies)
operator/ the optional Kubernetes operator (Go)
mock-engine/ a fake engine for running the loop without GPUs
docs/ guides, architecture and API contracts
- User guide: every page of the platform, starting with concepts and a first campaign
- How it works: the product view, without code
- Architecture: how the system is built
- Installing, After installing, Deploying, Upgrading
- Plugins: extending the platform without forking it
- APIs: policy contract, machine lease, agent API (off by default)
See CONTRIBUTING.md: uv run pytest in backend/,
npm run typecheck in frontend/, and helm lint on the chart.