Point your existing OpenAI SDK at llmux and get routing, fallbacks, per-key budgets, caching, and live cost — across every provider, with zero per-language code. Inference runs on your box by default, and a request is never silently sent off the box unless you explicitly, loggably opt in.
MIT OR Apache-2.0 · Go 1.25 · Tests
Quickstart · Docs · API · Configuration · Sovereignty gate · Status
llmux is a single Go binary that speaks the OpenAI HTTP API and routes
every request to the provider behind it — native adapters for Anthropic,
Gemini, Cohere, Bedrock, and Azure, passthrough for any OpenAI-shaped upstream
(OpenAI, DeepSeek, Groq, Mistral, Together, OpenRouter, and 100+ more), and a
local provider for an on-box Ollama/llama.cpp/vLLM server.
Every language already ships a mature OpenAI client that accepts a custom
base_url. Point it at llmux and the routing, budgets, caching, and cost
accounting happen underneath — no new SDK to learn.
It's self-hosted, open source, has no telemetry, and has no accounts — no login, no email, no sign-up; you authenticate with an operator-issued bearer token and configure it by editing a JSON file. It ships an admin dashboard inside the binary — usage, keys, and the live model catalog, nothing more. An optional control-plane seam adds centralized billing when you want it, and is invisible when you don't.
It also enforces a default-deny sovereignty gate before every dispatch, so inference stays on your box unless you explicitly opt a provider in. This is the reason the project exists — see the sovereignty gate below.
llmux is the reference implementation of default-deny egress elsewhere in this suite, so the single exception is stated up front rather than buried in a feature table.
config.Default()ships two public price feeds —openrouter.aiandraw.githubusercontent.com— so a stock gateway makes an outbound GET at startup and everysync_interval_minutes(default 360) without you configuring anything. It is a plain GET of a public price list: it carries no prompt, no completion, no API key, and no usage data, and it is not covered by the sovereignty gate, which governs inference dispatch only.If "no network calls unless I ask" is a requirement for you, turn it off:
{ "pricing": { "sources": [] } }Cost accounting still works offline from the built-in seed catalog; you can also point
sourcesat your own mirror. This default has not been changed — it is disclosed so the choice is yours and informed. Full detail: what the gate does not cover and theoutboundDialSitescensus incore/sovereign/egress_guard_test.go.
flowchart LR
client["any OpenAI client"] -->|"base_url = llmux"| mux["llmux"]
mux --> local["local — Ollama · llama.cpp · vLLM<br/>(on-box, always allowed)"]
mux --> native["Anthropic · Gemini · Cohere · Bedrock · Azure<br/>(native adapters)"]
mux --> pass["OpenAI · DeepSeek · Groq · OpenRouter · 100+ more<br/>(passthrough)"]
Prerequisites: Go 1.25+, and at least one provider API key. There is no Node toolchain anywhere in this repo — the embedded admin console (
web/ui.html) is one hand-written HTML file with inline CSS/JS, sogo buildalone is enough, including for UI changes.
git clone https://github.com/vul-os/llmux
cd llmux
make build # builds ./dist/llmux with the web UI embeddedexport OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
cp llmux.example.json llmux.json
./dist/llmux -config llmux.json # gateway on :4000, dashboard at /uiThen point any OpenAI client at it — the model string selects the route:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:4000/v1", api_key="sk-team-a")
resp = client.chat.completions.create(
model="cheapest", # least-cost route from your config
messages=[{"role": "user", "content": "hi"}],
)
print(resp.usage) # carries a "cost" object alongside
# the standard token countsThe usage block on every response is the standard OpenAI shape plus one
additive extension — a cost object computed from the live pricing catalog,
which any OpenAI client ignores harmlessly if it doesn't look for it:
17+ languages — copy-paste examples for Python, Node, TypeScript, Go, Ruby, PHP, Java, C#, Rust, C++, C, Swift, Kotlin, Elixir, R, and Dart, plus how to embed llmux as a local dependency with no separate server to run, live in Client examples.
Prefer the prebuilt archives? Every release publishes a SHA256SUMS manifest
covering all of its assets, plus a sigstore build-provenance attestation
minted from the release workflow's OIDC identity (there is no long-lived signing
key, so there is none to leak or rotate). scripts/verify.sh is what you run
before executing the bytes:
curl -fsSLO https://raw.githubusercontent.com/vul-os/llmux/v0.2.0/scripts/verify.sh
bash verify.sh --tag v0.2.0 --attest llmux_0.2.0_linux_amd64.zipIt fetches the manifest, looks up the exact entry for that asset (names are
compared as strings, not as regexes) and compares digests. Two outcomes only:
verified, or non-zero with a diagnostic naming what was wrong — missing or
malformed manifest, no entry for the asset, truncated download, digest mismatch,
HTML error page served where bytes were expected. There is no --skip-verify,
and a SHA256SUMS that 404s is a failure, never "nothing to check".
--attest additionally verifies the provenance (needs the gh CLI); leave it
off and the script says out loud that provenance was not checked, so a pass
never implies more than it checked.
make verify-selftest runs 24 synthetic-origin cases asserting that each
refusal still fires; CI runs the same matrix on every push.
This is the part other OpenAI-compatible gateways don't have. core/sovereign
classifies every configured provider by where its traffic goes, and the
server calls the gate before any network call, on every dispatch path —
chat, streaming chat, embeddings, the semantic-cache embedder, and every
model-bearing modality route. Providers resolve to a 4-tier dial, most→least
private:
| Tier | What it is | Default |
|---|---|---|
| local | inference on THIS box (loopback / unix socket) | always allowed |
| sovereign | an operator-declared endpoint the operator vouches for (unverified by Vulos) | allowed on the operator's declaration |
| brokered | a named third party under a claimed no-train agreement | blocked until allow_brokered |
| external | any other off-box endpoint (may mine/train) | blocked until allow_egress |
You opt a provider in per-provider, never globally, with fields on that
provider's config entry: "tier": "sovereign", "tier": "brokered" +
"allow_brokered": true, or "allow_egress": true. The gate fails closed:
an empty/unparseable base_url, an off-box endpoint marked local, or any
unrecognized tier value is treated as external and blocked — nothing
silently upgrades. A blocked provider never opens a socket; the denial is
logged and counted (egress_blocked metric), and every permitted off-box
call is logged with its tier so egress is always observable, never silent.
That's enforced structurally, not by convention: a source-parsing test asserts every dispatch function that can reach a provider also calls the gate, so adding a new dispatch path without wiring it in is a CI failure that names the file and line. See Architecture → the sovereignty gate for the full mechanism, and what the gate does not cover — stated plainly rather than left to be discovered.
| 🛡️ Sovereignty gate | A default-deny, 4-tier (local/sovereign/brokered/external) egress policy runs before every dispatch path. Fails closed; every permitted off-box call is logged with its tier. See above. |
| 🔌 OpenAI-compatible API | chat/completions, completions, embeddings, models, plus responses, rerank, moderations, images/generations, audio/speech, and audio/transcriptions+audio/translations (multipart speech-to-text). Works with any OpenAI SDK unchanged. chat/completions and embeddings are natively translated per provider; the other modality routes are forwarded and served only by passthrough providers — a translating native adapter (Anthropic/Gemini/Cohere/Bedrock/Azure) returns 501 for them. |
| 🌐 Multi-provider routing | Native adapters for Anthropic, Gemini, Cohere, Bedrock, and Azure — plus passthrough for any OpenAI-shaped upstream. Tool-calling, vision, and streaming translated per provider. |
| 🧭 Flexible routes | Model aliases, provider/model prefixes, wildcards (claude-*), catch-all routes, fallback chains with retries/backoff, and least-cost selection. |
| 📡 Byte-identical SSE | Streamed responses match OpenAI's wire format exactly, so every language's stream parser just works. |
| ⚡ Caching | Exact-match (LRU + TTL) and semantic (embedding-similarity), in-memory or shared via Redis. Scoped per virtual key. |
| 🔑 Virtual keys & budgets | Per-key USD budgets, RPM limits, and model allow-lists. Spend in Postgres, rate limits in Redis, when configured — otherwise fully in-memory on one replica. |
| 💲 Live pricing | A built-in seed (cost works offline) auto-syncs from OpenRouter + LiteLLM. Cost appears in each response's usage.cost; merged catalog at GET /v1/catalog.json. This sync is the one outbound call a stock gateway makes on its own — see the callout above. Turn it off with "pricing": {"sources": []}, or point it at your own mirror. |
| 📊 Embedded dashboard | Usage by model, key budgets, and the live catalog — served from the binary at /ui via go:embed. No separate service, and nothing else — it's an admin console, not a product page. |
| 🛡️ Hardened by default | Constant-time auth, size/body limits, upstream timeouts, error normalization, drop_params, Prometheus /metrics, structured logs, /health. |
llmux is pre-1.0 (v0.2.0) and under active development. Two things are
honestly not "everything works, fully hardened":
Provider adapters are on a stability ladder, promoted only once verified
against the real API (golden fixtures + make smoke), not on code maturity
alone:
| Provider | Type | Stability |
|---|---|---|
| OpenAI / DeepSeek / Groq / Mistral / Together / Fireworks / xAI / OpenRouter / Ollama / vLLM | passthrough |
stable |
| Anthropic | anthropic |
beta — translated + unit-tested, not yet live-verified |
| Google Gemini | gemini |
beta — translated + unit-tested, not yet live-verified |
| Azure OpenAI | azure |
beta — translated + unit-tested, not yet live-verified |
| Cohere | cohere |
experimental — written to spec, unverified |
| AWS Bedrock (Anthropic) | bedrock |
experimental — written to spec, unverified; streaming is synthesized, not native |
GET /health (master key) reports each configured provider's live stability.
Full table and the promotion criteria: SUPPORT.md.
Feature parity against LiteLLM is a tracked, ongoing program, not a finished checklist. Shipped: the endpoints above, fallback/least-cost routing, virtual keys with Postgres/Redis-backed cross-replica state, caching, observability basics. Not yet: multiple deployments per model / weighted load balancing, teams/orgs/end-users, TPM limits, guardrails, most of the long tail of provider integrations. Full breakdown, tier by tier: docs/parity.md.
Full documentation lives in docs/, and is also published at
vulos.org/projects/llmux.
| Getting started | Deploy the gateway, connect providers, auth and keys |
| Client examples | Copy-paste requests in curl and 17+ languages, plus embedding llmux locally |
| API reference | Endpoints, auth, errors, and cost |
| Configuration | Config file, environment variables, and the sovereignty fields |
| Architecture | How the gateway is laid out, and the sovereignty gate in full |
| Admin guide | Budgets, rate limits, cost accounting, model routing, the dashboard |
| LLM access: BYOK vs central | Per-account own-key vs central metered keys, billing |
| Control-plane seam | Optional centralized billing & entitlements |
| Operations | Building, testing, and self-hosting |
| Troubleshooting | Symptom-by-symptom fixes |
Clients authenticate with a single header, Authorization: Bearer <token> —
no accounts, no OAuth, no sessions. Virtual keys are issued by the operator in
the config file (or resolved via the optional control plane), each with a USD
budget, an RPM limit, and a model allow-list:
{ "key": "sk-team-a", "name": "team-a", "budget_usd": 100, "rpm": 600, "allowed_models": ["gpt-4o", "cheapest"] }Requests the gateway refuses before ever calling a provider: 401 invalid_api_key, 403 model_not_allowed, 404 model_not_found, 402 budget_exceeded (fails closed if spend can't be read), 429 rate_limit_exceeded, 403 model_not_priced (a budgeted key requested a model
the catalog can't price — refused pre-flight rather than metered at $0), and
403 egress_not_allowed (the sovereignty gate). Full list: API reference → Errors.
The admin dashboard ships inside the binary at /ui — no separate service,
no extra deploy. It's an admin console, not a product page: usage, keys, and
the live model catalog, and nothing else.
Screenshots — usage, keys, catalog
These predate the admin console's rewrite into
web/ui.htmland still show the retired React build's chrome (top nav with Home/Docs, a theme toggle, a marketing-style footer) — none of which the current page has. The three views themselves (usage/keys/models) are still accurate in substance; recapture against the current page is tracked separately.
 |
 |
 |
| Usage — requests, tokens, and cost, per model | Keys — per-key budgets, spend, and RPM | Models — the live, merged price catalog |
llmux is one self-hosted binary; its shape is defined by whether the optional control-plane seam is wired:
| Shape | How | Billing / sovereignty |
|---|---|---|
| Self-hosted (default) | Run the binary with provider keys and no LLMUX_CP_URL |
No telemetry, no central billing — fully sovereign; inference stays on-box until you opt a remote provider in |
| Control-plane-linked | Set LLMUX_CP_URL / LLMUX_CP_SECRET |
Virtual-key spend + usage metered through a central control plane; the seam is invisible when unset |
Both shapes are the same binary; the CP seam only adds central billing on
top, and core never imports it — delete integration/cp entirely and the
standalone build still compiles and runs. See control-plane.md.
A single binary with no required runtime dependencies — drop it on a host (or
use the Dockerfile), point it at a config, and set your
provider keys. Add Postgres and Redis when you scale to multiple replicas, so
keys, spend, rate limits, and cache stay consistent across them.
make build # go build -o dist/llmux ./cmd/llmux
make run # build and run on :4000
make docker # build the Docker imageBeyond serving, the binary exposes inspection subcommands against a running gateway:
./dist/llmux models # models with pricing + context window
./dist/llmux catalog # price catalog count and last sync time
./dist/llmux keys # virtual keys: budget, spend, rpmPrometheus metrics and structured logs are served at GET /metrics (master
key), and GET /health gives an unauthenticated liveness probe plus, to the
master key, the full provider/sovereignty topology. See
Operations and HARDENING.md for the
production security posture.
make test # go test -race ./...
make vet # go vet ./...
make cover # coverage summaryIntegration tests against Postgres/Redis activate when LLMUX_TEST_POSTGRES /
LLMUX_TEST_REDIS are set; provider conformance fixtures and the live smoke
suite (make record, make smoke) need real provider keys. Full layer
breakdown — unit, contract, gated integration, conformance, live smoke, fuzz,
and the structural guards that read source instead of running it — in
TESTING.md.
Issues and PRs welcome — see CONTRIBUTING.md for dev setup, branch conventions, and the pre-PR gates. SUPPORT.md has the provider stability ladder and where to get help, the roadmap says what's planned, and the changelog says what's shipped.
Please report vulnerabilities privately — see SECURITY.md. Do not file public issues for security problems.
The mark in brand/ is the source of truth. Every icon this repo
ships — favicon, PWA and app icons, the mark in the README and on the site — is
rendered from brand/logo.svg rather than redrawn, so there is one approved
drawing and no second copy to drift.
Copy it outward, never edit a derived copy, and never edit brand/ to match
something downstream.
MIT OR Apache-2.0 — © VulOS. llmux is a VulOS project; source and issues at github.com/vul-os/llmux.
vulos — open by design
{ "usage": { "prompt_tokens": 9, "completion_tokens": 12, "total_tokens": 21, "cost": { "input_cost": 0.0000014, "output_cost": 0.0000072, "total_cost": 0.0000086, "currency": "USD" } } }