llm-gateway

English · 简体中文

A local gateway that speaks the Anthropic Messages API to your clients and either the Anthropic or the OpenAI Chat Completions API to your upstream providers — converting in both directions, streaming included.

Your clients only speak one protocol. Your providers are split across two. The gateway absorbs the difference, so an OpenAI-compatible vendor becomes an Anthropic-compatible one and back, without the client knowing.

Why

Every "Anthropic-compatible" endpoint is compatible in its own way. Anthropic's own client sends fields like thinking: {"type": "adaptive"}, output_config and context_management that a third-party implementation may reject with an opaque 400, and it puts a system-role message inside messages, which plenty of servers refuse. The gateway normalizes those shapes, maps model names onto whatever the provider calls them, and gives you one place to see what actually happened.

What it does

• Conversion in all four directions. Anthropic ⇄ OpenAI requests and responses, SSE included. Inbound protocol and upstream protocol are independent.
• Providers and model aliases. Add endpoints, declare their models, and map the client name onto the provider's own name. Unrecognized model names are reported, never silently rerouted.
• Several keys per provider. Separate accounts carry separate quotas, so requests rotate across their keys and fall through to the next one on 429, 401 or 403.
• Quota-aware rotation. For CommandCode endpoints, the gateway reads each account's remaining credits and five-hour/weekly windows in the background, prefers an account that can answer, skips one whose window is closed until it resets, and stops choosing a key that was just refused. The provider panel shows the same numbers.
• A GUI, in a window or in a browser. The same interface runs as a desktop window and as a page on the gateway's port.
• Named inbound keys. Issue one key per client and revoke them independently.
• A request log in SQLite. Model, provider, status, duration and token counts for every call — never message bodies — plus the share of input that came from the upstream's cache, and a one-click clear.
• Protocol modes. Expose the Anthropic API, the OpenAI API, or both; switching takes effect without a restart.
• Fetch models from the upstream. One call fills the model table instead of typing names by hand.
• Automatic failover. When the provider an alias names refuses (quota, credential, plan), the other enabled providers are tried in order and the log records who answered.
• Stall detection. A route that has not produced its first byte within a configurable deadline is abandoned and the request answers elsewhere; a stream that goes silent for a configurable interval is ended with a timeout instead of hanging the client. A route that keeps stalling is avoided for longer each time (two minutes, doubling up to thirty), and a successful answer clears its record.
• A speed test page. One click runs every alias through the same fixed prompt and reports first-byte latency, total time and tokens per second.

Quick start

Prebuilt binaries are on the Releases page. To build from source:

Desktop app (the window runs the gateway inside it):

cd apps/desktop && pnpm install && pnpm build
cargo run -p llm-gateway-desktop

Headless, as a plain local service:

cargo run -p gateway-server

Either way the gateway listens on 127.0.0.1:8787 with no providers. Open the GUI, add a provider, and it starts routing.

Point at a specific configuration file instead of the platform default:

LLM_GATEWAY_CONFIG=/path/to/config.json cargo run -p gateway-server

Talking to it

# Anthropic-shaped request
curl http://127.0.0.1:8787/v1/messages \
  -H 'content-type: application/json' \
  -d '{"model":"<alias>","max_tokens":256,"stream":true,
       "messages":[{"role":"user","content":"hello"}]}'

# OpenAI-shaped request (needs a mode that serves it)
curl http://127.0.0.1:8787/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"<alias>","stream":true,
       "messages":[{"role":"user","content":"hello"}]}'

<alias> is whatever you named a model in the GUI, not the provider's own model id.

Endpoints

Endpoint	Available in
POST /v1/messages	primary_anthropic, route
POST /v1/messages/count_tokens	primary_anthropic, route
POST /v1/chat/completions	primary_open_ai, route
GET /v1/models	always
GET /admin/config, PUT /admin/config	always
GET /admin/logs, DELETE /admin/logs	always
GET /admin/health	always
POST /admin/provider-models	always
GET /admin/provider-quota, POST /admin/provider-quota	always
GET / (the GUI)	when the web GUI is enabled

Authentication accepts either x-api-key or Authorization: Bearer. With no key configured the gateway accepts anything, which is what lets SDK clients present an arbitrary placeholder key.

The four combinations

                 ┌──────────────────────────────┐
 Anthropic SDK ──┤ POST /v1/messages            │──▶ Anthropic-compatible provider
 OpenAI SDK    ──┤ POST /v1/chat/completions    │──▶ OpenAI-compatible provider
                 └──────────────────────────────┘
                        llm-gateway

Inbound	Upstream	Works
Anthropic	Anthropic	✅ pass-through with normalization
Anthropic	OpenAI	✅
OpenAI	Anthropic	✅
OpenAI	OpenAI	✅

A request always flows client protocol -> IR -> upstream protocol, and a response always flows upstream SSE -> ModelEvent -> client SSE. No converter knows which protocol sits on the other side, which is what keeps the four combinations from needing four code paths.

Configuration

config.json lives in the platform's config directory, or wherever LLM_GATEWAY_CONFIG points. The GUI edits this file; so can you.

{
  "protocol_mode": "route",            // primary_anthropic | primary_open_ai | route
  "server": {
    "bind": "127.0.0.1",
    "port": 8787,
    "api_key": null,                   // legacy single key; either this or api_keys gates inbound calls
    "api_keys": [                      // one key per client
      { "id": "…", "name": "Claude Desktop", "key": "sk-gw-…" }
    ]
  },
  "web_ui": {
    "enabled": false,                  // serve the GUI on the port above
    "token": null                      // required for admin access when enabled
  },
  "default_provider_id": null,         // catch-all for unrecognized model names
  "providers": [
    {
      "id": "…",                       // assigned by the gateway if omitted
      "name": "Example",
      "protocol": "anthropic",         // anthropic | open_ai
      "base_url": "https://api.anthropic.com",
      "api_key": "sk-…",
      "api_keys": [                    // extra accounts for the same endpoint
        { "id": "…", "name": "second account", "key": "sk-…" }
      ],
      "enabled": true,
      "custom_headers": {},
      "models": [
        {
          "id": "…",
          "alias": "claude-opus-4-7",  // what clients ask for
          "upstream_model": "claude-opus-4-7",
          "enabled": true,
          "max_output_tokens": 8192,
          "max_tokens_field": "max_tokens",   // or max_completion_tokens
          "extra": {}                  // merged into the upstream request body
        }
      ]
    }
  ]
}

max_tokens_field exists because the ecosystem is genuinely split: OpenAI's reasoning models reject max_tokens, while most third-party OpenAI-compatible vendors reject max_completion_tokens. It defaults to max_tokens, which is what most compatible vendors accept.

extra is an escape hatch for vendor-specific parameters the gateway does not model. It cannot overwrite fields that carry the request's meaning (model, messages, tools, stream, and the output-limit field); doing so is rejected rather than silently applied.

Two things worth knowing

Provider API keys are stored in plain text. It is a local single-user tool and the keys have to survive a round trip through the GUI to be editable. The GUI masks them on screen. If you bind to a non-loopback address with the web GUI enabled, the gateway refuses to save unless you set web_ui.token, because that same API reads and writes those keys.

PUT /admin/config replaces the whole document. A body containing only part of the configuration clears the rest. The GUI always sends the complete document.

Proxies

The gateway keeps two paths to the internet — direct, and the proxy Windows is configured to use — and prefers whichever has been faster lately. Each path carries a decaying average of how long it took to produce its first byte (a failure counts as a long stall). Direct wins ties and starts as the default; the proxy takes over only once it has two samples and is a clear 15% ahead, or when direct has failed. Every tenth request still probes direct, so a network that changed underneath the numbers can win the preference back.

ALL_PROXY / HTTP_PROXY / HTTPS_PROXY are read from the environment (with NO_PROXY respected) and are used as-is; the system proxy is the second path. GET /admin/health reports proxy_from_env, system_proxy_available and preferred_path, and the mode page shows the current preference.

If both paths fail, the gateway posts a Windows notification — at most once every ten minutes — saying the network or proxy needs attention, and the request log records it.

The GUI

The usage guide page shows the address, the live endpoints, every model alias, and lets you issue and revoke inbound keys without leaving the app.

Development

cargo test --workspace                 # protocol and server tests
cargo clippy --workspace --all-targets
cargo fmt --check
cd apps/desktop && pnpm typecheck

The protocol tests replay recorded SSE against a stand-in upstream, so the conversion logic is verifiable without credentials or network access.

See CONTRIBUTING.md for the layout and the review checklist.

Not implemented

• Anthropic's /v1/responses-style and OpenAI's Responses API are not supported; the gateway speaks Messages and Chat Completions.
• count_tokens is an estimate derived from the request's characters, not a real tokenizer. The response says so in an x-token-count: approximate header.
• Prompt-cache breakpoints (cache_control) are not added to Anthropic upstream requests, and prompt_cache_key is not added to OpenAI ones. Cache fields reported by an upstream are passed through faithfully.
• Prebuilt binaries cover Windows, Linux and macOS on the Releases page. The macOS build ships the headless server only; the desktop shell has to be built from source there.

License

MIT