Skip to content

About

Self-hosted OpenAI-compatible gateway with round-robin, cache-aware and TPS-aware routing and load balancing. One Node process, one SQLite database, one admin UI.

Resources

Contributing

Security policy

Stars

9 stars

Watchers

2 watching

Forks

Repository files navigation

SmolRouter

SmolRouter

Self-hosted OpenAI-compatible gateway. One Node process, one SQLite database, one admin UI.

License: AGPL v3 Node SQLite

Overview

SmolRouter sits between OpenAI-compatible clients and one or more upstream providers. Clients authenticate with a gateway key scoped to specific model aliases. The gateway handles routing, key pools, failover, TPS-aware routing, caching, budgets, and telemetry.

There are no runtime npm dependencies, no database server, and no build step. Node 22.13 or newer is required for node:sqlite.

Features

  • OpenAI-compatible POST /v1/chat/completions, GET /v1/models, and generic /v1/* JSON paths.
  • Model aliases and provider routes with priority tiers and round-robin, least-used, or cache-aware balancing.
  • Provider key pools with round-robin rotation and automatic auth-failure benching.
  • Failover with capped backoff, cooldowns, recovery probes, and proactive health checks.
  • TPS-aware routing with a preferred route, effective-TPS floor, fallback, and recovery probes.
  • Exact response cache, singleflight coalescing, and prompt-cache savings tracking.
  • Upstream and downstream pricing, per-key budget rules, and currency conversion.
  • Capability routing for tools, vision, audio, json, reasoning, and max context.
  • Attempt telemetry, usage reports, latency and TPS reports, Prometheus metrics, and audit rows.
  • SQLite backups, restore tooling, retention maintenance, and config export/import.
  • Admin UI in plain HTML, CSS, and ES modules.

Quick start

Requirements: Node.js 22.13 or newer.

npm run setup
npm start
  • Gateway: http://127.0.0.1:8787/v1
  • Admin UI: http://127.0.0.1:8787/admin

First request:

curl http://127.0.0.1:8787/v1/chat/completions \
  -H "Authorization: Bearer YOUR_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-alias",
    "messages": [{ "role": "user", "content": "Hello via SmolRouter." }],
    "stream": true
  }'

See Installation for setup details, reverse proxies, upgrades, and restore.

Documentation

Guide Contents
Documentation index All guides.
About Design, architecture, tech stack, compatibility.
Installation Setup, first start, reverse proxy, upgrades, restore.
Usage Client configuration, requests, headers, dashboard, errors.
TPS-aware routing Preferred routes, demotion, fallback, probes, configuration.
Billing and budgets Pricing layers, budget rules, currency, LiteLLM surfaces.
Operations Health, metrics, jobs, backups, retention, configuration.
API reference Response envelopes, endpoint map, Postman collection.
Security Authentication, key handling, CSRF, privacy, audit.
Development Tests, layout, migrations, contribution rules.
Changelog Release history.

Every environment variable is documented in .env.example. The Postman collection is at postman/smolrouter.postman_collection.json.

Security and privacy

  • Admin passwords use scrypt. Sessions are signed, HttpOnly cookies.
  • Admin mutations require a session cookie and X-CSRF-Token.
  • Gateway keys are SHA-256 hashed and scoped to models.
  • Provider keys are AES-256-GCM encrypted at rest and never returned.
  • Telemetry, usage, and TPS records store metadata only. Prompt and completion content is not stored.
  • The optional response cache keeps response bodies in memory only. It is bounded and can be disabled per model.

See Security.

Compatibility

Any provider that implements the OpenAI API can be used. Tested or documented providers include OpenAI, OpenRouter, Google Gemini's OpenAI-compatible endpoints, Vertex AI adapters, Nvidia NIM, DeepSeek, Groq, and Cerebras.

Other POST /v1/* paths are routed when the JSON body contains a model field.

Testing

npm test

Tests use temporary SQLite databases and stub upstreams; no live provider is required. See Development.

License

SmolRouter is licensed under the GNU Affero General Public License v3 (AGPL-3.0). See LICENSE.

About

Self-hosted OpenAI-compatible gateway with round-robin, cache-aware and TPS-aware routing and load balancing. One Node process, one SQLite database, one admin UI.

Resources

Contributing

Security policy

Stars

9 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages