LimitlessLLM Proxy

An OpenAI-compatible router for using multiple LLM providers through one API endpoint.

LimitlessLLM Proxy lets your app send AI requests to one local/proxy endpoint while the proxy handles provider selection, fallback, rate-limit tracking, and model routing.

It is designed for developers who want to add AI features to apps without hardcoding one provider directly into the client.

Example: BadaBook, an ebook reader with AI-assisted translation, can send page translation requests to LimitlessLLM instead of connecting directly to Gemini, Groq, Mistral, or another provider. The app keeps one API format while the router handles the configured providers behind the scenes.

---

Features

Feature	Description
OpenAI-compatible API	Works with OpenAI SDKs and tools using a custom base_url
Automatic fallback	Tries another configured model when one provider fails or hits limits
Config-based routing	Manage models, providers, limits, and priority from config.yaml
Multiple provider support	Supports Gemini, Groq, Mistral, OpenRouter, SambaNova, Cohere, and more
Multiple API keys	Supports multiple keys per provider for configured quota management
Sticky sessions	Keeps multi-turn conversations on the same model when possible
Free-tier friendly	Useful with providers that offer free or low-cost API access

---

Quick Start

1. Start the server

Windows:

run.bat

Linux / macOS:

chmod +x run.sh
./run.sh

The startup script creates a virtual environment, installs dependencies, copies .env.example to .env, and starts the server on port 3001.

---

2. Add API keys

Open .env and fill in the providers you want to use.

UNIFIED_API_KEY=change-me

GOOGLE_API_KEY=
GROQ_API_KEY=
GROQ_API_KEY_2=
MISTRAL_API_KEY=
OPENROUTER_API_KEY=
SAMBANOVA_API_KEY=
COHERE_API_KEY=

You do not need keys for every provider. Empty keys are skipped.

UNIFIED_API_KEY is the key your own app uses when calling the proxy.

---

3. Send a request

Python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="change-me",
)

response = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "Translate this text to German: Hello, world!"}
    ],
)

print(response.choices[0].message.content)

cURL

curl http://localhost:3001/v1/chat/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      { "role": "user", "content": "Hello!" }
    ]
  }'

Use model="auto" to let LimitlessLLM choose the best available configured model.

You can also request a specific model by using its display_name from config.yaml.

---

Configuration

LimitlessLLM uses two main files:

File	Purpose
.env	Stores API keys and secrets
config.yaml	Stores providers, models, limits, server settings, and fallback order

---

Environment Variables

Example .env:

UNIFIED_API_KEY=change-me

GOOGLE_API_KEY=

GROQ_API_KEY=
GROQ_API_KEY_2=

CEREBRAS_API_KEY=
SAMBANOVA_API_KEY=
NVIDIA_API_KEY=
MISTRAL_API_KEY=
OPENROUTER_API_KEY=
GITHUB_API_KEY=
COHERE_API_KEY=
CLOUDFLARE_API_KEY=
HUGGINGFACE_API_KEY=
OLLAMA_API_KEY=

For Cloudflare, use this format:

CLOUDFLARE_API_KEY=account_id:api_token

---

Config Example

Example config.yaml:

server:
  host: "0.0.0.0"
  port: 3001
  log_level: "info"

unified_api_key: "change-me"

providers:
  google:
    keys:
      - "${GOOGLE_API_KEY}"

  groq:
    keys:
      - "${GROQ_API_KEY}"
      - "${GROQ_API_KEY_2}"

fallback_chain:
  - platform: google
    model_id: gemini-2.5-flash
    display_name: Gemini 2.5 Flash
    enabled: true
    limits:
      rpm: 10
      rpd: 1500
      tpm: 250000
      tpd: null

  - platform: groq
    model_id: meta-llama/llama-4-scout-17b-16e-instruct
    display_name: Llama 4 Scout
    enabled: true
    limits:
      rpm: 30
      rpd: 14400
      tpm: 6000
      tpd: null

---

Fallback Chain

The fallback_chain controls model priority.

Models higher in the list are tried first. If a model is unavailable, rate-limited, disabled, or fails, the router can move to the next available model.

fallback_chain:
  - platform: google
    model_id: gemini-2.5-flash
    display_name: Gemini 2.5 Flash
    enabled: true

  - platform: groq
    model_id: meta-llama/llama-4-scout-17b-16e-instruct
    display_name: Llama 4 Scout
    enabled: true

To temporarily disable a model:

enabled: false

---

Model Fields

Field	Required	Description
platform	Yes	Provider name. Must match a provider under providers:
model_id	Yes	Model ID sent to the provider API
display_name	Yes	Human-readable model name shown by /v1/models
enabled	No	Set to false to disable the model
limits.rpm	No	Requests per minute
limits.rpd	No	Requests per day
limits.tpm	No	Tokens per minute
limits.tpd	No	Tokens per day

Use null for unlimited or unknown limits.

---

Supported Providers

Platform	Provider	Adapter Type
google	Google Gemini	Native Gemini API
groq	Groq	OpenAI-compatible
cerebras	Cerebras	OpenAI-compatible
sambanova	SambaNova	OpenAI-compatible
nvidia	NVIDIA NIM	OpenAI-compatible
mistral	Mistral AI	OpenAI-compatible
openrouter	OpenRouter	OpenAI-compatible
github	GitHub Models	OpenAI-compatible
cohere	Cohere	Native Cohere API
cloudflare	Cloudflare Workers AI	Custom URL structure
huggingface	HuggingFace Router	OpenAI-compatible
ollama	Ollama Cloud	OpenAI-compatible

---

Adding a New OpenAI-Compatible Provider

For a provider that uses an OpenAI-compatible API, add it to:

app/providers/__init__.py

Example:

"myprovider": {
    "name": "My Provider",
    "base_url": "https://api.myprovider.com/v1"
}

Then add its API key to .env and add a model entry to config.yaml.

---

Manual Installation

Create a virtual environment:

python -m venv .venv

Activate it:

Linux / macOS:

source .venv/bin/activate

Windows:

.venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

Create the environment file:

cp .env.example .env

Start the server:

uvicorn app.main:app --host 0.0.0.0 --port 3001 --reload

---

Docker

cp .env.example .env

Edit .env, then run:

docker compose up -d --build

The container listens on port 3001 by default.

---

API Endpoints

Chat Completions

POST /v1/chat/completions

OpenAI-compatible chat completions endpoint.

Models

GET /v1/models

Returns the configured enabled models.

---

Routing Behavior

For each request, LimitlessLLM:

1. Checks for an existing sticky session.
2. Sorts models by fallback priority and penalty score.
3. Checks configured rate limits.
4. Selects an available API key.
5. Sends the request to the provider.
6. Falls back to the next model when needed.

If all configured models are unavailable, the proxy returns:

503 - All models exhausted. Add more API keys or wait for rate limits to reset.

Routing state, rate-limit counters, penalty scores, and sticky sessions are stored in memory.

Limitations

LimitlessLLM is intended for development, prototypes, personal apps, student projects, internal tools, and non-critical workloads.

Important limitations:

• Provider free tiers, pricing, rate limits, and terms can change.
• Users are responsible for following each provider's terms of service.
• Routing state is stored in memory and resets on restart.
• Rate-limit counters are stored in memory and reset on restart.
• Sticky sessions are stored in memory and reset on restart.
• Multi-instance deployments need shared state, which is not included by default.
• Request history is not persisted.
• This project does not guarantee production uptime or provider availability.

LimitlessLLM does not bypass provider rules, provider limits, or provider terms of service. It is a routing proxy for managing configured LLM providers through one OpenAI-compatible API.

---

License

MIT — free to use, modify, and distribute.