LLM Key Rotator

OpenAI-compatible reverse proxy that rotates across providers on failure, with Anthropic↔OpenAI format translation. Built with Rust + axum.

Features

• Per-failure rotation through providers from CSV
• Always uses CSV model (ignores client's model field)
• Streaming (SSE) + non-streaming support for both OpenAI and Anthropic formats
• Anthropic↔OpenAI translation — full message, tool call, and tool result conversion
• --cache flag — injects Anthropic cache_control hints into system prompt and last user message; falls back to uncached on 400
• --compress flag — injects a brevity system prompt into all requests to reduce token usage
• Permanent provider skip on 413 (payload too large); skipped providers excluded from rotation
• Structured logging with tracing including TTFB (time to first byte) metrics
• /health and /stats endpoints
• Static musl binaries for Android ARM64 (via NDK); Ubuntu x64 requires musl-tools
• Optimized streaming with proper line buffering to eliminate artificial latency
• Lock-free stats using atomic operations for minimal contention
• Connection pooling with configurable idle timeouts (10 connections per host, 90s idle timeout)
• Fast timeouts (5s connect, 120s total) to quickly fail over to next provider
• Optional custom DNS resolver (disabled by default, enable with USE_CUSTOM_DNS env var)

CSV Format

name,base_url,model,api_key
gemini,https://generativelanguage.googleapis.com/v1beta/openai/,gemini-2.0-flash,AIza...
groq,https://api.groq.com/openai/v1,llama-3.3-70b,gsk_...

Configuration

Variable	Default	Description
CSV_PATH	$HOME/code/freellmkeys.csv	Path to provider CSV
BASE_URL	http://0.0.0.0:3001/v1	Bind address openai-compatible
ANTHROPIC_BASE_URL	http://0.0.0.0:3001/anthropic	Bind address anthropic-compatible
API_KEY	(none)	Require Bearer auth header
RUST_LOG	llmkeyrotator=info	Log level
USE_CUSTOM_DNS	(unset)	Set to any value to use Google DNS (8.8.8.8, 8.8.4.4) + Cloudflare (1.1.1.1) instead of system DNS

Endpoints

• POST /v1/* — OpenAI-compatible proxy (e.g. /v1/chat/completions). Strips /v1 prefix, forwards to provider base_url + rest_of_path. Always uses model from CSV.
• POST /anthropic/* — Anthropic-compatible proxy (e.g. /anthropic/v1/messages). Converts Anthropic ↔ OpenAI format, strips /anthropic prefix, forwards to provider. Always uses model from CSV.
• GET /health — Status, provider count, current provider
• GET /stats — Request/error/rotation counts, per-provider failure stats
• /* — Catch-all proxy to current provider (strips /v1 or /anthropic prefix if present, always uses model from CSV)

Build

# Native debug build (x86_64 Linux)
cargo build

# Native release build (x86_64 Linux)
cargo build --release

Static Binaries

Android ARM64 (via NDK)

# Prerequisites: Have Android NDK installed (tested with r29)
# Configure .cargo/config.toml with NDK toolchain paths

# Add target and build
rustup target add aarch64-linux-android
cargo build --release --target aarch64-linux-android

Output: target/aarch64-linux-android/release/llmkeyrotator

Ubuntu x64 (musl)

# Prerequisites: Install musl-tools
# sudo apt install musl-tools

# Add target and build
rustup target add x86_64-unknown-linux-musl
cargo build --release --target x86_64-unknown-linux-musl

Output: target/x86_64-unknown-linux-musl/release/llmkeyrotator (statically-linked ELF)

Helper Script

./build.sh  # Attempts both targets, skips x86_64 if musl-gcc missing