Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ left untouched.

The example config ships ready-to-use entries for **GPT‑5.5 (Codex login)**,
**MiniMax‑M3**, **MiMo v2.5 Pro**, **DeepSeek V4 Pro/Flash**, **Step Flash**,
**Ollama Cloud**, **OpenCode Go**, **OpenRouter**, and **local models** — keep
**Ollama Cloud**, **OpenCode Go**, **OpenRouter**, **Requesty**, and **local models** — keep
the ones you have a plan for, delete the rest. (Cursor's Composer needs the
`cursor-agent` CLI and isn't HTTP-based — see
[docs/ADD_A_MODEL.md](docs/ADD_A_MODEL.md).)
Expand Down Expand Up @@ -283,12 +283,16 @@ Example — MiMo and an OpenRouter model:
Put your key right in `config.json` (it's gitignored) or use `${ENV_VAR}` and
export it — or drop keys into a gitignored `ultracode.env` the launchers load.

[Requesty](https://requesty.ai) works the same way — another `openai_compat`
gateway with `upstream` `https://router.requesty.ai/v1`, `provider/model` naming
(e.g. `openai/gpt-4o-mini`), and `auth` `Bearer ${REQUESTY_API_KEY}`.

Route types:

| `type` | Use for | Needs |
|-----------------|----------------------------------------------------|-------|
| *(omit)* | Real Claude or any Anthropic-compatible endpoint | nothing, or `auth`/`upstream` |
| `openai_compat` | MiMo, DeepSeek, OpenRouter, OpenAI, Ollama, local llama.cpp — anything that speaks OpenAI Chat Completions (tools included) | an API key |
| `openai_compat` | MiMo, DeepSeek, OpenRouter, Requesty, OpenAI, Ollama, local llama.cpp — anything that speaks OpenAI Chat Completions (tools included) | an API key |
| `codex_oauth` | GPT‑5.5 via a ChatGPT/Codex login (no API key) | `codex login` once |
| `cursor_agent` | Cursor Composer (experimental) | `cursor-agent login` |
| `auto` | The [Auto Router](docs/AUTO_ROUTER.md) — score candidates per task and route to the cheapest that's good enough | a `router` block + a classifier model |
Expand Down
8 changes: 8 additions & 0 deletions config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
{ "id": "claude-ollama-cloud", "display_name": "Ollama Cloud" },
{ "id": "claude-opencode", "display_name": "DeepSeek V4 Pro (OpenCode Go)" },
{ "id": "claude-openrouter", "display_name": "Llama 3.3 70B (OpenRouter)" },
{ "id": "claude-requesty", "display_name": "GPT-4o mini (Requesty)" },
{ "id": "claude-local", "display_name": "Local model" },
{ "id": "claude-composer", "display_name": "Composer 2.5 (Cursor, experimental)" }
],
Expand Down Expand Up @@ -103,6 +104,13 @@
"model": "meta-llama/llama-3.3-70b-instruct",
"auth": "Bearer REPLACE_WITH_YOUR_OPENROUTER_KEY"
},
"claude-requesty": {
"_": "Requesty (https://requesty.ai) - an OpenAI-compatible LLM gateway. Models use provider/model naming (e.g. openai/gpt-4o-mini); swap model for any slug Requesty lists. Get a key at https://app.requesty.ai/api-keys.",
"type": "openai_compat",
"upstream": "https://router.requesty.ai/v1",
"model": "openai/gpt-4o-mini",
"auth": "Bearer ${REQUESTY_API_KEY}"
},
"claude-local": {
"_": "A local OpenAI-compatible server (Ollama :11434, llama.cpp :8080, LM Studio :1234). The key is usually ignored.",
"type": "openai_compat",
Expand Down
16 changes: 16 additions & 0 deletions docs/ADD_A_MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,21 @@ llama.cpp / LM Studio server, etc. Tool calls are translated both ways.
}
```

[Requesty](https://requesty.ai) is another OpenAI-compatible gateway — same
shape, just a different `upstream` and `provider/model` naming
([docs](https://docs.requesty.ai), keys at
[app.requesty.ai/api-keys](https://app.requesty.ai/api-keys)):

```json
"claude-requesty": {
"type": "openai_compat",
"upstream": "https://router.requesty.ai/v1",
"model": "openai/gpt-4o-mini",
"auth": "Bearer ${REQUESTY_API_KEY}"
}
```


- `upstream` is the OpenAI **base URL exactly as the provider documents it**
(usually ends in `/v1`). The proxy appends `/chat/completions`.
- `model` is the backend's real model id, **not** the `claude-…` alias.
Expand Down Expand Up @@ -204,6 +219,7 @@ plan/key for and delete the rest:
| Ollama Cloud | `claude-ollama-cloud` | `openai_compat` | Ollama Cloud |
| DeepSeek V4 Pro (OpenCode Go) | `claude-opencode` | `openai_compat` | OpenCode Zen (Go subscription) |
| Llama 3.3 70B (OpenRouter) | `claude-openrouter` | `openai_compat` | OpenRouter |
| GPT-4o mini (Requesty) | `claude-requesty` | `openai_compat` | Requesty |
| Local model | `claude-local` | `openai_compat` | local server |
| Composer 2.5 (experimental) | `claude-composer` | `cursor_agent` | cursor-agent |
| Auto (smart routing) | `claude-auto` | `auto` | picks among your backends per task |
Expand Down
146 changes: 88 additions & 58 deletions proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,13 @@ def _context_length_hint(detail):
except Exception:
UC_MODEL_MAP = {}

# Ensure sibling modules (providers/*.py) are importable even when the
# interpreter's default sys.path[0] does not point here (e.g. zipped/embedded
# Python builds). This is a no-op when the script dir is already on sys.path.
_HERE = os.path.dirname(os.path.abspath(__file__))
if _HERE not in sys.path:
sys.path.insert(0, _HERE)

# Optional Codex/ChatGPT OAuth helper (only needed for "codex_oauth" routes).
try:
from providers import codex_oauth as _codex_oauth # type: ignore
Expand Down Expand Up @@ -1932,9 +1939,12 @@ def _classifier_complete(slot, system_prompt, user_content, timeout):
headers[hk] = hv
req = urllib.request.Request(url, data=data, headers=headers, method="POST")
resp = urllib.request.urlopen(req, timeout=timeout)
obj = json.loads(resp.read().decode("utf-8"))
msg = ((obj.get("choices") or [{}])[0].get("message") or {})
return msg.get("content") or ""
try:
obj = json.loads(resp.read().decode("utf-8"))
msg = ((obj.get("choices") or [{}])[0].get("message") or {})
return msg.get("content") or ""
finally:
resp.close()

# Anthropic passthrough (real Claude or any Anthropic-compatible endpoint).
url = (slot.get("upstream") or UPSTREAM).rstrip("/") + "/v1/messages"
Expand All @@ -1951,8 +1961,11 @@ def _classifier_complete(slot, system_prompt, user_content, timeout):
headers[hk] = hv
req = urllib.request.Request(url, data=data, headers=headers, method="POST")
resp = urllib.request.urlopen(req, timeout=timeout)
obj = json.loads(resp.read().decode("utf-8"))
return _text_from_anthropic_content(obj.get("content"))
try:
obj = json.loads(resp.read().decode("utf-8"))
return _text_from_anthropic_content(obj.get("content"))
finally:
resp.close()


def _parse_scores(text, candidate_ids):
Expand Down Expand Up @@ -2264,15 +2277,18 @@ def _handle_models(self) -> bool:
try:
req = urllib.request.Request(url, headers=fwd_headers, method="GET")
resp = urllib.request.urlopen(req, timeout=30)
parsed = json.loads(resp.read().decode("utf-8"))
if isinstance(parsed, dict) and isinstance(parsed.get("data"), list):
base = parsed
# Learn the current real-Claude ids from this successful fetch and
# cache them, so a newly released Opus is remembered for next time
# (even when upstream is later unreachable). Recompute stock after
# learning so any new id also lands in THIS response.
_learn_stock_from_upstream(base["data"])
stock = _stock_models()
try:
parsed = json.loads(resp.read().decode("utf-8"))
if isinstance(parsed, dict) and isinstance(parsed.get("data"), list):
base = parsed
# Learn the current real-Claude ids from this successful fetch and
# cache them, so a newly released Opus is remembered for next time
# (even when upstream is later unreachable). Recompute stock after
# learning so any new id also lands in THIS response.
_learn_stock_from_upstream(base["data"])
stock = _stock_models()
finally:
resp.close()
except Exception as e:
# No usable upstream list (e.g. no Anthropic credential to forward,
# or an offline blip). We still serve stock + custom below, so real
Expand Down Expand Up @@ -2451,26 +2467,34 @@ def _handle_openai_compat(self, body: bytes, route: dict):

def _mk_events():
req = urllib.request.Request(url, data=payload, headers=headers, method="POST")
resp = None
try:
resp = urllib.request.urlopen(req, timeout=600)
except urllib.error.HTTPError as e:
detail = ""
try:
detail = e.read().decode("utf-8", "replace")[:800]
except Exception:
pass
hint = _context_length_hint(detail)
log("openai_compat upstream HTTP %s for %s: %s" % (e.code, url, detail))
yield {"type": "error", "status": e.code,
"message": "openai_compat upstream %s: %s%s"
% (e.code, detail, hint)}
return
except Exception as e:
log("openai_compat upstream error %s for %s" % (e, url))
yield {"type": "error", "status": 502,
"message": "openai_compat upstream error: %s" % e}
return
yield from _oai_response_to_events(resp)
resp = urllib.request.urlopen(req, timeout=600)
except urllib.error.HTTPError as e:
detail = ""
try:
detail = e.read().decode("utf-8", "replace")[:800]
except Exception:
pass
hint = _context_length_hint(detail)
log("openai_compat upstream HTTP %s for %s: %s" % (e.code, url, detail))
yield {"type": "error", "status": e.code,
"message": "openai_compat upstream %s: %s%s"
% (e.code, detail, hint)}
return
except Exception as e:
log("openai_compat upstream error %s for %s" % (e, url))
yield {"type": "error", "status": 502,
"message": "openai_compat upstream error: %s" % e}
return
yield from _oai_response_to_events(resp)
finally:
if resp is not None:
try:
resp.close()
except Exception:
pass

events = _events_with_retry(_mk_events, label="openai_compat %s" % model_id)
if want_stream:
Expand Down Expand Up @@ -2690,33 +2714,39 @@ def _json_anthropic_from_events(self, events, model_id: str):

# ---- low-level response helpers -------------------------------------
def _relay_response(self, resp, streaming: bool):
status = getattr(resp, "status", None) or resp.getcode()
self.send_response(status)
for k, v in resp.headers.items():
kl = k.lower()
if kl in _HOP_BY_HOP or kl in ("content-length", "content-encoding"):
continue
self.send_header(k, v)
if streaming:
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.close_connection = True
self.send_header("Connection", "close")
self.end_headers()
try:
status = getattr(resp, "status", None) or resp.getcode()
self.send_response(status)
for k, v in resp.headers.items():
kl = k.lower()
if kl in _HOP_BY_HOP or kl in ("content-length", "content-encoding"):
continue
self.send_header(k, v)
if streaming:
self.send_header("Content-Type", "text/event-stream")
self.send_header("Cache-Control", "no-cache")
self.close_connection = True
self.send_header("Connection", "close")
self.end_headers()
try:
while True:
chunk = resp.read(8192)
if not chunk:
break
self.wfile.write(chunk)
self.wfile.flush()
except Exception as e:
vlog("stream relay ended: %s" % e)
else:
data = resp.read()
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
finally:
try:
while True:
chunk = resp.read(8192)
if not chunk:
break
self.wfile.write(chunk)
self.wfile.flush()
except Exception as e:
vlog("stream relay ended: %s" % e)
else:
data = resp.read()
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
resp.close()
except Exception:
pass

def _send_error(self, status: int, message: str):
body = json.dumps({"type": "error",
Expand Down
Loading