Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,20 @@ project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- Expanded built-in priors to 130 current OpenRouter models across 23 providers.
- Added reproducible profile generation from an official Models API snapshot.
- Added profile provenance, modalities, context length, feature support, and
token-price metadata.

### Changed

- Default routing now uses a bounded 14-model cross-provider shortlist instead
of every bundled profile.
- Hard-budget estimates now use a documented 2,000-input/1,000-output token
reference request.

## [0.1.0] - 2026-09-20

### Added
Expand Down
5 changes: 5 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ pytest
Do not commit API keys, routing histories containing private prompts, generated
build artifacts, or model-profile claims without a traceable source.

For catalog changes, follow
[`docs/MODEL_PROFILES.md`](docs/MODEL_PROFILES.md). Profile updates must include
the snapshot provenance, a deterministic regeneration check, and tests for the
new catalog invariants.

## Design principles

- Keep routing training-free and simple to adopt.
Expand Down
57 changes: 42 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,9 @@ from reflexroute import Router

router = Router(
models=[
"openai/gpt-5.6",
"google/gemini-flash",
"anthropic/claude-sonnet",
"openai/gpt-5.6-sol",
"google/gemini-3.8-flash",
"anthropic/claude-sonnet-5",
]
)

Expand All @@ -91,7 +91,7 @@ result = router.route("Implement a distributed rate limiter.")
History can be JSONL or CSV. Its canonical shape is:

```json
{"query":"Solve this math problem...","model":"openai/gpt-5.6","performance":1.0,"cost":0.018,"latency":1.42}
{"query":"Solve this math problem...","model":"openai/gpt-5.6-sol","performance":1.0,"cost":0.014,"latency":1.42}
```

`query`, `model`, and numeric `performance` are required. `cost` and `latency`
Expand Down Expand Up @@ -125,16 +125,18 @@ is never asked to perform budget arithmetic.
- If nothing remains, ReflexRoute raises `BudgetError`.
- If one candidate remains, ReflexRoute returns it locally without an API call.

Bundled costs are illustrative. Production deployments should maintain a
profile file with estimates appropriate to their workloads.
`estimated_cost` is reproducibly calculated from the catalog's token prices
using a reference request of 2,000 input and 1,000 output tokens. It excludes
cache, images, audio, web search, and provider-specific fees. Production
deployments should maintain estimates appropriate to their real workloads.

## CLI

```bash
reflexroute route "Write a CUDA kernel"

reflexroute route \
--models openai/gpt-5.6 google/gemini-flash \
--models openai/gpt-5.6-sol google/gemini-3.8-flash \
"Write a CUDA kernel"

reflexroute route \
Expand All @@ -149,11 +151,11 @@ limit. Quote multi-word queries.
Typical output:

```text
Selected: openai/gpt-5.6
Selected: openai/gpt-5.6-sol

openai/gpt-5.6 0.67
google/gemini-flash 0.21
model-c 0.12
openai/gpt-5.6-sol 0.67
google/gemini-3.8-flash 0.21
model-c 0.12

Confidence: 0.74
```
Expand All @@ -174,7 +176,7 @@ Jev receives structured state rather than a chat prompt:
"target_query": "...",
"candidate_models": {
"model-a": {
"prior": {"description": "...", "profile_version": "0.1.0"},
"prior": {"description": "...", "profile_version": "2026-09-20.1"},
"historical_evidence": []
}
}
Expand All @@ -188,9 +190,34 @@ rather than invented strengths.

## Model profiles

ReflexRoute includes **130 model profiles across 23 OpenRouter providers** as
of 2026-09-20. The catalog includes OpenAI, Anthropic, Google, Qwen, DeepSeek,
Mistral, Z.ai, Meta, xAI, Moonshot, MiniMax, Amazon, NVIDIA, Perplexity,
Cohere, ByteDance, and selected specialists.

Bundled profiles live in
[`reflexroute/profiles/models.yaml`](reflexroute/profiles/models.yaml). Supply a
deployment-specific file with `Router(profile_path="my-models.yaml")`:
[`reflexroute/profiles/models.yaml`](reflexroute/profiles/models.yaml). Each one
records:

- A concise routing description and conservative strength tags.
- Context length and input/output modalities.
- Tool use, structured output, and reasoning support.
- Prompt and completion price per million tokens at snapshot time.
- Reproducible reference-request cost plus heuristic cost/latency tiers.
- Snapshot version and official source provenance.

`Router()` uses a bounded 14-model cross-provider default shortlist so that a
cold-start request does not become a 130-way Jev decision. Every bundled model
remains available when explicitly named:

```python
router = Router(models=["openai/gpt-6-astra", "deepseek/deepseek-v4.1-flash"])
```

See [Model profile methodology](docs/MODEL_PROFILES.md) for provider coverage,
field provenance, heuristics, and update instructions.

Supply a deployment-specific file with `Router(profile_path="my-models.yaml")`:

```yaml
profile_version: "2026-09-20"
Expand All @@ -214,7 +241,7 @@ context, making profiles traceable and extensible.

| API | Purpose |
| --- | --- |
| `Router(models, history_top_k=4)` | Create a router; omit `models` to use every bundled profile. |
| `Router(models, history_top_k=4)` | Create a router; omit `models` to use the bounded default shortlist. |
| `router.load_history(path)` | Replace history from a JSONL or CSV file. |
| `router.set_history(records)` | Replace history from Python objects. |
| `router.eligible_models(budget)` | Inspect the result of hard budget filtering. |
Expand Down
99 changes: 99 additions & 0 deletions docs/MODEL_PROFILES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Model profile methodology

ReflexRoute ships a broad but bounded snapshot of the models available through
OpenRouter. Profiles are routing priors—not benchmark verdicts, guarantees, or
live service telemetry.

## Current coverage

The `2026-09-20.1` snapshot contains 130 non-batch, non-free model IDs across
23 providers:

| Provider | Profiles |
| --- | ---: |
| `openai` | 18 |
| `qwen` | 16 |
| `google` | 12 |
| `anthropic` | 10 |
| `deepseek` | 8 |
| `mistralai` | 8 |
| `z-ai` | 8 |
| `meta-llama` | 5 |
| `moonshotai` | 5 |
| `perplexity` | 5 |
| `x-ai` | 5 |
| `amazon` | 4 |
| `bytedance-seed` | 4 |
| `cohere` | 4 |
| `minimax` | 4 |
| `nvidia` | 4 |
| `inclusionai` | 2 |
| `inference-net` | 2 |
| `sakana` | 2 |
| `inception` | 1 |
| `microsoft` | 1 |
| `prism-ml` | 1 |
| `unbiased` | 1 |

Batch variants and `:free` duplicates are omitted to keep model choices
distinct. Dynamic OpenRouter meta-routers are also omitted because their costs
cannot be represented as one hard-budget estimate.

## Field provenance

Directly sourced from the official OpenRouter Models API snapshot:

- Model ID and display name.
- Context length.
- Input and output modalities.
- Supported request parameters.
- Prompt and completion token prices.
- Official description, used only as input to conservative tag inference.

Derived by ReflexRoute:

- `description`: a compact summary of structured metadata and inferred focus.
- `strengths`: keyword- and capability-based routing tags.
- `latency_level`: a family-name heuristic, not measured latency.
- `cost_level`: a tier derived from the reference-request estimate.
- `estimated_cost`: token-price estimate for 2,000 input plus 1,000 output
tokens.
- `default_candidate`: membership in the bounded cold-start shortlist.

Cost tiers use that reference-request estimate: `low` is at most $0.001,
`medium` at most $0.01, `high` at most $0.05, and `premium` above $0.05.
Zero-priced models would use `free`, although free duplicates are not part of
this snapshot.

The estimate excludes prompt caching, images, audio, web search, and
provider-specific fees. Users with different traffic shapes should maintain a
deployment-specific profile file.

## Updating the snapshot

Profile updates are explicit and reviewable. The generator never fetches the
network itself:

```bash
curl -fsSL https://openrouter.ai/api/v1/models -o /tmp/openrouter-models.json
python scripts/build_model_profiles.py \
--catalog /tmp/openrouter-models.json
pytest
```

To verify deterministic regeneration against the same snapshot:

```bash
python scripts/build_model_profiles.py \
--catalog /tmp/openrouter-models.json \
--check
```

When updating, review `SELECTED_MODELS`, `DEFAULT_MODELS`, all inferred tags,
the snapshot date/version, and the generated diff before committing.

## Precedence

Profiles intentionally remain weak priors. When relevant user history conflicts
with a bundled profile, the Jev instruction tells the model to prefer the user
evidence.
6 changes: 3 additions & 3 deletions examples/basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@


models = [
"openai/gpt-5.6",
"google/gemini-flash",
"anthropic/claude-sonnet",
"openai/gpt-5.6-sol",
"google/gemini-3.8-flash",
"anthropic/claude-sonnet-5",
]

# Zero-shot: built-in, explicitly versioned priors only.
Expand Down
12 changes: 6 additions & 6 deletions examples/history.jsonl
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{"query":"Implement a distributed rate limiter with Redis and discuss failure modes.","model":"openai/gpt-5.6","performance":0.97,"cost":0.028,"latency":2.31}
{"query":"Debug a concurrent Python worker pool with intermittent deadlocks.","model":"openai/gpt-5.6","performance":0.95,"cost":0.027,"latency":2.08}
{"query":"Summarize this customer support email in three bullets.","model":"google/gemini-flash","performance":0.94,"cost":0.002,"latency":0.42}
{"query":"Classify a short product review by sentiment.","model":"google/gemini-flash","performance":0.92,"cost":0.001,"latency":0.31}
{"query":"Rewrite a sensitive policy memo with a clear and diplomatic tone.","model":"anthropic/claude-sonnet","performance":0.96,"cost":0.014,"latency":1.67}
{"query":"Review this TypeScript API design and suggest maintainable abstractions.","model":"anthropic/claude-sonnet","performance":0.93,"cost":0.015,"latency":1.75}
{"query":"Implement a distributed rate limiter with Redis and discuss failure modes.","model":"openai/gpt-5.6-sol","performance":0.97,"cost":0.014,"latency":2.31}
{"query":"Debug a concurrent Python worker pool with intermittent deadlocks.","model":"openai/gpt-5.6-sol","performance":0.95,"cost":0.013,"latency":2.08}
{"query":"Summarize this customer support email in three bullets.","model":"google/gemini-3.8-flash","performance":0.94,"cost":0.005,"latency":0.42}
{"query":"Classify a short product review by sentiment.","model":"google/gemini-3.8-flash","performance":0.92,"cost":0.004,"latency":0.31}
{"query":"Rewrite a sensitive policy memo with a clear and diplomatic tone.","model":"anthropic/claude-sonnet-5","performance":0.96,"cost":0.014,"latency":1.67}
{"query":"Review this TypeScript API design and suggest maintainable abstractions.","model":"anthropic/claude-sonnet-5","performance":0.93,"cost":0.015,"latency":1.75}
2 changes: 2 additions & 0 deletions reflexroute/profiles.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ def load_profiles(path: str | Path | None = None) -> dict[str, ModelProfile]:
"estimated_cost_per_request",
"profile_version",
"source",
"default_candidate",
}
result[name] = ModelProfile(
model=name,
Expand All @@ -62,6 +63,7 @@ def load_profiles(path: str | Path | None = None) -> dict[str, ModelProfile]:
estimated_cost=estimated_cost,
profile_version=_optional_string(raw_profile.get("profile_version", version)),
source=raw_profile.get("source", source),
default_candidate=raw_profile.get("default_candidate") is True,
extra={key: value for key, value in raw_profile.items() if key not in known},
)
return result
Expand Down
Loading
Loading