Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ __pycache__/
dist/
build/
.env
benchmarks/results/**/decisions.jsonl
30 changes: 30 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,23 @@ chat completion to make the routing decision.
- **Auditable decisions.** Inspect the complete state, question, and opaque
choice mapping before sending anything.

## Small result

On a 191-query complete-case slice of
[LLMRouterBench](https://github.com/ynulihao/LLMRouterBench), adding K=4
retrieved observations per candidate improved ReflexRoute's mean quality from
`0.5497` to `0.6832` (+24.3% relative). Routing remained inexpensive:

| Mode | Mean quality | Router latency P50 | Router cost / 1,000 decisions |
| --- | ---: | ---: | ---: |
| Zero-shot | 0.5497 | 722 ms | $0.108 |
| Few-shot, K=4 | 0.6832 | 1,333 ms | $0.403 |

This is a small exploratory measurement of ReflexRoute's two modes, not a
comparison against other routers. Candidate outputs were cached, so the run
only paid `$0.09757` for 382 Jev decisions. See the
[setup, cost definition, caveats, and reproduction command](benchmarks/README.md).

## Install

ReflexRoute requires Python 3.10 or newer.
Expand All @@ -59,6 +76,19 @@ Set your OpenRouter API key:
export OPENROUTER_API_KEY="your-key"
```

## 60-second demo

The demo routes one query twice: first from built-in priors, then after adding
eight illustrative user observations.

```bash
python examples/adaptation_demo.py
```

It prints the selected model, distribution, confidence, routing latency, and
the Jev charge for each decision. It makes two Decisions API calls but does not
invoke either selected model.

## Quick start

### Zero-shot routing
Expand Down
61 changes: 61 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Small benchmark

This is a deliberately small, low-cost measurement of ReflexRoute itself. It
shows the cold-start and few-shot modes without claiming a comparison against
other routing systems.

## Result

| Mode | Mean quality | Router latency P50 / P95 | Router cost / 1,000 decisions | Selected-model cost / 1,000 queries |
| --- | ---: | ---: | ---: | ---: |
| Zero-shot | 0.5497 | 722 / 1,376 ms | $0.108 | $11.786 |
| Few-shot, K=4 | 0.6832 | 1,333 / 2,033 ms | $0.403 | $18.111 |

Adding four retrieved observations per candidate increased mean quality by
0.1335, or 24.3% relative to the zero-shot result. The complete run made 382
Jev decisions and incurred $0.09757 in actual routing charges.

These are exploratory measurements, not broad quality or latency guarantees.

## Setup

- Source: [LLMRouterBench](https://github.com/ynulihao/LLMRouterBench) cached
model outcomes.
- Evaluation: 191 validation queries with complete quality and cost records for
all four candidates.
- Candidates: Llama 3.1 8B Instruct, DeepSeek R1 0528, Gemini 2.5 Flash, and
Gemini 2.5 Pro.
- Few-shot context: 256 training queries, producing 1,024 observations; local
TF-IDF retrieves K=4 observations independently for each candidate.
- Quality: mean of the benchmark's per-query score, normalized to [0, 1].
- Cost: the router column is the amount billed for Jev decisions; selected-model
cost comes from the benchmark's cached executions. No candidate model was
called again for this run.
- Latency: wall-clock routing latency only. It excludes candidate-model
generation latency and was measured from one network location on 2026-09-20.

The machine-readable aggregate is in
[`results/mini_v1/summary.json`](results/mini_v1/summary.json). Per-query
checkpoints are intentionally gitignored.

## Reproduce

Install the benchmark dependency:

```bash
python -m pip install -e '.[benchmark]'
```

Prepare `train.pkl` and `val.pkl` dense tables under `data/llmrouterbench`, or
pass another directory. Each row needs `sample_id`, `eval_name`, `query_text`,
and the performance/cost columns used in the script.

```bash
export OPENROUTER_API_KEY="your-key"
python benchmarks/mini_llmrouterbench.py \
--data-dir data/llmrouterbench \
--output-dir benchmarks/results/mini_v1
```

Completed decisions are appended to `decisions.jsonl`, so rerunning the same
command resumes without paying for successful calls again.
Loading
Loading