Routing-as-a-service for AI agents An agent with 60+ tools can't carry every schema in its prompt — token cost explodes and selection accuracy collapses among look-alike tools. Our router sits between the query and the catalog: register your tools once (bulk JSON, or auto-import from any MCP server / OpenAPI spec — all validated by a strict intake guardrail), then every query returns the minimal, ordered, high-precision tool plan the agent should see.
Measured on the organizers' harness (29 cases, 64-tool catalog):
| Configuration | Accuracy | Token savings |
|---|---|---|
| Organizer baseline | 7/29 (24%) | 94% |
| Ours — local-only (zero LLM, offline) | 24/29 (83%) | 98% |
| Ours — hybrid (LLM planner + HyDE + scope-arbiter) | 26/29 (90%) | 98% |
Plus: registry guardrail 11/11 fixtures · catalog-churn scenarios 8/8 (add / remove / replace / version / scale 64→104) · held-out generalization set 10/12 (fresh unscripted queries — no overfitting).
How it works: fused local retrieval (TF-IDF/dense embeddings + BM25 + capability signals +
side-effect/version/PII penalties, all combined into one score) → version/duplicate collapse +
confidence gate (dynamic-K, clarify/refuse) → one LLM call for final ordering/repeat-counts/
version reasoning. On low-confidence queries it adds conditional HyDE (hypothetical-doc
retrieval) for recall, with the LLM as an explicit scope arbiter so out-of-scope queries
still return nothing. A deterministic local planner sits beneath it all, so the engine never
crashes and runs fully offline if no key is present. One engine, three entry points: Python
class (the grader), REST API (any language), MCP tool find_tools (any agent).
- docs/design/architecture.md — master architecture + diagram
- docs/design/engine-design.md — engine internals & cost model
- docs/eval_report.md — measured iteration trajectory (24%→90%)
Python 3.11+ · pydantic · platformdirs · FastAPI/uvicorn (REST) · MCP SDK · provider-agnostic
LLM/embedding layer — local TF-IDF is the default (no LLM, fully offline), Gemini is
live-verified, an AWS Bedrock provider exists in code but is not live-tested · Docker.
Lean core — hosted providers are optional extras; the engine runs with zero external
dependencies out of the box. Storage is a plain JSON file cache (no vector DB — see
docs/design/storage.md for why that's the right call at this scale).
# 1. install (venv recommended)
python3 -m venv .venv && .venv/bin/pip install -e ".[api,mcp]"
# 2. (optional) configure an LLM — engine runs fully offline without it
.venv/bin/tool-router init # wizard: provider, key (hidden), model
# or: export GEMINI_API_KEY=... TOOL_ROUTER_LLM_PROVIDER=gemini \
# TOOL_ROUTER_LLM_MODEL=gemini-flash-lite-latest
# 3. run the REST service (docs at /docs, config page at /setup)
.venv/bin/tool-router serve --catalog path/to/tools_catalog.json
curl -X POST localhost:8080/route -H 'content-type: application/json' \
-d '{"query": "Pull last quarter revenue and email the chart to finance"}'
# or as an MCP server (find_tools / register_tools):
.venv/bin/tool-router mcp --catalog path/to/tools_catalog.json
# or Docker:
docker build -t tool-router . && docker run -p 8080:8080 tool-router
# score it on the organizers' benchmark:
cd <testkit>/harness && PYTHONPATH=<this-repo> python3 run_benchmark.py \
--router tool_router.adapters.harness_router:Router
# auto-import a catalog instead of hand-writing one:
.venv/bin/tool-router import-mcp --command <mcp-server-cmd> -o tools.json
.venv/bin/tool-router import-openapi openapi.json -o tools.jsonReal vs. mocked: the router, guardrail, churn engine, REST/MCP surfaces, importers and persistence are fully real and tested. The 64 catalog tools are mock definitions (the organizers' testkit) — the router selects tools; it never executes them.