A text-to-SQL agent that decides where to look before it thinks
A fast System One model picks the databases and tables, so the LLM only has to write the query.
When a question could be answered by any of dozens of databases, a standard agent pays for it on every question.
|
It has to read every schema before it can start. With 30 databases, that is over 66,000 input tokens per question. |
When the data does not exist, it still finds something close and answers anyway: 1 in 5 times in our benchmark. |
A long prompt on every call means a slower executor and a bigger bill, even for a simple count. |
The agent treats it as a routing problem before a generation problem. In Thinking, Fast and Slow, Daniel Kahneman describes System 1, fast and intuitive, and System 2, slow and deliberate. TypeSafe AI borrowed the name for System One models such as Jev, which make typed decisions instead of generating text. Here System 1 routes and System 2 writes the SQL.
- Pick databases: Jev answers one yes/no question per database ("does it contain the data?") in a single call.
- Pick tables: a second Jev call scores every table of the chosen databases.
- Load columns: only the chosen tables' columns are read from Postgres and preloaded into the prompt.
- Write the SQL: the LLM writes read-only SQL against the routed databases. Every query runs inside a
READ ONLYtransaction with a statement timeout, and is always rolled back.
When no database clears the threshold, the agent says the data is not available and skips the LLM entirely. See docs/features.md for how each piece works.
100 questions written like a real user would ask them, without looking at the schemas. 75 can be answered with the data and 25 cannot (out-of-scope topics or data that is not loaded), so the benchmark also measures hallucinations. Both agents use openai/gpt-6-sol (through OpenRouter) as the executor and see the same 30 Postgres databases.
| Metric | With Jev routing | Standard agent |
|---|---|---|
| Accuracy on answerable questions | 59.7% | 62.7% |
| Hallucination rate on unanswerable questions | 0.0% | 20.0% |
| Out-of-scope questions answered correctly | 100% | 71.4% |
| Average input tokens per question | 11,071 | 66,665 |
| Estimated cost per question (list price) | $0.026 | $0.138 |
| Latency p50 | 13.6 s | 14.3 s |
| Executor time (average) | 11.6 s | 16.7 s |
| Latency p95 | 42.2 s | 37.9 s |
- It never hallucinated. On the 25 questions the data cannot answer, the routed agent always said so.
- Overall it is more accurate: 70.1% vs 67.0%. It is 3 points behind on answerable questions, ahead on counts, lookups and rankings, and behind on ambiguous questions, where reading every schema helps guess what the user meant.
- Routing cuts input tokens by ~6x and makes the executor ~30% faster.
- It is cheaper, by less than the tokens suggest. About 5x at list price, roughly 2-3x billed in spot checks, because the standard agent's repeated prompt benefits from prompt caching.
- The p95 is dominated by AI Gateway retries during outages, not by Jev, which spends about 0.15 s per call inside the provider.
Charts: accuracy, tokens and latency
Note
Treat these as a baseline, not a verdict: thresholds are not tuned yet, 3 routed runs failed on gateway outages (excluded), and 27 answers were flagged for manual review. Earlier runs with gpt-5.6-luna showed the same token savings but a larger accuracy gap (49% vs 61%), so a stronger executor narrows the difference. p50 is the median; p95 means 95% of questions were faster.
A small TypeScript service with a plain HTML UI. Jev decides, the executor LLM writes SQL, Postgres answers.
| Component | Technology | Description |
|---|---|---|
| Agent | Node.js 22+ run with tsx. Routing, execution and the HTTP server in src/. |
|
| Router | Jev through the Vercel AI Gateway. | |
| Executor |  |
Any OpenAI-compatible model via openai: or openrouter:, with the AI SDK. |
| Database | PostgreSQL, read-only transactions with a statement timeout. | |
| UI | Side-by-side comparison at / and the benchmark runner at /bench. |
- Node.js 22+ and pnpm
- A PostgreSQL server with the databases you want to query
- An AI Gateway key (for Jev) and an OpenAI or OpenRouter key (for the executor)
git clone https://github.com/JoseVelazcoH/system-one-sql-agent.git
cd system-one-sql-agent
pnpm install
cp config.example.yaml config.yaml # your model, databases and benchmark files
cp .env.example .env # your API keys and database password
pnpm catalog # build catalog.json from your databasescatalog.json holds a short description of every database and table. Routing quality depends on it: review the generated descriptions and rewrite the weak ones by hand. Your edits survive the next pnpm catalog.
pnpm dev # UI on http://localhost:3000/compares both agents on the same question./benchruns the benchmark (5, 20 or 100 questions), shows the charts and per-question results, and exports them to PDF or PNG.
From the command line:
pnpm ask "¿Cuántos habitantes tiene Jalisco?"
pnpm bench # run the benchmark in both modes
pnpm bench:grade # grade the latest run, write the report and charts
pnpm bench:refresh # re-run the gold SQL to check the answers are still validYour model, router thresholds, databases and secrets live in config.yaml and .env, both gitignored. See docs/configuration.md for every option, the environment variables and how to bring your own benchmark.
- Jev and System One models by TypeSafe AI: the router behind every decision
- AI SDK and AI Gateway by Vercel: model access for both systems
- Thinking, Fast and Slow by Daniel Kahneman: the idea behind the name
Important
The UI has no authentication and /bench can start paid LLM runs, so it only listens on localhost by default. Put it behind authentication before exposing it with HOST=0.0.0.0.
Released under the GPL-3.0 license.