A retrieval-augmented question answering engine that refuses to answer beyond its sources, and proves it with a test suite that runs without an API key.
The interesting question about a RAG system is not whether it can answer things. It is what it does when it cannot, and how you would know. This repository is an answer to that question with numbers attached.
- How it refuses
- The measurement
- Quick start
- Design decisions
- The evaluation set
- Testing strategy
- Limitations
- How AI was used
Three layers, in the order a question meets them. Two of the four stages are the model's, and neither is trusted.
question
│
├─ retrieve ours, deterministic
│
├─ LAYER 1: gate ours, deterministic ── refuse ──▶ the model is never called
│
├─ generate the model's
│
├─ LAYER 2: the model's own "I could not answer"
│
└─ LAYER 3: verify ours, deterministic ── refuse ──▶ the answer is discarded
The most reliable way to stop a language model inventing an answer is not to ask it.
The usual approach is a strongly worded system prompt: answer only from the context, say you don't know otherwise. That instruction is necessary and it is not sufficient, because it is a request. It competes with everything else in the context window, and its failure mode is a confident, well-formatted, wrong answer. Worse: whether it held on any given call is not something you can assert in a test without calling the model.
The gate is a comparison between two numbers. When nothing in the corpus is close enough to the question, the request stops there — no tokens spent, no network call, and a refusal that is a property of the code rather than of the model's mood.
For questions that do retrieve something plausible, the versioned prompt
(prompts/answer.v1.md) requires the model to set
answered: false when the passages do not contain the answer. The reply shape is
enforced by the API's structured output rather than requested in prose, so a
malformed reply is not one of the things that can go wrong.
This layer is trusted in one direction only: "I could not answer" is believed, "I answered" still has to survive layer 3.
Every answer must cite the passages it came from, by id. The verifier asks one question the code can answer with certainty:
Does every passage this answer cites actually exist in the context we supplied?
A model that invents a citation has left its sources, and that is detectable without understanding a word of what it wrote. A failing answer is replaced by a refusal, not shown with a warning — a warning next to a fluent paragraph gets read as a formality.
The verifier also catches the id that exists in the corpus but was not in this request's context, which a naive check against the corpus would pass.
Here is why there are three layers and not one. Every question in the evaluation set, with the similarity score of its best-matching passage:
Questions the corpus answers (lowest first)
| Score | Best passage | Question |
|---|---|---|
| 0.172 | 02-borrowing#loan-periods |
How long can I borrow a DVD for? |
| 0.224 | 01-membership#renewal-and-expiry |
How much does a Visitor card cost each year? |
| 0.284 | 03-fines-and-fees#lost-and-damaged-items |
After how many days overdue is an item declared lost? |
| 0.302 | 03-fines-and-fees#overdue-charges |
What is the daily overdue charge per item? |
| … | ||
| 0.608 | 02-borrowing#interlibrary-loan |
How many interlibrary loans can I request per year? |
Questions it does not (highest first)
| Score | Best passage | Question |
|---|---|---|
| 0.211 | 03-fines-and-fees#appeals |
Who is the head librarian? |
| 0.086 | 03-fines-and-fees#borrowing-suspension |
What is the library's annual budget? |
| 0.066 | 01-membership#who-can-join |
Is there parking outside the building? |
| 0.062 | 03-fines-and-fees#lost-and-damaged-items |
Can I get a refund on my membership if I move away? |
| 0.000 | — | What is the capital of France? |
Two things fall out of this table, and both are load-bearing.
There is a clean gap between 0.086 and 0.172. The threshold is set at 0.13, its midpoint — not chosen by taste, measured. Seventeen of eighteen questions are decided correctly by that one number, before any model runs.
And then there is "Who is the head librarian?" at 0.211. The Appeals section mentions a duty librarian, so the question matches it — and it matches it better than the lowest genuine question in the set scores. No threshold separates those two. Not a badly chosen one: none, because the ordering itself is wrong.
That is not a tuning problem, and pretending a better number would fix it would
be the dishonest move. It is the case layers 2 and 3 exist for, it is marked as
requiring a model in eval/questions.json, and the test
suite skips it rather than counting it as a pass when no model is present.
Everything below works with no API key. That is deliberate: retrieval, the gate and the citation verifier are local, so the part worth seeing is the part that needs nothing.
npm ci
npm run evalPASS What is the daily overdue charge per item?
PASS How many holds can a member have active at once?
...
PASS What is the wifi password?
refused: no_relevant_context
SKIP Who is the head librarian?
PASS What is the capital of France?
refused: no_relevant_context
17 passed, 0 failed, 1 skipped (of 18)
Ask it something:
npm run ask -- "How many holds can I have at once?"
npm run ask -- "What is the wifi password?"The second one prints a refusal, the retrieved passages and their scores — so you can see why it refused rather than taking its word for it.
cp .env.example .env # then set ANTHROPIC_API_KEY
npm run ask -- "How much does it cost to replace a lost card?"Generation uses Claude Opus 5 through the Anthropic SDK, with the reply shape constrained by structured outputs. Without a key the engine falls back to an extractive stand-in that quotes the passage it was given: honest about its sources, not a language model, and enough to exercise every layer except the wording.
| Command | What it does |
|---|---|
npm run ingest |
Chunk the corpus and write index.json |
npm run ask -- "..." |
Ask one question |
npm run eval |
Run the evaluation set and print the report |
npm test |
Unit tests plus the evaluation set |
Markdown headings are free semantic boundaries — the author already said where one idea stops. A 500-character sliding window ignores that and routinely puts the end of one policy and the start of an unrelated one in the same chunk, which is precisely the passage that then gets retrieved and quoted for both.
Tables are kept whole even when they overshoot the size target. The row
| DVDs and Blu-ray | 7 days | 1 | retrieved without its header says nothing —
a reader cannot tell whether 7 is days, renewals, or a shelf number.
Context comes from the heading trail, not from overlap. Prefixing
Borrowing > Holds costs a few words, restores what the heading carried, and
doubles as the human-readable citation id.
02-borrowing#holds, not chunk_7f3a. That id is what the model is asked to
cite and what the verifier checks, so when a citation is wrong it should be
wrong in a way a person reading the logs sees immediately.
Ingestion fails on a duplicate id rather than warning, because a duplicate makes a citation ambiguous and quietly weakens the guardrail everything rests on.
Anthropic publishes no embeddings endpoint — the Claude API's supporting endpoints are Batches, Files, Token Counting and Models. A semantic vector here would mean a second vendor, a second key, and a network call inside the one code path this project exists to make verifiable.
A TF-IDF model is fitted from the corpus in milliseconds, needs no credentials,
and makes every test in this repository bit-for-bit reproducible. The
Embedder interface is where a real model would
go; nothing above embedding/ would move.
What it costs is real — see Limitations.
It is not linguistically correct and does not need to be. The only property that
matters is that a question and the passage answering it fold the same way:
expires and expire must land on one token, and it does not matter that the
token is expir.
This mattered more than it sounds. Before -ing stripping, "how long can I
borrow a DVD" missed the section headed Borrowing entirely — three of the
ten answerable questions were being refused for it.
prompts/answer.v1.md shows up in git log as its own
change, so a reworded instruction reviews as what it is: a behaviour change. The
version is recorded in every evaluation report, because "refusal accuracy was
100%" is not a fact about the system — it is a fact about the system at a
prompt version.
Every path out of ask() returns an AskResult. An unreachable model, a
malformed reply and an unanswerable question are all conditions this system
expects to meet, and the right response to each is a refusal with a named reason
— not an exception for a caller to guess at.
The refusal reasons are distinct strings, not one generic apology: "nothing on that" and "found something that did not answer it" send a reader to different next steps.
eval/questions.json — ten questions the corpus answers,
eight it does not. The second eight are the point. A retrieval system is easy to
make look good on the first ten.
The corpus is the operating manual of a fictional public library, written for
this repository: membership, borrowing, fines, and spaces. It has concrete
figures — 0.50 BOB per day overdue, a 20.00 BOB cap, six study rooms — so
that an invented answer is obviously invented rather than merely unverifiable.
109 tests across 6 files, 97% line and 88% branch coverage, thresholds enforced
in vitest.config.ts. No network, no API key, no fixtures
that need refreshing.
The tests worth reading are in
tests/unit/engine.test.ts. Each one takes a
specific way a model can go wrong and asserts the engine refuses:
| The model… | Asserted outcome |
|---|---|
| is never given the chance (out-of-corpus question) | refused, and NeverCalledClient proves no call was made |
| cites a passage that does not exist | fabricated_citation |
| cites a real passage that was not in this context | fabricated_citation |
| asserts an answer with no citation at all | missing_citation |
| claims an answer and returns an empty string | unparseable_response |
| reports it could not answer | model_reported_no_answer |
| cannot be reached at all | refused, not thrown |
throws a TypeError (a bug in this repo) |
rethrown, because containment is for the model being unavailable, not for our own bugs |
These need a model that misbehaves on demand, which is what the scripted test
doubles in src/generation/scripted.ts are for.
Waiting for a real model to invent a citation is not a test strategy.
Stated plainly, because a reviewer will find them anyway:
- Lexical retrieval cannot match a synonym. A question phrased entirely in other words — "what happens if I bring a book back late" against a section that only ever says overdue — scores low and gets refused. That is the safe direction to fail in, and it is still a failure. A real embedding model behind the same interface fixes it.
- The verifier checks that a citation exists, not that it supports the claim. An answer citing a real passage and then misstating what it says passes. Doing better means checking a claim against a source, which is the original problem one level down.
- The thresholds are fitted to this corpus. 0.13 is the midpoint of a gap measured on 18 questions over 16 chunks. Another corpus needs another number, and the evaluation set is how you would find it.
- One turn, no conversation. No history, no follow-ups, no pronoun resolution back to a previous question.
- The corpus is 16 chunks and the search is a flat scan. Correct at this size and wrong at a hundred thousand, where this would need a real vector store.
- The
minMarginknob is off by default. It is implemented and tested, and on this corpus it does nothing. A knob tuned to do nothing is more honest than one tuned to look busy.
This repository was built with Claude Code as an active participant, and
AI-WORKFLOW.md documents that honestly: what was decided
before any code, what the model got wrong, and the measurement that changed the
design.
MIT — see LICENSE.