jev-ragcheck

RAG evaluation with typed decisions. It points to the exact sentence that is hallucinated and the passage behind each verdict, in one call per answer.

Give jev-ragcheck a question, the passages your retriever returned and the answer your model wrote. It tells you:

• which sentences of the answer the passages support, contradict or do not mention;
• which passage supports each sentence;
• whether each passage was relevant;
• whether the answer addresses the question.

Every verdict comes with a probability and with the exact character offsets of the sentence it judges.

Other evaluators ask an LLM to rewrite the answer into statements and then grade them in prose. jev-ragcheck works differently. Deterministic code splits the answer, and Jev, TypeSafe's typed-decision model, only chooses among fixed options. It never writes text.

import jev_ragcheck as rc

r = rc.check(
    question="When and where was Acme Tecnologia founded?",
    contexts=["Acme Tecnologia was founded in 1998 by Ana Lima in Curitiba."],
    answer="Acme Tecnologia was founded in 1999 in Curitiba. It also has an office in Lisbon.",
)
r.hallucinated, r.faithfulness          # (True, 0.0)
for c in r.claims:
    print(c.verdict, round(c.hallucination_probability, 2), (c.start, c.end), c.text)
# contradicted  1.0  (0, 48)   Acme Tecnologia was founded in 1999 in Curitiba.
# unsupported   1.0  (49, 81)  It also has an office in Lisbon.

Why jev-ragcheck

• Localised. Each verdict is attached to a span of the answer (answer[c.start:c.end] == c.text), and supported sentences cite the passage behind them. You can see where the answer goes wrong, not just a score.
• One request per answer. Faithfulness, passage relevance and answer relevance are all typed questions inside a single Jev call. There is no decomposition call, no per-statement verification loop and no JSON repair.
• Probabilities, not prose. Every verdict is a distribution over its options. You get a hallucination score for ranking, a threshold you can set, and a needs_review flag for close calls.
• Drop-in data format. It reads its own format or Ragas column names (user_input, retrieved_contexts, response). A CLI gate fails your CI when faithfulness drops.
• Portuguese and English. Both are tested, and ragcheck-bench, the new benchmark shipped here, is bilingual.
• Measured, not claimed. It is compared with Ragas, DeepEval, an LLM judge and HHEM on RAGTruth, HaluEval and ragcheck-bench. The results below include where jev-ragcheck loses.

How it works

flowchart LR
    Q[Question] --> ST[State<br/>question · numbered passages · numbered claims]
    P[Retrieved passages] --> ST
    A[Answer] --> C[Claims<br/>sentences with exact offsets]
    C --> ST
    ST --> R[Typed questions<br/>per claim: verdict + citation<br/>per passage: relevant?<br/>answer: full · partial · off-topic · refusal]
    R -->|1 call per sample| J{{Jev}}
    J --> PO[Policy<br/>P hallucinated = P contradicted + P unsupported<br/>threshold · review flags]
    PO --> M[(Verdicts and metrics)]

Loading

1. Split the answer into claims: sentences, semicolon clauses and list items, keeping exact offsets. Abbreviations are handled for Portuguese and English. Nothing is rewritten.
2. Ask Jev one typed request per sample:
   • for each claim, does the context support it, contradict it, leave it unsupported, or is it not a factual claim? and which passage supports it?;
   • for each passage, is it relevant to the question?;
   • for the answer, does it fully answer, partially answer, go off-topic or refuse?
3. Decide in code.
   • A claim is hallucinated when P(contradicted) + P(unsupported) ≥ threshold.
   • The answer's hallucination score is the highest such probability among its claims.
   • Faithfulness is the share of factual claims that are supported.
   • Context precision is rank-aware, as in Ragas.

Details: docs/architecture.md.

Install

pip install git+https://github.com/gabazureus/jev-ragcheck   # or: uv add git+https://github.com/gabazureus/jev-ragcheck
export TYPESAFE_API_KEY=...        # Jev directly, or
export OPENROUTER_API_KEY=...      # Jev through OpenRouter (typesafe/jev-1.13)
jev-ragcheck doctor                # one tiny live call to check the setup

Python 3.10+. The only runtime dependency is typesafe-sdk, the official client.

Use it

import jev_ragcheck as rc

samples = rc.load_samples("eval.jsonl")          # jev-ragcheck or Ragas column names
checker = rc.Checker(concurrency=8, cache_dir=".ragcheck-cache")
report = checker.evaluate(samples)               # or: await checker.aevaluate(samples)

report.summary()        # faithfulness, hallucination_rate, context_precision, context_relevance,
                        # answer_relevance, claims, hallucinated_claims, calls, cost_usd, latency
for r in report.results:
    for c in r.hallucinated_claims:
        print(r.sample.id, c.verdict, c.text)
rc.save_html(report, "report.html")              # the interactive page shown above

async for event in checker.astream(samples):     # per-sample events, for live UIs
    ...

A sample is a JSON object:

{"id": "q1", "question": "…", "contexts": ["passage 1", "passage 2"], "answer": "…", "reference": "optional"}

A complete runnable example is in examples/quickstart.py.

CLI and CI gate

jev-ragcheck eval eval.jsonl -o results.jsonl --html report.html
jev-ragcheck eval eval.jsonl --fail-under 0.9      # exit code 1 when mean faithfulness < 0.9
jev-ragcheck visualize results.jsonl -o report.html

Option	Default	What it does
decider	"jev"	"jev:<model>", or "llm:<openrouter model>" to answer the same typed questions with a chat LLM (pip install jev-ragcheck[llm])
metrics	all three	any of faithfulness, context_relevance, answer_relevance; fewer metrics make a smaller request
threshold	0.5	P(hallucinated) at which a claim is flagged
concurrency	8	samples in flight
context_char_limit	12,000	characters of retrieved context sent per sample
cache_dir	none	content-addressed cache of decisions
on_error	"raise"	"skip" keeps going when a sample fails

Benchmark

We compared jev-ragcheck with four other evaluators:

• Ragas 0.4.3 Faithfulness;
• DeepEval 4.2.6 Faithfulness;
• a do-it-yourself LLM judge that labels every sentence in one JSON call;
• HHEM-2.1-open, Vectara's local hallucination classifier.

The three LLM-based evaluators all run on Gemini 3.5 Flash-Lite. The datasets are:

• ragcheck-bench, new, 40 test items in Brazilian Portuguese and English, double-annotated at sentence, passage and answer level;
• RAGTruth, human word-level hallucination spans over QA, summarisation and data-to-text;
• HaluEval QA.

The seven criteria were fixed before scoring: detection quality, localisation, coverage, calibration, latency, cost and reliability (methodology).

Response-level balanced accuracy (%) for detecting hallucinated answers, and sentence-level F1 for localising them. Latency is per item; cost is API fees per 1,000 items:

Evaluator	ragcheck-bench BAcc	RAGTruth BAcc	HaluEval BAcc	RAGTruth sentence F1	p50 latency	$ / 1k items
jev-ragcheck (Jev 1.13)	100.0	78.8	86.7	51.4	0.3–0.5 s	$0.03–0.13
Ragas Faithfulness (Flash-Lite)	97.2	72.7	79.3	–	2.9–5.5 s	$0.78–3.19
DeepEval Faithfulness (Flash-Lite)	90.9	67.8	74.7	–	2.9–3.9 s	$0.85–2.26
LLM judge (Flash-Lite)	100.0	82.7	86.0	59.8	1.3–1.5 s	$0.11–0.42
HHEM-2.1-open (local CPU)	69.4	70.8	75.3	37.6	5.3–22.6 s	$0 (local)

What the numbers say, plainly:

• Cost and latency: jev-ragcheck wins on every dataset.
  • It is the cheapest and fastest hosted evaluator: $0.03–0.13 per 1,000 items at a median of 0.3–0.5 s.
  • On RAGTruth that is 24× cheaper and 11× faster than Ragas, and 3× cheaper and 3× faster than the LLM judge.
• Against Ragas, DeepEval and HHEM: jev-ragcheck is more accurate. It is significantly ahead on RAGTruth and HaluEval (paired bootstrap, p < 0.05).
• Against a plain LLM judge: a draw at the pre-registered settings.
  • The judge is a single JSON call on the same sentences.
  • On HaluEval the two tie: 86.7 against 86.0.
  • On RAGTruth the judge is ahead, 82.7 against 78.8, though not significantly. It also localises hallucinated sentences better (sentence F1 59.8 against 51.4).
• The gap on RAGTruth is the threshold, not the ranking.
  • jev-ragcheck's scores rank well: AUROC 93.1.
  • The default threshold of 0.5 flags too many long, clean answers.
  • A threshold chosen on RAGTruth's train split (0.9) lifts test balanced accuracy to 86.2. This analysis is post-hoc and reported as such. For your own data, set threshold from a few hundred labelled examples.
• Swapping Jev for Flash-Lite in the same pipeline lowers quality on every dataset and costs 16× more on RAGTruth. The decider matters.
• Beyond faithfulness (ragcheck-bench), jev-ragcheck judges passage relevance with 96.7% accuracy against 78.3% for the judge, and labels answer relevance with 72.5% accuracy against 50.0%. Detection on ragcheck-bench is saturated: its planted errors are easy for the strong methods.
• Reliability. jev-ragcheck is not deterministic. Two full runs agreed on 98% of RAGTruth answers and 99% of sentences, and on every HaluEval and ragcheck-bench decision.

When to use which

Situation	Better choice
Continuous evaluation or guardrails at volume, where cost and latency matter	jev-ragcheck
You need to see which sentence is wrong and which passage backs each one	jev-ragcheck
You also want passage relevance and answer relevance, in the same call	jev-ragcheck
You want a verdict explained in prose, or the best localisation on long answers at default settings	LLM judge
You already run Ragas or DeepEval faithfulness	either of the above: cheaper, faster and more accurate on our data
No network allowed	HHEM (local), accepting lower accuracy

Full tables, per-task scores, intervals, paired tests and the ablation are in benchmarks/README.md and the paper.

jev-ragcheck vs Ragas at a glance

	jev-ragcheck	Ragas / DeepEval faithfulness
Unit judged	the answer's own sentences, with offsets	statements an LLM rewrote from the answer
Calls per answer	1 (all metrics)	2 or more (decompose, then verify), plus calls for each extra metric
Output per claim	verdict distribution, citation, review flag	verdict and a free-text reason
Tells contradicted from unsupported	yes	no (both count as "not supported")
Score for ranking	a probability per claim and per answer	a ratio of counts
Model	Jev (typed decisions); any chat LLM as an ablation	any chat LLM
Breadth of metrics	faithfulness, context relevance and precision, answer relevance	many more (reference-based, agentic, custom)

Paper

One Call, Every Sentence: Localised RAG Evaluation with Typed Decisions. Gabriel Sorrentino, Federal University of São Carlos (UFSCar). PDF · LaTeX source. The tables and every number in the text are generated from benchmarks/results/*/results.json.

Project layout

src/jev_ragcheck/   library: claims, questions, resolve, deciders, engine, models, evaluation,
                    visualize, cli
tests/              offline test suite (fake decider; no network)
benchmarks/         datasets (incl. ragcheck-bench), method adapters, runner, report, charts,
                    paper tables, raw per-item results
examples/           quickstart and sample data
docs/               architecture and benchmark methodology
paper/              the paper (LaTeX + PDF)

Contributing

Contributions are welcome, especially new languages for the claim splitter and new benchmark adapters. Read CONTRIBUTING.md first. Two rules matter most:

• verdicts must point at the answer's own text;
• nothing is tuned on benchmark test items.

Please follow the Code of Conduct, and report security issues privately as described in SECURITY.md.

License

MIT © 2026 Gabriel Sorrentino. See LICENSING.md for dependencies, hosted-model terms, benchmark data licenses and trademarks.

Credits

Author and maintainer: Gabriel Sorrentino, Computer Engineer, Federal University of São Carlos (UFSCar).

jev-ragcheck builds on the work of others:

• Jev and the official typesafe-sdk and system-one-adapter by TypeSafe AI.
• Ragas and DeepEval, which defined how RAG faithfulness is measured in practice, and HHEM by Vectara.
• Alden DoRosario's single-claim verification with Jev on LLM-AggreFact, the prior evidence that typed decisions can verify grounded claims.
• RAGTruth and HaluEval, used in the benchmark.
• The open-source Jev community catalogued in awesome-jev and awesomejev.com, and jevextract, this project's sibling for extraction.

If you use jev-ragcheck or ragcheck-bench, please cite it (CITATION.cff).