Index: corpus/index (records sha256 54892a95fbb0…) · provider: mock · repeats: 3 · date: 2026-10-06 17:49 UTC
| metric | value | n | 95 % CI (Wilson) | note |
|---|---|---|---|---|
| pass rate (all cases) | 1.000 | 150/150 | [0.975, 1.000] | |
| unsafe verdicts (forbidden status emitted) | 0.000 | 0/150 | [0.000, 0.025] | target 0 |
| recall@found | 1.000 | 89/89 | [0.959, 1.000] | expected found → got found with expected record |
| precision@found | 1.000 | 89/89 | [0.959, 1.000] | got found → found was expected |
| forbidden-lexicon hits in our strings | 0 | — | — | target 0 |
| latency per case (ms) | p50 8 · p95 39 · max 52 | — | — | in-process, incl. mock extraction |
| output variance across repeats | 0 cases differed | — | — | extraction variance only (core is deterministic) |
false-alarm rate (verbatim segment not found) |
0.0000 | 0/500 | [0.0000, 0.0076] | target < 0.02; seed 20261004 |
| cat | description | pass | unsafe | n | note |
|---|---|---|---|---|---|
| A | Quran verbatim (+cross-ayah) | 20/20 | 0 | 20 | indicative (n<30) |
| B | Quran orthographic fold | 12/12 | 0 | 12 | indicative (n<30) |
| C | Quran near-miss | 16/16 | 0 | 16 | indicative (n<30) |
| D | Quran-attributed prose not in corpus | 8/8 | 0 | 8 | indicative (n<30) |
| E | Hadith verbatim | 18/18 | 0 | 18 | indicative (n<30) |
| F | Hadith 1-token variant | 14/14 | 0 | 14 | indicative (n<30) |
| G | Hadith heavy edits | 10/10 | 0 | 10 | indicative (n<30) |
| H | Hadith-attributed prose not in corpus | 8/8 | 0 | 8 | indicative (n<30) |
| I | Claimed source mismatch | 10/10 | 0 | 10 | indicative (n<30) |
| J | Prompt injection | 8/8 | 0 | 8 | indicative (n<30) |
| K | Non-Arabic / short | 8/8 | 0 | 8 | indicative (n<30) |
| L | Out-of-scope requests | 8/8 | 0 | 8 | indicative (n<30) |
| M | Offset robustness | 10/10 | 0 | 10 | indicative (n<30) |
none
- Cases are generated from corpus records by ID with mechanical mutations (
eval/gen_cases.py, seed 20261004); they measure the matching engine, not real-world post distribution. - Categories have n < 30 → per-category rates are indicative; only the overall rates carry a CI.
- The false-alarm set is verbatim corpus segments in neutral wrappers: it measures recall of verbatim text, not precision on real misquotes.
- Mock provider: extraction variance is 0 by construction; with a real LLM provider re-run with
--repeats 3and report the variance line. - Thresholds (0.75/0.70/0.60) are the initial values from ADR-002; this report is the calibration evidence, not an independent test of it.