From 5c9b2c7a319c62055463b75b67b52fcb4045d3fa Mon Sep 17 00:00:00 2001 From: Carlos Navarro <68403497+0xnavarro@users.noreply.github.com> Date: Wed, 23 Sep 2026 19:49:02 -0600 Subject: [PATCH 1/2] docs: lead ReflexBench v1 with raw engine lane --- README.md | 27 ++++++++++++++++++++++----- RESULTS.md | 22 +++++++++++----------- 2 files changed, 33 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index bb1c536..241ad4b 100644 --- a/README.md +++ b/README.md @@ -8,12 +8,29 @@ This repository contains **ReflexBench v1.0.0**, the first stable public benchma > ReflexBench does not publish a single "best model" score. Semantic quality, calibration, language consistency, option-order robustness, cardinality, failures, latency boundaries and harness/policy value are reported separately. -## v1 headline results +## v1 headline: same-corpus raw engine quality -- **Same-response Reflex Core proof:** TypeSafe Jev raw semantic accuracy **96.4% (106/110) -> 100.0% (110/110)** after deterministic policy, with 4 rescues / 0 harms and **0 extra model calls**. -- **Independent blind workflow replication:** **76.7% raw -> 82.0% Core** on 150/150 completed cases; paired delta **+5.3 pp**, bootstrap 95% **+1.3..+10.0 pp**. -- The first blind gate is retained as a formal **FAIL** because one provider 503 left it at 149/150. Failures are evidence, not rows to delete. -- Engine comparisons include the same public hard cohort, multilingual Choice lanes, calibration metrics and explicit capability/compatibility boundaries. See [RESULTS.md](RESULTS.md). +The canonical launch leaderboard uses the frozen **111-case public-hard cohort** and reports raw semantic accuracy on identical rows. It does not blend latency, calibration or Reflex policy effects into the headline. + +| Engine | Raw semantic accuracy | +|---|---:| +| TypeSafe Jev | **73.0%** | +| upstream Reflex / Qwen3.5-2B | 41.4% | +| frozen Qwen3.5-0.8B readout control | 39.6% | +| jeff / GLiFormer ~400M | 37.8% | +| openJev Verdict 1.4 / 151M | 36.9% | +| Laya base / 421M | 35.1% | +| Kev-0.8B | 32.4% | + +These are same-corpus Brida measurement receipts, not a universal model ranking. Hosted and local latency are different deployment boundaries and are reported separately in [RESULTS.md](RESULTS.md). + +### Supplemental Reflex Core evidence + +ReflexBench also retains policy/harness ablations as a **separate research lane**, not part of the raw-engine leaderboard: + +- TypeSafe Jev same-response public fixtures: **96.4% raw -> 100.0% Core**, 4 rescues / 0 harms, 0 extra model calls. +- Independent blind workflow Gate v2: **76.7% raw -> 82.0% Core** on 150/150 completed cases; +5.3 pp, bootstrap 95% +1.3..+10.0 pp. +- Gate v1 remains a formal **FAIL** at 149/150 after one provider 503. Failures stay in the evidence ledger. ## Why this exists diff --git a/RESULTS.md b/RESULTS.md index 8a466d8..ffc47d6 100644 --- a/RESULTS.md +++ b/RESULTS.md @@ -2,17 +2,7 @@ All numbers below are frozen Brida measurement receipts unless explicitly marked as external context. Hosted and local latency are different deployment boundaries and are **not** normalized into a speed ranking. -## Reflex Core v1: same-response value - -| Evidence | Raw engine | Reflex Core | Delta | Rescues / harms | Completion | -|---|---:|---:|---:|---:|---:| -| Public product fixtures (Jev) | 96.4% | **100.0%** | +3.6 pp | 4 / 0 | 110/110 | -| Blind workflow Gate v1 (Jev) | 45.0% | **64.4%** | +19.5 pp | 29 / 0 | 149/150 — **FAIL** | -| Blind workflow Gate v2 (Jev) | 76.7% | **82.0%** | +5.3 pp | 10 / 2 | 150/150 — **PASS** | - -Gate v1 remains failed because one provider 503 prevented 100% completion. Gate v2 was independently frozen before inference, used a different case pack, completed 150/150 and used zero transport retries. Gate v2 proves positive net same-response harness value; it does **not** prove automatic-action safety or global policy optimality. - -## Same-corpus public hard cohort (111 cases) +## Primary raw-engine lane: same-corpus public hard cohort (111 cases) | Engine | Accuracy | ECE ↓ | Score MAE ↓ | p50 | Boundary | |---|---:|---:|---:|---:|---| @@ -26,6 +16,16 @@ Gate v1 remains failed because one provider 503 prevented 100% completion. Gate The 111 public-hard rows are now development evidence for Brida and may not later be presented as an untouched Brida model gate. +## Supplemental research: Reflex Core v1 same-response value + +| Evidence | Raw engine | Reflex Core | Delta | Rescues / harms | Completion | +|---|---:|---:|---:|---:|---:| +| Public product fixtures (Jev) | 96.4% | **100.0%** | +3.6 pp | 4 / 0 | 110/110 | +| Blind workflow Gate v1 (Jev) | 45.0% | **64.4%** | +19.5 pp | 29 / 0 | 149/150 — **FAIL** | +| Blind workflow Gate v2 (Jev) | 76.7% | **82.0%** | +5.3 pp | 10 / 2 | 150/150 — **PASS** | + +Gate v1 remains failed because one provider 503 prevented 100% completion. Gate v2 was independently frozen before inference, used a different case pack, completed 150/150 and used zero transport retries. Gate v2 proves positive net same-response harness value; it does **not** prove automatic-action safety or global policy optimality. + ## Multilingual Choice ### Common 20-way lane (200 EN/ES decisions) From c4f6f3e54357e80762f3e83e65c2bed65d692eb0 Mon Sep 17 00:00:00 2001 From: Carlos Navarro <68403497+0xnavarro@users.noreply.github.com> Date: Wed, 23 Sep 2026 20:02:46 -0600 Subject: [PATCH 2/2] docs: preserve locked supplemental Core claims --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 241ad4b..8f7ba8f 100644 --- a/README.md +++ b/README.md @@ -28,8 +28,8 @@ These are same-corpus Brida measurement receipts, not a universal model ranking. ReflexBench also retains policy/harness ablations as a **separate research lane**, not part of the raw-engine leaderboard: -- TypeSafe Jev same-response public fixtures: **96.4% raw -> 100.0% Core**, 4 rescues / 0 harms, 0 extra model calls. -- Independent blind workflow Gate v2: **76.7% raw -> 82.0% Core** on 150/150 completed cases; +5.3 pp, bootstrap 95% +1.3..+10.0 pp. +- **Same-response Reflex Core proof:** TypeSafe Jev raw semantic accuracy **96.4% (106/110) -> 100.0% (110/110)** after deterministic policy, with 4 rescues / 0 harms and **0 extra model calls**. +- **Independent blind workflow replication:** **76.7% raw -> 82.0% Core** on 150/150 completed cases; paired delta **+5.3 pp**, bootstrap 95% **+1.3..+10.0 pp**. - Gate v1 remains a formal **FAIL** at 149/150 after one provider 503. Failures stay in the evidence ledger. ## Why this exists