Skip to content

docs: lead ReflexBench v1 with raw engine lane #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
0xnavarro merged 2 commits into from
Sep 24, 2026
Merged

docs: lead ReflexBench v1 with raw engine lane #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5c9b2c7
docs: lead ReflexBench v1 with raw engine lane
0xnavarro Sep 24, 2026
c4f6f3e
docs: preserve locked supplemental Core claims
0xnavarro Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,29 @@ This repository contains **ReflexBench v1.0.0**, the first stable public benchma

> ReflexBench does not publish a single "best model" score. Semantic quality, calibration, language consistency, option-order robustness, cardinality, failures, latency boundaries and harness/policy value are reported separately.

## v1 headline results
## v1 headline: same-corpus raw engine quality

The canonical launch leaderboard uses the frozen **111-case public-hard cohort** and reports raw semantic accuracy on identical rows. It does not blend latency, calibration or Reflex policy effects into the headline.

| Engine | Raw semantic accuracy |
|---|---:|
| TypeSafe Jev | **73.0%** |
| upstream Reflex / Qwen3.5-2B | 41.4% |
| frozen Qwen3.5-0.8B readout control | 39.6% |
| jeff / GLiFormer ~400M | 37.8% |
| openJev Verdict 1.4 / 151M | 36.9% |
| Laya base / 421M | 35.1% |
| Kev-0.8B | 32.4% |

These are same-corpus Brida measurement receipts, not a universal model ranking. Hosted and local latency are different deployment boundaries and are reported separately in [RESULTS.md](RESULTS.md).

### Supplemental Reflex Core evidence

ReflexBench also retains policy/harness ablations as a **separate research lane**, not part of the raw-engine leaderboard:

- **Same-response Reflex Core proof:** TypeSafe Jev raw semantic accuracy **96.4% (106/110) -> 100.0% (110/110)** after deterministic policy, with 4 rescues / 0 harms and **0 extra model calls**.
- **Independent blind workflow replication:** **76.7% raw -> 82.0% Core** on 150/150 completed cases; paired delta **+5.3 pp**, bootstrap 95% **+1.3..+10.0 pp**.
- The first blind gate is retained as a formal **FAIL** because one provider 503 left it at 149/150. Failures are evidence, not rows to delete.
- Engine comparisons include the same public hard cohort, multilingual Choice lanes, calibration metrics and explicit capability/compatibility boundaries. See [RESULTS.md](RESULTS.md).
- Gate v1 remains a formal **FAIL** at 149/150 after one provider 503. Failures stay in the evidence ledger.

## Why this exists

Expand Down
22 changes: 11 additions & 11 deletions RESULTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,7 @@

All numbers below are frozen Brida measurement receipts unless explicitly marked as external context. Hosted and local latency are different deployment boundaries and are **not** normalized into a speed ranking.

## Reflex Core v1: same-response value

| Evidence | Raw engine | Reflex Core | Delta | Rescues / harms | Completion |
|---|---:|---:|---:|---:|---:|
| Public product fixtures (Jev) | 96.4% | **100.0%** | +3.6 pp | 4 / 0 | 110/110 |
| Blind workflow Gate v1 (Jev) | 45.0% | **64.4%** | +19.5 pp | 29 / 0 | 149/150 — **FAIL** |
| Blind workflow Gate v2 (Jev) | 76.7% | **82.0%** | +5.3 pp | 10 / 2 | 150/150 — **PASS** |

Gate v1 remains failed because one provider 503 prevented 100% completion. Gate v2 was independently frozen before inference, used a different case pack, completed 150/150 and used zero transport retries. Gate v2 proves positive net same-response harness value; it does **not** prove automatic-action safety or global policy optimality.

## Same-corpus public hard cohort (111 cases)
## Primary raw-engine lane: same-corpus public hard cohort (111 cases)

| Engine | Accuracy | ECE ↓ | Score MAE ↓ | p50 | Boundary |
|---|---:|---:|---:|---:|---|
Expand All @@ -26,6 +16,16 @@ Gate v1 remains failed because one provider 503 prevented 100% completion. Gate

The 111 public-hard rows are now development evidence for Brida and may not later be presented as an untouched Brida model gate.

## Supplemental research: Reflex Core v1 same-response value

| Evidence | Raw engine | Reflex Core | Delta | Rescues / harms | Completion |
|---|---:|---:|---:|---:|---:|
| Public product fixtures (Jev) | 96.4% | **100.0%** | +3.6 pp | 4 / 0 | 110/110 |
| Blind workflow Gate v1 (Jev) | 45.0% | **64.4%** | +19.5 pp | 29 / 0 | 149/150 — **FAIL** |
| Blind workflow Gate v2 (Jev) | 76.7% | **82.0%** | +5.3 pp | 10 / 2 | 150/150 — **PASS** |

Gate v1 remains failed because one provider 503 prevented 100% completion. Gate v2 was independently frozen before inference, used a different case pack, completed 150/150 and used zero transport retries. Gate v2 proves positive net same-response harness value; it does **not** prove automatic-action safety or global policy optimality.

## Multilingual Choice

### Common 20-way lane (200 EN/ES decisions)
Expand Down
Loading