chore(core): remove legacy benchmark judge

benchmarks/AGENTS.md

@@ -7,7 +7,7 @@ directory for comparing Basic Memory against other memory systems.
 
 Primary goals:
 - Deterministic retrieval benchmarks
-- Optional LLM-as-a-judge benchmarks
+- End-to-end QA scoring with fixed answerer and judge models
 - Public, reproducible artifact publication (including provenance metadata)
 
 The benchmark package keeps its own `pyproject.toml` and lockfile so benchmark
@@ -16,7 +16,6 @@ dependencies do not pollute the Core product environment.
 ## Build / Test Commands
 
 - Install: `uv sync --group dev`
-- Install judge extras: `uv sync --group dev --extra judge`
 - Run tests: `uv run pytest -q`
 - Lint: `uv run ruff check .`
 - Type check: `uv run pyright`
@@ -37,8 +36,8 @@ Dataset and conversion:
 Run retrieval:
 - `uv run bm-bench run retrieval --providers bm-local,mem0-local --dataset-id locomo --dataset-path benchmarks/datasets/locomo/locomo10.json --corpus-dir benchmarks/generated/locomo/docs --queries-path benchmarks/generated/locomo/queries.json --output-root benchmarks/runs --allow-provider-skip`
 
-Run judge (optional):
-- `uv run bm-bench run judge --run-dir benchmarks/runs/<run-id>`
+Run end-to-end QA scoring:
+- `uv run bm-bench run qa --run-dir benchmarks/runs/<run-id> --answerer claude:claude-haiku-4-5 --judge claude:claude-sonnet-4-6`
 
 Validate and publish:
 - `uv run bm-bench validate-artifacts --run-dir benchmarks/runs/<run-id>`
@@ -53,15 +52,15 @@ Validate and publish:
 - `just bench-run-bm-local`
 - `just bench-run-mem0-local`
 - `just bench-run-full`
-- `just bench-judge RUN_DIR=benchmarks/runs/<run-id>`
-- `just bench-publish RUN_DIR=benchmarks/runs/<run-id>`
+- `just bench-qa benchmarks/runs/<run-id>`
+- `just bench-publish benchmarks/runs/<run-id>`
 
 ## Repository Layout
 
 - `src/basic_memory_benchmarks/cli.py` - CLI surface
 - `src/basic_memory_benchmarks/runner.py` - run orchestration
 - `src/basic_memory_benchmarks/providers/` - provider adapters (`bm-local`, `bm-cloud`, `mem0-local`, `zep-reference`)
-- `src/basic_memory_benchmarks/scoring/` - retrieval + judge scoring
+- `src/basic_memory_benchmarks/scoring/` - retrieval + end-to-end QA scoring
 - `src/basic_memory_benchmarks/reporting/` - artifact writers / comparison helpers
 - `src/basic_memory_benchmarks/converters/` - dataset conversion logic
 - `src/basic_memory_benchmarks/datasets/` - dataset fetch/load helpers

benchmarks/README.md

@@ -7,7 +7,7 @@ dependencies remain isolated from the product environment.
 ## Goals
 
 - Deterministic retrieval benchmarks (Recall@5/10, MRR, Precision@5, content-hit, latency)
-- Optional LLM-as-judge scoring (Pydantic Evals)
+- End-to-end QA scoring with fixed answerer and judge models
 - Public artifacts with provenance and reproducibility metadata
 - Clean dependency isolation from the core Basic Memory package
 
@@ -29,12 +29,6 @@ dependencies remain isolated from the product environment.
 uv sync --group dev
 ```
 
-Optional judge dependencies:
-
-```bash
-uv sync --group dev --extra judge
-```
-
 ## Quickstart
 
 ### 1) Fetch LoCoMo dataset
@@ -85,16 +79,7 @@ know") when the retrieved memories don't contain the answer, and abstention is
 graded correct only when the gold answer marks the question unanswerable
 (LoCoMo adversarial cases).
 
-### 5) Optional retrieval-context judge (legacy)
-
-Scores whether the *retrieved context* contains the expected answer, without
-answer generation:
-
-```bash
-uv run bm-bench run judge --run-dir benchmarks/runs/<run-id>
-```
-
-### 6) Publish run artifacts
+### 5) Publish run artifacts
 
 ```bash
 uv run bm-bench publish --run-dir benchmarks/runs/<run-id>
@@ -290,8 +275,6 @@ Per run (`benchmarks/runs/<run-id>/`):
 - `retrieval-summary.json`
 - `per-query-qa.jsonl` (optional)
 - `qa-summary.json` (optional)
-- `per-query-judge.jsonl` (optional)
-- `judge-summary.json` (optional)
 - `summary.md`
 
 ## Just commands
@@ -303,8 +286,8 @@ just bench-convert-locomo
 just bench-run-bm-local
 just bench-run-mem0-local
 just bench-run-full
-just bench-judge
-just bench-publish RUN_DIR=benchmarks/runs/<run-id>
+just bench-qa benchmarks/runs/<run-id>
+just bench-publish benchmarks/runs/<run-id>
 ```
 
 ## Notes on dataset publication

benchmarks/docs/benchmarks.md

@@ -12,9 +12,9 @@ It covers:
 
 | Area | Status |
 | --- | --- |
-| Single run execution (`run retrieval`, `run full`, `run judge`) | Implemented |
+| Single run execution (`run retrieval`, `run full`, `run qa`) | Implemented |
 | Concurrent write convergence (`run concurrent-write`) | Implemented |
-| `just` one-command pipelines (`bench-full`, `bench-full-judge`) | Implemented |
+| `just` retrieval and QA workflows (`bench-full`, `bench-qa`) | Implemented |
 | Artifact generation and publish/compare commands | Implemented |
 | Manual BM revision comparison via worktrees + `--bm-local-path` | Implemented workflow, manual orchestration |
 | `bm-bench run revision-matrix` | Planned, not implemented yet |
@@ -58,12 +58,6 @@ cd /path/to/basic-memory/benchmarks
 just sync
 ```
 
-If you plan to run judge metrics:
-
-```bash
-just sync-judge
-```
-
 ### Dataset assumptions
 
 LoCoMo source and converted outputs are created by:
@@ -78,15 +72,14 @@ just bench-prepare-long
 ### `just` commands (current)
 
 - `bench-full`
-- `bench-full-judge`
+- `bench-qa`
 - `bench-concurrent-write-smoke`
 - `bench-concurrent-write-load`
 - `bench-prepare-short`
 - `bench-prepare-long`
 - `bench-run-short`
 - `bench-run-long`
 - `bench-run-full`
-- `bench-judge`
 - `bench-validate`
 - `bench-publish`
 - `bench-compare`
@@ -101,7 +94,9 @@ Top-level commands:
 - `run retrieval`
 - `run concurrent-write`
 - `run full`
-- `run judge`
+- `run qa`
+- `run rejudge`
+- `run review`
 - `compare`
 - `validate-artifacts`
 - `publish`
@@ -120,17 +115,15 @@ This runs:
 2. `just bench-prepare-long`
 3. `just bench-run-full`
 
-### One-command full retrieval + judge
+### End-to-end QA scoring
 
 ```bash
 cd /path/to/basic-memory/benchmarks
-just bench-full-judge
+just bench-qa benchmarks/runs/<run_id>
 ```
 
-This runs:
-1. `just sync-judge`
-2. `just bench-prepare-long`
-3. `just bench-run-full-judge`
+This generates answers from each provider's retrieved context, applies the
+same judge to every provider, and writes QA artifacts into the retrieval run.
 
 ### Short vs long workflows
 
@@ -196,10 +189,15 @@ Required files:
 - `retrieval-summary.json`
 - `summary.md`
 
-Optional judge files:
+Optional QA files:
 
-- `per-query-judge.jsonl`
-- `judge-summary.json`
+- `per-query-qa.jsonl`
+- `qa-summary.json`
+- `per-query-qa-rejudge.jsonl`
+- `qa-rejudge-summary.json`
+- `qa-rejudge-flips.json`
+- `review.html`
+- `qa-diagnosis.json`
 
 ### Key provenance fields
 
@@ -483,7 +481,7 @@ Dry-run checks:
 just --dry-run bench-run-short
 just --dry-run bench-run-long
 just --dry-run bench-full
-just --dry-run bench-full-judge
+just --dry-run bench-qa benchmarks/runs/<run_id>
 ```
 
 Artifact field checks:

benchmarks/justfile

@@ -18,9 +18,6 @@ longmemeval_dev_output_dir := "benchmarks/generated/longmemeval-s-dev"
 sync:
     uv sync --group dev
 
-sync-judge:
-    uv sync --group dev --extra judge
-
 test:
     uv run pytest -q
 
@@ -110,13 +107,6 @@ bench-full:
     just bench-prepare-long
     just bench-run-full
 
-# Full retrieval + judge pipeline:
-# 1) sync deps (+judge extras), 2) fetch+convert long dataset, 3) run full with judge
-bench-full-judge model="gpt-4o-mini":
-    just sync-judge
-    just bench-prepare-long
-    just bench-run-full-judge model="{{model}}"
-
 # --- Benchmark execution ---
 
 bench-smoke:
@@ -198,18 +188,6 @@ bench-run-full:
       {{bm_local_path_flag}} \
       --allow-provider-skip
 
-bench-run-full-judge model="gpt-4o-mini":
-    uv run bm-bench run full \
-      --dataset-id locomo \
-      --dataset-path {{locomo_dataset_path}} \
-      --corpus-dir benchmarks/generated/locomo/docs \
-      --queries-path benchmarks/generated/locomo/queries.json \
-      --providers bm-local,mem0-local \
-      {{bm_local_path_flag}} \
-      --allow-provider-skip \
-      --judge \
-      --judge-model "{{model}}"
-
 # --- Concurrency benchmark (basic-memory#1248) ---
 
 # Small-scale smoke: 4 writers x 25 notes; strict so divergence fails the command
@@ -244,8 +222,11 @@ bench-latest-run:
     set -euo pipefail
     ls -1dt benchmarks/runs/* | head -n 1
 
-bench-judge run_dir model="gpt-4o-mini":
-    uv run bm-bench run judge --run-dir "{{run_dir}}" --model "{{model}}"
+bench-qa run_dir answerer="claude:claude-haiku-4-5" judge="claude:claude-sonnet-4-6":
+    uv run bm-bench run qa \
+      --run-dir "{{run_dir}}" \
+      --answerer "{{answerer}}" \
+      --judge "{{judge}}"
 
 bench-validate run_dir:
     uv run bm-bench validate-artifacts --run-dir "{{run_dir}}"

benchmarks/pyproject.toml

@@ -18,11 +18,6 @@ dependencies = [
     "typer>=0.16.1",
 ]
 
-[project.optional-dependencies]
-judge = [
-    "pydantic-evals>=0.4.0",
-]
-
 [project.scripts]
 bm-bench = "basic_memory_benchmarks.cli:main"
 basic-memory-benchmarks = "basic_memory_benchmarks.cli:main"

benchmarks/src/basic_memory_benchmarks/cli.py

@@ -30,7 +30,6 @@
 )
 from basic_memory_benchmarks.runner import (
     run_diagnose_stage,
-    run_judge,
     run_qa_stage,
     run_rejudge_stage,
     run_review_stage,
@@ -430,15 +429,6 @@ def run_rejudge_command(
     console.print(f"Flips: [cyan]{out / 'qa-rejudge-flips.json'}[/cyan]")
 
 
-@run_app.command("judge")
-def run_judge_command(
-    run_dir: Path = typer.Option(..., "--run-dir"),
-    model: str = typer.Option("gpt-4o-mini", "--model"),
-) -> None:
-    out = run_judge(run_dir=run_dir, model=model)
-    console.print(f"Judge run complete: [green]{out}[/green]")
-
-
 @run_app.command("full")
 def run_full_command(
     providers: str = typer.Option("bm-local,mem0-local", "--providers"),
@@ -456,8 +446,6 @@ def run_full_command(
     bm_source: str = typer.Option("github:basicmachines-co/basic-memory@main", "--bm-source"),
     bm_local_path: str | None = typer.Option(None, "--bm-local-path"),
     allow_provider_skip: bool = typer.Option(True, "--allow-provider-skip/--strict-providers"),
-    judge: bool = typer.Option(False, "--judge"),
-    judge_model: str = typer.Option("gpt-4o-mini", "--judge-model"),
 ) -> None:
     run_retrieval_command(
         providers=providers,
@@ -473,18 +461,6 @@ def run_full_command(
         allow_provider_skip=allow_provider_skip,
     )
 
-    if judge:
-        resolved_run_id = run_id
-        if resolved_run_id is None:
-            # run_retrieval_command generated uuid when run_id is None. infer by latest dir.
-            run_dirs = sorted(Path(output_root).glob("*"), key=lambda path: path.stat().st_mtime)
-            if not run_dirs:
-                raise RuntimeError("Unable to locate run directory for judge step")
-            run_dir = run_dirs[-1]
-        else:
-            run_dir = Path(output_root) / resolved_run_id
-        run_judge_command(run_dir=run_dir, model=judge_model)
-
 
 @app.command("compare")
 def compare_runs(

benchmarks/src/basic_memory_benchmarks/llm/runners.py

@@ -16,6 +16,7 @@
 from __future__ import annotations
 
 import json
+import os
 import subprocess
 import time
 from abc import ABC, abstractmethod
@@ -187,7 +188,8 @@ def create_runner(spec: str, *, api_key: str | None = None) -> LLMRunner:
             raise ValueError(
                 f"openai-compat spec must be 'openai-compat:<model>@<base_url>', got: {spec}"
             )
-        return OpenAICompatRunner(model=model, base_url=base_url, api_key=api_key)
+        resolved_api_key = api_key if api_key is not None else os.getenv("OPENAI_API_KEY")
+        return OpenAICompatRunner(model=model, base_url=base_url, api_key=resolved_api_key)
     raise ValueError(
         f"Unknown runner spec '{spec}'. Expected 'claude:<model>' or "
         f"'openai-compat:<model>@<base_url>'."

benchmarks/src/basic_memory_benchmarks/models.py

@@ -79,26 +79,6 @@ class RetrievalSummary(BaseModel):
     adversarial_breakout: RetrievalMetrics
 
 
-class JudgeCaseResult(BaseModel):
-    provider: str
-    query_id: str
-    category: str
-    passed: bool
-    score: float
-    reason: str
-    evaluator: str
-
-
-class JudgeSummary(BaseModel):
-    provider: str
-    evaluator: str
-    model: str
-    total_cases: int
-    pass_count: int
-    accuracy: float
-    skipped_reason: str | None = None
-
-
 class QACategoryMetrics(BaseModel):
     total: int = 0
     correct: int = 0
@@ -213,8 +193,6 @@ class RunConfig(BaseModel):
     top_k: int = 10
     bm_source: str = "github:basicmachines-co/basic-memory@main"
     bm_local_path: str | None = None
-    judge_enabled: bool = False
-    judge_model: str = "gpt-4o-mini"
     allow_provider_skip: bool = True
 
 
@@ -237,8 +215,6 @@ class RunArtifacts(BaseModel):
     provider_status: list[ProviderStatus]
     retrieval_summaries: list[RetrievalSummary]
     retrieval_rows: list[PerQueryRetrievalResult]
-    judge_summaries: list[JudgeSummary] = Field(default_factory=list)
-    judge_rows: list[JudgeCaseResult] = Field(default_factory=list)
     fairness_warnings: list[str] = Field(default_factory=list)