chore(core): remove legacy benchmark judge

[phernandez]

Why

The benchmark package had two overlapping LLM evaluation paths. The legacy
run judge path depended on Pydantic Evals internals and silently converted an
upstream API change into bogus all-zero results. The newer end-to-end QA path
already covers answer generation, judging, re-judging, diagnosis, and manual
review, so retaining the legacy path made benchmark output less trustworthy.

While validating the replacement path live, the documented
OPENAI_API_KEY configuration was also found to be disconnected from the
OpenAI-compatible runner. That made every CLI-driven OpenAI QA case fail with
401 responses even though the key itself was valid.

What Changed

• Removed the legacy Pydantic Evals judge implementation, CLI command, model
  fields, artifact writers, optional dependency, lockfile packages, and just
  recipes.
• Made run qa the documented benchmark evaluation workflow and added a
  bench-qa convenience recipe.
• Made OpenAI-compatible runners use OPENAI_API_KEY when no explicit key is
  supplied.
• Added CLI-surface, manifest-schema, and API-key regression coverage.

Implementation Details

The modern QA path remains separate from retrieval: run full produces the
deterministic retrieval artifacts, then run qa generates and judges answers
from those stored results. This preserves the ability to inspect or re-score a
retrieval run without repeating provider ingestion and search.

The OpenAI-compatible runner only consults OPENAI_API_KEY when its caller did
not supply an API key explicitly, preserving explicit configuration precedence.

Testing

Automated

• uv sync --group dev: passed; removed the legacy Pydantic Evals dependency tree.
• uv run pytest -q: 171 passed.
• uv run ruff check .: passed.
• uv run ruff format --check src/basic_memory_benchmarks tests/llm/test_runners.py tests/test_cli_surface.py tests/test_manifest_schema.py: passed (42 files checked).
• uv run pyright src/basic_memory_benchmarks/llm/runners.py src/basic_memory_benchmarks/cli.py src/basic_memory_benchmarks/runner.py src/basic_memory_benchmarks/models.py src/basic_memory_benchmarks/reporting/artifacts.py tests/llm/test_runners.py tests/test_cli_surface.py tests/test_manifest_schema.py: passed with 0 errors.
• uv lock --check: passed.

Manual / Live Benchmark

• Strict 25-query BM-vs-Mem0 retrieval smoke: 25 rows/provider, 0 errors, 0 fairness warnings.
• Modern QA smoke with gpt-4o-mini: 50/50 answers generated and judged, 0 errors.
• Complete strict LoCoMo retrieval at 91b2299d96a3c97809735c7f620c12d79098d196: 1,986 rows/provider, 0 errors, 0 fairness warnings.
• Complete modern QA: 3,972/3,972 answers generated and judged, 0 errors; BM 52.77% accuracy, Mem0 45.87%.
• validate-artifacts, run diagnose, and run review: passed on the complete run.

Risks / Follow-ups

• Full-package uv run pyright still reports two pre-existing errors in untouched provider tests (test_baseline_providers.py and test_mem0_normalization.py); all changed Python surfaces type-check cleanly.
• The complete benchmark artifacts are local/ignored run output and are not part of this source PR.
• QA results are model-dependent; this run intentionally used the same fixed gpt-4o-mini model as answerer and judge for both providers.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91b2299d96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restrict the implicit API key to OpenAI endpoints

When OPENAI_API_KEY is set—for example, to run the mem0-local provider—and an answerer or judge uses Ollama, LM Studio, vLLM, or another custom openai-compat URL, this fallback sends the reusable OpenAI credential to that arbitrary server as a Bearer token, potentially even over plain HTTP. Only infer this key for a verified OpenAI API URL, or require an endpoint-specific explicit credential for other hosts.

AGENTS.md reference: benchmarks/AGENTS.md:L102-L106

Useful? React with 👍 / 👎.