🔧(evals) add run_eval management command

[maxenceh]

Purpose

This PR introduces a behavioral eval framework for ConversationAgent. Unlike unit tests, these evals exercise LLM behaviour end-to-end:

• does the model call the right tool (or none)?
• does it respect system instructions and avoid known bad patterns?
• are answers grounded in retrieved context?

A failing eval means a documented behaviour has regressed. The system is designed to grow: adding a new dataset requires one YAML file, one config file, and a registry entry.

Proposal

Running evals

• make eval / make eval-debug — runs evals inside Docker; debug mode exposes debugpy on port 5678 for VS Code remote attach
• Git metadata (commit, branch, dirty) is injected from the host via EVAL_GIT_* env vars (Docker has no .git mount)
• run_evals management command: --dataset, --case, --runs, --verbose, --no-llm-judge, --save, --comment, --include-outputs
• Tested model = LLM_DEFAULT_MODEL_HRID; judge = LLM_EVAL_JUDGE_MODEL_HRID (falls back to default; warns when judge == tested model)
• --save is rejected when combined with --case (partial runs would register coverage gaps vs baseline)

Datasets (4)

Dataset	What it tests
url_hallucination	Never invents http(s):// URLs; only cites URLs from tool output or user message
faithfulness_rag	Answers grounded in RAG chunks; must call document_search_rag, must not call web_search
incertitude	On high-stakes French service-public questions, asks for missing personal context instead of guessing
tool_selection	Calls the right tool (web_search, self_documentation, document_search_rag, summarize) or none — includes adversarial French phrasing

self_documentation is covered as cases within tool_selection, not as a standalone dataset.

RAG/summarize cases use inputs.requires_documents: true + a fake document listing. Per-case stub payloads are staged via tool_stub_responses.py (contextvar) so the model must actually call tools.

Evaluators & agent wiring

• EvalConfig registry — each dataset declares rubric, evaluators, and optional make_task_fn / agent_class
• UrlRegexEvaluator — deterministic URL check; strips trailing punctuation; reads runtime tool returns via set_eval_attribute
• HasNoMatchingSpan — correct "tool was NOT called" check (avoids not_(HasMatchingSpan) false positives)
• production_agent.py — builds agents with production tool wiring; stubs external calls (no DB/API)
• Per-case YAML evaluators are merged with config evaluators (not replaced)

Persistence, baselines & comparison

• --save writes JSON runs under chat/evals/runs/ (gitignored) with git metadata, dataset hashes, per-case avg scores, repeat pass rates
• overall_pass_rate weighted by case count (not per-dataset average)
• make eval-baseline — promotes a run to committed baseline (baselines/main.json + main_run.json)
• make eval-compare — CLI diff with regression detection and coverage-gap handling (--fail-on-regression)
• make eval-reset — wipe local runs/baselines/dashboard (--keep-baselines, --keep-dashboard, --dry-run)

HTML dashboard

• make eval-dashboard — self-contained HTML from template.html + embedded run data
• Compare runs A/B with per-dataset pass-rate summary (including total), filters (dataset / case / changes only), per-case deltas with failure reasons
• Coverage gaps aligned with compare.py semantics
• Embeds latest 20 runs + baseline runs; dataset/case descriptions from YAML

Albert API compatibility

• AlbertOpenAIChatModel._validate_completion — normalizes Albert non-stream responses for pydantic-ai / pydantic_evals:
  • tool_calls[].type → function
  • non-standard object / non-list choices on multi-turn tool-call conversations
• Required for evals and LLM judge to work with Albert's OpenAI-compatible API

Docs & tests

• chat/evals/README.md — structure, datasets, commands, how to add a dataset
• Unit tests for storage, compare, report aggregation, dashboard generation, production agent wiring, tool stubs, reset

Test plan

• make eval EVAL_ARGS="--dataset url_hallucination" passes on current default model
• make eval EVAL_ARGS="--dataset tool_selection --runs 3" — repeat aggregation looks sane
• make eval EVAL_ARGS='--save --comment "test"' then make eval-compare — no unexpected regressions vs baseline
• make eval-dashboard — open dashboard.html, verify A/B summary and case drill-down
• make eval EVAL_ARGS="--no-llm-judge" works when Albert structured output is unavailable
• bin/pytest chat/tests/evals/ green

Summary by CodeRabbit

• New Features
  • Added behavioral evaluation tooling with configurable datasets, repeat runs, saved runs, baselines, run-to-run comparisons, and regression detection.
  • Added a self-contained dashboard to filter and compare evaluation results.
  • Added eval support for URL hallucinations, retrieval faithfulness, uncertainty handling, and tool-selection behavior.
  • Added management commands for running, debugging, baselining, comparing, dashboard generation, and resetting.
• Bug Fixes
  • Improved normalization/validation for malformed tool-call completion payloads and evaluation tool-call extraction.
• Documentation
  • Expanded end-to-end documentation for behavioral evals and added dashboard guidance.
• Chores
  • Updated shared tool guidance/prompts for correct routing and safer behavior.