EvalBench Documentation

Reference documentation for EvalBench. Start with the main README for installation and a first run.

Configuration

Every evaluation is driven by a run config that points at a dataset, a database or agent, a model, and a set of scorers.

Doc	Contents
Run config	The top-level YAML for an evaluation run — dataset, generation, setup/teardown, scorers, reporting.
NL2SQL dataset format	Evaluation items for single-turn SQL evaluations: prompts, golden SQL, eval queries.
Agentic dataset format	Scenario evalsets for multi-turn agent evaluations: prompts, conversation plans, expected trajectories.
Database config	Connection details per database and the list of supported dialects.
Model config	Model selection and generation settings.

Doc	Contents
Scorers	Full catalog of every scorer — SQL, agentic, skills, transform tooling, and custom Python scorers — with config options.
Judge tools	Giving LLM-judged scorers function-calling access instead of single-shot prompting.
Summarizer	How run summaries are aggregated and the rationale behind the formulas.

Doc	Contents
Agentic evaluations	Execution model, sandboxing, and tool paradigms. Start here.
Gemini CLI	Setup and configuration for evaluating Gemini CLI.
Claude Code	Setup and configuration for evaluating Claude Code.
Codex CLI	Setup and configuration for evaluating Codex CLI.
Antigravity CLI	Setup and configuration for evaluating the Antigravity (agy) CLI.
Data agent spec	ADKDataAgent support — multi-turn database agents with clarification turns.

Runnable notebooks in docs/examples/:

Doc	Contents
Contributing	How to submit patches and contributions.
Code of conduct	Community guidelines.
Dependency graph	External dependency graph, dependency groups by purpose, and supply-chain risk surfacing.