EvalTuitor is a terminal-native evaluation runner for LLM applications. It allows developers to define evaluation suites in TOML, execute them against local or remote OpenAI-compatible models, and browse and compare results inside the terminal.
- Performance: Compiled Rust binary with asynchronous parallel execution support.
- Suite Definition: Configuration-driven test suites defined using standard TOML.
- Interactive Interface: Terminal workspace to inspect test runs, failure outputs, and logs.
- Run Comparison: Side-by-side comparison interface to examine output changes between historical runs.
- Git Integration: Optional git hooks to gate commits and pushes on eval pass rate, with threshold or baseline-delta regression detection.
A Rust toolchain must be installed on your system.
Clone the repository and compile the release binary:
cargo build --releaseThe resulting binary will be generated at ./target/release/evaltuitor.
Initialize a new project context in the current directory:
./target/release/evaltuitor --initThis creates:
evaltuitor.toml(Configuration file)evals/example.toml(Sample evaluation suite)
Execute all evaluation suites in the evals/ directory and launch the interface:
./target/release/evaltuitorExecute a specific suite:
./target/release/evaltuitor evals/suite.tomlOverride the default model endpoint:
./target/release/evaltuitor --model openai/gpt-4oRun evaluations and output results to stdout without launching the interface:
./target/release/evaltuitor --no-tuiGlobal and provider configurations are set in the project root:
[defaults]
model = "ollama/llama3.1"
temperature = 0.0
max_tokens = 2048
timeout_secs = 30
parallelism = 4
[providers.ollama]
base_url = "http://localhost:11434"
[providers.openai]
api_key_env = "OPENAI_API_KEY"
[providers.vllm]
base_url = "http://localhost:8000"Suite files should be stored under the evals/ directory:
[suite]
name = "Summarization Suite"
description = "Verifies LLM summary behaviors"
[config]
model = "openai/gpt-4o"
temperature = 0.2
[[tests]]
id = "summary-length-check"
prompt = "Summarize this: {{input}}"
input = "Evaluating AI systems requires structured testing..."
assert.type = "contains-all"
assert.values = ["AI", "testing"]contains-all/contains-any/contains-none(substring checks)exact-match(string equivalence)regex(regular expression verification)llm-judge(semantic evaluation scoring using a rubric prompt)max-length/min-length(character length boundaries)json-schema(structured output JSON validation)custom(arbitrary external shell command execution)
j / k or Arrow keys: Navigate suites and test cases.Tab: Cycle focus between Suites list, Tests list, and Details pane.f: Toggle the display of failed tests only./: Filter test cases by ID or output string.C: Open the run comparison list.R: Re-execute failed test cases.?: Toggle the help overlay.q / Esc: Close overlays or exit the application.
EvalTuitor can install git hooks that run your evals before commit/push and after merge, blocking the operation on a failed gate. Hooks are marker-tagged so --uninstall-hooks only ever removes EvalTuitor's own; any pre-existing hook is backed up to <name>.pre-evaltuitor rather than overwritten. Normal eval runs perform no git operations — git is only invoked during install/uninstall/list.
# Install pre-commit, pre-push, and post-merge hooks
./target/release/evaltuitor --install-hooks
# Show install status
./target/release/evaltuitor --list-hooks
# Remove EvalTuitor-managed hooks (user hooks and backups are left intact)
./target/release/evaltuitor --uninstall-hooksA gate fails the run (non-zero exit, which blocks the git operation) when the configured condition is met. Both modes are independent and composable:
--min-pass-rate <0.0–1.0>— fail if the pass rate falls below a flat threshold.--baseline-compare— fail only if the pass rate dropped vs. the most recent prior saved run, reporting the count of newly-regressed tests. Falls back to passing when no baseline exists yet.
The installed pre-commit and pre-push hooks gate by default (--min-pass-rate 0.9); post-merge is informational and never blocks. Override the gate policy at install time, for example to use baseline regression detection:
./target/release/evaltuitor --install-hooks --baseline-compare --min-pass-rate 0.95The same flags also work as a standalone CI gate with no hooks installed:
./target/release/evaltuitor --no-tui --min-pass-rate 0.9 --baseline-compareThe generated hook scripts invoke the binary used at install time (by absolute path). Override at runtime without reinstalling via environment variables:
EVALTUITOR_BIN=/path/to/evaltuitor— use a different binary (e.g. a freshly built release).EVALTUITOR_HOOK_SUITE=path/to/evals— point the hooks at a different suite directory or file.