Local, terminal-first observability and evaluation for Python agents.
Evalon records agent runs as traces in local SQLite, turns reviewed traces into versioned dataset cases, and runs deterministic, Python, and rubric-based evaluations. The terminal UI covers both trace inspection and evaluation history. There is no hosted service, cloud account, or web dashboard.
- Root traces and nested spans
- Function, tool, and provider inputs and outputs
- Events, metrics, token usage, latency, errors, and optional cost estimates
- Sessions that group related traces
Evalon's trace, dataset, evaluator, and run data stays on the machine that
produced it. Everything shares one system database at
~/.evalon/evalon-runs.sqlite. Scope by project name (suites, traces, TUI
--project) — do not create per-repo runs.sqlite files.
Evalon supports Python 3.11+ and uses UV.
uv add evalonFor a local checkout:
git clone https://github.com/sidmanale643/evalon.git
cd evalon
uv sync --all-extras --devFrom a local checkout:
uv sync --all-extras --dev
cd examples/support-agent
export PYTHONPATH="$PWD"
uv run evalon run suite.yaml --concurrency 2
uv run evalon run suite.yaml --static-only --min-pass-rate 1.0That flow:
- Imports
cases.yamlas a versioned dataset - Creates suite
support-agent/support regression - Calls
agent:run_casefor each case - Scores with static evaluators (exact match, JSON path, tool limits, errors)
See examples/support-agent/README.md for
export, background workers, and CI threshold checks:
uv run evalon run suite.yaml --background --min-pass-rate 1.0
# then, with the printed run id:
uv run evalon eval show RUN_ID --wait --min-pass-rate 1.0If you use Codex, Claude Code, Cursor, or Grok Build, point the agent at the
Evalon skills under skills/:
| Skill | Purpose |
|---|---|
evalon-audit |
Profile the agent and model failure risks (read-only) |
evalon-wire |
Scaffold evalon-evals/<slug>/ adapter + static smoke suite and run it |
Typical prompt:
Use the evalon-wire skill to wire Evalon into this repo for <agent>.
Create a static-only smoke harness under evalon-evals/<slug>/ and run it.
Copy or symlink the skill into your agent’s skills directory if it does not
read this repository’s skills/ tree by default.
import evalon
evalon.init("support-agent")
@evalon.observe
def answer(question: str) -> str:
return f"Answering: {question}"
print(answer("Where is my order?"))uv run python main.py
uv run evalon # opens the terminal viewer on the same SQLite databaseAll projects write to the same SQLite file:
~/.evalon/evalon-runs.sqlite
evalon.init("support-agent") # always the system DB unless you overrideuv run evalon # open the TUI on the system DB
uv run evalon --project support-agent # filter one projectOverride only for tests or deliberate isolation:
| Mechanism | Use |
|---|---|
| (default) | ~/.evalon/evalon-runs.sqlite |
EVALON_HOME |
Change the data directory (default ~/.evalon) |
EVALON_DB |
Point at a full SQLite path |
--database PATH / output= |
Same override for CLI / SDK |
Prefer filtering with suite project / --project over separate database files.
Evalon wraps existing OpenAI-compatible and Anthropic clients while preserving their normal calling interface. Wrapped calls become spans and capture request details, responses, token usage, latency, and errors.
Install provider SDKs when Evalon should create clients for you:
uv add "evalon[providers]"import evalon
evalon.init("support-agent")
client = evalon.openai()
@evalon.observe
def answer(question: str) -> str:
response = client.responses.create(
model="gpt-4.1-mini",
input=question,
)
return response.output_textYou can also wrap an already configured client:
from openai import OpenAI
import evalon
evalon.init("support-agent")
client = evalon.openai(OpenAI(timeout=30))Available wrappers are evalon.openai(), evalon.openrouter(), and
evalon.anthropic(). A provider call outside an active trace creates its own
root trace, so it remains visible.
For optional model cost estimates, install the cost extra:
uv add "evalon[cost]"Use trace when you want to control a boundary, attach metadata, or record
an output yourself. Use tool to create a child span for a function.
import evalon
evalon.init("support-agent", environment="development")
@evalon.tool
def lookup_order(order_id: str) -> dict[str, str]:
return {"order_id": order_id, "status": "delivered"}
def handle(order_id: str) -> dict[str, str]:
with evalon.trace("handle-order", input={"order_id": order_id}) as run:
result = lookup_order(order_id)
run.record_output(result)
run.record_metric("orders_looked_up", 1)
return resultEvalon also exposes session, span, record_event, record_metric, and
record_output for more detailed instrumentation.
evalon with no workflow subcommand opens the trace viewer. Dataset, suite,
judge, and eval workflows are available as:
evalon [--database PATH] dataset|suite|judge|eval|run ...The database is always the system file above unless you pass --database.
Dataset edits always publish a new immutable version. Prefer YAML for review:
# cases.yaml
dataset:
name: support-cases
description: Support agent regression cases
cases:
- case_id: refund-policy
input:
question: Can I request a refund?
expected_output:
answer: Refunds are available within 30 days.
tags: [refunds, p0]
- case_id: order-status
input:
question: Where is my order?
expected_output:
answer: Your order has been delivered.
tags: [orders]expected is accepted as an alias for expected_output. A bare list of cases
is also valid.
# Create dataset from the file header (dataset.name) or path stem
uv run evalon dataset import cases.yaml
# Or import into an existing dataset:
# uv run evalon dataset import support-cases cases.yaml
uv run evalon dataset show support-cases
uv run evalon dataset export support-cases cases.yaml
uv run evalon dataset diff support-cases --from-version 1 --to-version 2Add one case without a file:
uv run evalon dataset add support-cases \
--case-id refund-policy \
--input '{"question":"Can I request a refund?"}' \
--expected '{"answer":"Refunds are available within 30 days."}'Capture an observed trace (expected is never set silently):
uv run evalon dataset add support-cases \
--case-id observed-refund \
--from-trace TRACE_ID
# optional: --use-trace-output-as-expected | --use-trace-output-as-referenceA suite binds a dataset, a Python target (module:callable), and evaluators.
The target receives case.input and may be sync or async.
# suite.yaml
name: support regression
project: support-agent
version: "1"
target: agent:run_case
dataset_file: cases.yaml
evaluators:
- catalog: static.json_path_equal
params:
path: $.answer
expected: Refunds are available within 30 days.
- catalog: static.maximum_tool_calls
params:
max: 3
- type: python
name: no secrets
configuration:
callable: evals.judges:no_secret_leak
# Optional LLM judge (requires provider credentials):
# - catalog: llm.correctness
# params:
# provider: openrouter
# model: openai/gpt-4.1-miniApply and run:
uv run evalon suite apply suite.yaml
uv run evalon suite list
uv run evalon suite export "support regression" suite.exported.yaml
uv run evalon run suite.yaml --concurrency 4 --timeout-seconds 30
# or: uv run evalon run "support regression" --static-onlyevalon run path/to/suite.yaml applies the file (create or update by
project + name + version) then executes. Use --apply-only to load
without running. Per-case candidate timeouts can be set with
candidate_timeout_seconds in the suite file or --timeout-seconds on the CLI.
from evalon.evals import (
EvalStore,
EvalSuite,
EvaluatorDefinition,
EvaluatorType,
apply_suite_file,
)
store = EvalStore() # ~/.evalon/evalon-runs.sqlite
applied = apply_suite_file(store, "suite.yaml")
print(applied.suite.id)
# Or construct a suite directly:
suite = store.create_suite(
EvalSuite(
name="support regression",
project="support-agent",
dataset_id="DATASET_ID",
target_ref="agent:run_case",
evaluators=[
EvaluatorDefinition(
name="exact answer",
version="1",
evaluator_type=EvaluatorType.STATIC,
configuration={"type": "exact_match"},
),
],
)
)Built-in static evaluator types cover exact match, contains, excludes, regular expressions, JSON Schema, JSON path equality, required or forbidden tools, tool order and arguments, tool-call count, latency, token and cost limits, and candidate execution errors. Configuration is validated before any candidate case starts.
Custom evaluators are loaded only from an explicit module:callable
configuration. They receive the shared EvaluationContext. Exceptions become
an evaluator_error for that result and do not crash the run.
from evalon.evals import EvaluationContext, evaluator
@evaluator(name="no secret leak", version="1")
def no_secret_leak(context: EvaluationContext) -> dict:
text = str(context.candidate_output)
passed = "api_key" not in text
return {
"score": 1.0 if passed else 0.0,
"passed": passed,
"reason": "Candidate output must not expose an API key.",
"evidence": [],
}Evalon ships a catalog of static heuristics and binary LLM judges. List them:
uv run evalon judge catalogIn the TUI, open Datasets or Judges, select a dataset, press c
(Configure), check the evaluators you want, fill required values (for example
max tool calls or an LLM provider/model), then Save suite or
Save & run. Selections are bound to a suite (dataset + target + evaluators).
From Python:
from evalon.evals import EvalStore, EvalSuite, list_catalog, materialize
# Inspect the catalog
for entry in list_catalog():
print(entry.id, entry.evaluator_type.value, entry.name)
# Materialize selections into suite-ready definitions
max_tools, _ = materialize("static.maximum_tool_calls", {"max": 3})
correct, judge = materialize(
"llm.correctness",
{"provider": "openrouter", "model": "openai/gpt-4.1-mini"},
)
store = EvalStore()
if judge is not None:
judge = store.create_judge_definition(judge)
suite = store.create_suite(
EvalSuite(
name="support checks",
project="support-agent",
dataset_id="DATASET_ID",
target_ref="agent:run_case",
evaluators=[max_tools, correct],
)
)
if judge is not None:
store.bind_judge_to_suite(suite.id, judge.id)Predefined static types include exact match, contains/excludes, regex, JSON schema and path checks, required/forbidden tools, tool order and arguments, maximum or exact tool-call counts, latency/token/cost caps, and error-free execution. The five predefined LLM judges cover user goal completion, correctness and grounding, instruction following, tool and action quality, and safety and authorization.
Judges store the provider, model, rubric, temperature, retry policy, and
included context. Prefer declaring them in suite YAML via catalog: llm.*,
or create a reusable judge:
uv run evalon judge create correctness \
--provider openrouter \
--model openai/gpt-4.1-mini \
--rubric "Pass only if the answer is correct, complete, and supported."
uv run evalon judge test JUDGE_ID \
--input '{"question":"Can I request a refund?"}' \
--output '{"answer":"Yes, within 30 days."}' \
--expected '{"answer":"Refunds are available within 30 days."}'
uv run evalon judge attach JUDGE_ID --suite SUITE_IDEvery judge call creates its own trace linked to the candidate trace. Judge latency, tokens, errors, and cost are stored separately from candidate execution.
List catalog presets:
uv run evalon judge cataloguv run evalon run suite.yaml --concurrency 4
uv run evalon eval run SUITE_ID --concurrency 4
uv run evalon eval list --project support-agent
uv run evalon eval show RUN_ID
uv run evalon eval rerun RUN_ID
uv run evalon eval baseline RUN_ID
uv run evalon eval compare BASELINE_RUN_ID RUN_IDUse --background to execute candidates and judges in a worker process. The
command returns the durable run ID and worker PID immediately. Results remain
in SQLite after the CLI exits.
Foreground runs support minimum pass rate, maximum regression, maximum cost,
and maximum latency thresholds. Progress is written to stderr; use --json
for machine-readable stdout.
uv run evalon run suite.yaml \
--json \
--static-only \
--min-pass-rate 0.95 \
--max-cost 1.00 \
--max-latency-ms 5000
# Background worker: thresholds are checked after completion
uv run evalon run suite.yaml --background --min-pass-rate 0.95
uv run evalon eval show RUN_ID --wait --min-pass-rate 0.95
uv run evalon eval export RUN_ID \
--format junit --output evalon-junit.xmlExports support JSON, Markdown, CSV, and JUnit. Exit codes distinguish
configuration (2), execution (3), evaluation (4), threshold (5), and
storage (6) failures.
Evalon is local observability and evaluation software:
- Trace and evaluation data share one system SQLite database (
~/.evalon), or an explicitly selected JSONL/SQLite path when you override. - Observed trace rows are immutable from all dataset and evaluation workflows.
- The TUI uses a separate writable evaluation store for explicit dataset and run actions; its observability store remains read-only.
- There is no Supabase integration, cloud synchronization, hosted dashboard, annotation marketplace, or prompt playground.
- Static evaluators never make network requests. LLM judges run only when selected and confirmed, and their cost is reported separately.
- Set
EVALON_REDACT=0only if you intentionally want to disable the default redaction of common secret-like fields.
evalon/
├── common/ # errors, JSON helpers, system paths (~/.evalon)
├── observe/ # tracing SDK: client, traces, tools, providers
├── storage/ # SQLite/JSONL traces + EvalStore
├── evals/ # datasets, suites, runners, judges, evaluators
├── cli/ # dataset/suite/judge/eval/run workflows
├── tui/ # Textual observability + evaluation UI
├── web/ # local read-only HTTP API
└── clustering/ # optional semantic trace clustering
uv sync --all-extras --dev
uv run python -m compileall -q evalon
uv run pytestThe browser frontend reads the same local SQLite database as the terminal UI. It is intentionally read-only: recording, dataset changes, and evaluation execution remain CLI operations.
In one terminal, start the local API (defaults to ~/.evalon/evalon-runs.sqlite):
uv run python -m evalon.webIn a second terminal, start Vite from the repository's frontend directory:
cd /Users/sidmanale/Development/evalon/frontend
npm install
npm run devOpen the URL Vite prints, normally http://localhost:5173. The development
server proxies /api requests to the local API on port 8000.
Contributions that keep Evalon focused on reliable local observability and evaluation are welcome. Please include focused verification with every change.