codesteward-prompt-evaluator

Evaluates summarizer output quality with a strong LLM and automatically optimizes prompts via A/B validation.

---

What it does

codesteward-prompt-evaluator is a batch service that scores outputs produced by codesteward-session-summarizer using a strong hosted model (Claude, GPT-4o), and optionally generates improved prompt variants through an automated optimization loop.

The evaluator communicates with the summarizer exclusively through shared ClickHouse tables — no runtime dependency, no APIs, no message queues.

---

Features

• Two-layer evaluation — chunk extractions scored on 4 aggregate dimensions + 11 per-category scores; summaries scored on 5 dimensions — pinpoints whether quality issues originate at extraction or synthesis
• Automated prompt optimization — analyzes low/high-scoring outputs, generates candidate prompts, validates via paired A/B comparison, and auto-promotes if the win margin is met
• Safety guardrails — lineage depth limits prevent unbounded prompt drift, auto-rollback reverts on post-promotion score drops, one-active-per-role invariant enforced
• Statistical sampling — evaluates a configurable random sample instead of every output, bounding LLM costs regardless of session volume
• Quality trend reporting — generates tabular reports from stored evaluation results with no LLM calls
• Dual provider support — Anthropic and OpenAI SDKs; no local models (evaluator requires strong hosted models)

---

Architecture

  Summarizer tables (read-only)              Evaluator tables (owned)
  ─────────────────────────────              ────────────────────────
  session_chunk_extractions ───┐         ┌──► chunk_evaluation_results
  chunk_evaluation_contexts ───┤         │
  session_summaries ───────────┤         ├──► summary_evaluation_results
  summary_evaluation_contexts ─┤         │
                               ▼         │
                    ┌─────────────────┐   ├──► prompt_optimization_runs
                    │    Evaluator    │───┘
                    │                 │
                    │  sample → score │──────► Claude / GPT-4o
                    │  optimize → A/B │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │ prompt_registry  │  (shared read/write)
                    └─────────────────┘

---

Quick start

Prerequisites: Python 3.12+, uv, a running ClickHouse instance with summarizer migrations applied (010-012).

1. Apply the evaluator schema

clickhouse-client --multiquery < migrations/001_evaluation_results.sql
clickhouse-client --multiquery < migrations/002_prompt_optimization_runs.sql

2. Run evaluation

uv sync

ANTHROPIC_API_KEY=sk-... uv run python -m evaluator.main

3. Try other modes

# Generate and validate prompt candidates
RUN_MODE=optimize ANTHROPIC_API_KEY=sk-... uv run python -m evaluator.main

# Quality trend report (read-only, no LLM calls)
RUN_MODE=report uv run python -m evaluator.main

# Manually promote a candidate
RUN_MODE=promote uv run python -m evaluator.main \
  --prompt-role extraction --prompt-id extraction-v2

---

Docker

docker build -t codesteward-prompt-evaluator .

docker run --rm \
  -e ANTHROPIC_API_KEY=sk-... \
  -e CLICKHOUSE_URL=http://host.docker.internal:8123 \
  codesteward-prompt-evaluator

---

Run modes

Mode	Description	LLM calls
evaluate	Score a sample of recent outputs, store results in ClickHouse	Yes
optimize	Run evaluation, then generate and validate prompt candidates	Yes
promote	Manually promote a specific candidate prompt to active	No
report	Generate a quality trend report from stored evaluation results	No

---

Configuration

All via environment variables. No config files.

Variable	Default	Description
CLICKHOUSE_URL	http://localhost:8123	ClickHouse HTTP interface
CLICKHOUSE_USER	default	ClickHouse user
CLICKHOUSE_PASSWORD	""	ClickHouse password
CLICKHOUSE_DATABASE	audit	Database name
EVALUATOR_PROVIDER	anthropic	LLM provider: openai or anthropic
EVALUATOR_MODEL	claude-sonnet-4-6	Strong model for scoring and optimization
OPENAI_API_KEY	""	OpenAI API key (required when provider=openai)
ANTHROPIC_API_KEY	""	Anthropic API key (required when provider=anthropic)
RUN_MODE	evaluate	evaluate | optimize | promote | report
SAMPLE_SIZE	20	Number of recent outputs to evaluate per run
EVALUATION_LOOKBACK_DAYS	7	How far back to sample outputs
MIN_PROMOTION_SAMPLE	20	Minimum samples required to promote a candidate
WIN_MARGIN_THRESHOLD	0.3	Minimum avg score improvement to auto-promote
MAX_AUTO_LINEAGE_DEPTH	5	Max auto-generated prompt generations before human review
AUTO_ROLLBACK_THRESHOLD	0.5	Score drop that triggers automatic rollback
LOG_LEVEL	info	Logging level

---

Evaluation dimensions

Chunk extractions

Dimension	What it measures
completeness	Did it extract all visible facts from the context?
accuracy	Are extracted facts correct vs the input context?
hallucination_free	No facts invented beyond what the context shows?
structure	Proper format, clear entries, actionable wording?

Plus per-category scores (1-5) for each of the 11 fact categories.

Summaries

Dimension	What it measures
completeness	Does the summary cover all key activities?
accuracy	Are stated facts correct vs the input context?
conciseness	Appropriate length, no filler or redundancy?
decision_coverage	Are key_decisions complete and well-stated?
tag_relevance	Are tags meaningful and appropriate?

---

Safety guardrails

Guard	What it prevents
Never hot-swap	A candidate must pass A/B validation before becoming active
One active per role	Promoting a candidate automatically retires the previous active prompt
Minimum sample size	Require N sessions before comparing scores
Win margin threshold	Candidate must score meaningfully better, not just within noise
Lineage depth limit	After N auto-generations, require human review
Auto-rollback	If post-promotion scores drop below threshold, revert automatically

---

Repository structure

├── src/evaluator/
│   ├── main.py               Entry point, run mode routing
│   ├── config.py              Pydantic Settings (env vars)
│   ├── models.py              Dataclasses for results, contexts, prompts
│   ├── llm.py                 LLM client (Anthropic + OpenAI)
│   ├── clickhouse.py          Read contexts/outputs, write results, manage prompts
│   ├── sampling.py            Select outputs for evaluation
│   ├── evaluation.py          Build evaluator prompts, parse scores
│   ├── optimization.py        Generate candidate prompts, validate, compare
│   ├── promotion.py           Promote/retire/rollback prompt status transitions
│   └── reporting.py           Quality trend reports from evaluation results
├── tests/                     Unit tests (mocked ClickHouse + LLM calls)
├── migrations/
│   ├── 001_evaluation_results.sql
│   └── 002_prompt_optimization_runs.sql
├── .github/workflows/
│   ├── ci.yml                 Lint, test, build, Docker build on push/PR
│   └── release.yml            Test, package, Docker push, GitHub Release on tag
├── pyproject.toml
└── Dockerfile

---

Development

# Install with dev dependencies
uv sync

# Run tests
uv run pytest

# Lint and format
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/