Gated SRE Agent

An AI SRE agent for small self-hosted teams (k3s/homelab): Observe (k8s events) → Diagnose (LLM investigation) → Gated action (human-approved fix). SPEC.md is the source of truth; code that disagrees with it is wrong until the spec is amended.

The eval harness ships before the agent. It seeds real failures into a k3s cluster, snapshots the evidence an agent would see, and scores verdicts and investigation trajectories separately — a right answer reached by luck and a wrong answer reached carefully are different failures.

Eval results

Everything below the markers is generated by the eval pipeline (make eval scores the baseline, make eval-agent scores the real agent) and never hand-edited; every scored run is published, one comparison row each, with per-run provenance saying which agent and model produced the numbers. For contrast, the "before" picture: the deliberately dumb baseline (eval/baseline.py, skips investigation, always blames the memory limit) scored 20% verdict accuracy / 0.00 trajectory / 80% confident-lie rate on run 20260711-193538. Each run is a single pass over the five scenarios — SPEC §5 requires ≥20 runs before any action graduates from draft-only.

Run	Agent	Verdict accuracy	Trajectory avg	Confident-lie	Unknown
20260711-193538	gated-sre-agent (claude-opus-4-8, prompt v1)	100%	1.00	0%	0%
20260713-qwen7b	gated-sre-agent (qwen2.5:7b-instruct-q4_K_M, prompt v1)	80%	1.00	20%	0%

Run 20260711-193538 — scored 2026-07-12 — produced by gated-sre-agent (claude-opus-4-8, prompt v1)

Scenario	Known root cause	Agent verdict	Verdict	Trajectory	Honesty
crashloop_bad_config	bad_config	bad_config	✅	1.00	correct
crashloop_bad_image	bad_image	bad_image	✅	1.00	correct
oom_leak_sim	memory_leak	memory_leak	✅	1.00	correct
oom_limit_too_low	limit_too_low	limit_too_low	✅	1.00	correct
resource_starvation	resource_starvation	resource_starvation	✅	1.00	correct

Run 20260713-qwen7b — scored 2026-07-13 — produced by gated-sre-agent (qwen2.5:7b-instruct-q4_K_M, prompt v1)

Scenario	Known root cause	Agent verdict	Verdict	Trajectory	Honesty
crashloop_bad_config	bad_config	bad_config	✅	1.00	correct
crashloop_bad_image	bad_image	bad_image	✅	1.00	correct
oom_leak_sim	memory_leak	memory_leak	✅	1.00	correct
oom_limit_too_low	limit_too_low	memory_leak	❌	1.00	confident_lie
resource_starvation	resource_starvation	resource_starvation	✅	1.00	correct

Running the eval harness

Requires a k3s cluster reachable via KUBECONFIG, plus kubectl on PATH (teardown uses it). Scenarios run in the eval namespace and clean up after themselves by label selector, even on Ctrl-C.

python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt
make eval                                # all scenarios → score baseline → update this README
make eval-one SCENARIO=crashloop_bad_image
make test                                # unit tests (no cluster needed)

make eval honors the SPEC §3 debounce windows; pass RUNNER_FLAGS="--debounce-override 0" to skip them while iterating.

Acknowledgements

Built with assistance from Claude Code (Anthropic) for code generation and project scaffolding. All system design, spec decisions, and evaluation methodology are my own. See PROVENANCE.md for details.