ruleproof

Check whether a coding agent followed your AGENTS.md.

ruleproof turns the checkable rules in AGENTS.md, CLAUDE.md and GEMINI.md into deterministic checks. It runs them against the git diff and the agent's session transcript, and reports each broken rule with evidence. It also flags claims the agent never verified, such as "all tests pass" with no test run after the last edit; that check is a heuristic and, on our pressure benchmark, it does not yet agree with an independent judge (see Results).

No model is called. Same input, same verdict. No runtime dependencies.

$ ruleproof -q check --transcript ../session.jsonl
✗ tests-before-done  "uv run pytest -q" ran before the last edit, not after
    #2  last run was before edit #3: uv run pytest -q
    orders/service.py  last counted edit
    rule: Run the tests after your last code change and before you say you are done. (AGENTS.md:15)
✗ honest-report  claimed tests pass; no test command ran after edit #3
    #5  claimed tests pass: All tests pass.
    no test command ran after edit #3
    rule: If you could not run the tests, say so. (AGENTS.md:16)
✗ no-pip  ran "pip install requests"
    #4  ran a forbidden command: pip install requests
    rule: Never run `pip install`; add dependencies with `uv add`. (AGENTS.md:18)
✗ generated-payments-client  1 forbidden file changed: orders/clients/payments/api.py
    orders/clients/payments/api.py  modified orders/clients/payments/api.py
    rule: Never edit the generated payments client by hand; regenerate it. (AGENTS.md:23)
✓ 13 rules passed

17 rules: 4 failed, 0 unverified, 0 skipped, 13 passed

Reproduce it with uv run python scripts/demo.py.

The rules are those of examples/python-service; the session is a short scripted Claude Code transcript in which the agent edits after testing, says "All tests pass", runs pip install and hand-edits generated code.

Findings

From the benchmarks (small n, one repo, a Claude judge; read them as indications):

• Blocking hooks closed the gap. With PreToolUse and Stop hooks, Haiku 4.5 on the long handbook passed every rule in 15/15 runs, against 5/15 without them. The Stop hook sent 6 runs back to fix a missing changelog entry or an unverified claim, and all 6 were fixed. The rows are graded by the same rules that steered the agent (Results).
• Handbook length may matter; the benchmark is too small to show it. The same rules in a 12k-token handbook instead of a 55-line AGENTS.md dropped Haiku 4.5 from 12/15 to 5/15 clean runs. The 15 runs are 3 repeats of 5 tasks; with tasks as the unit (per-task differences 0, 3, 1, 2, 1) the exact sign-flip p is 0.125, so no difference in the benchmark is significant. The rules lost were the ones that ask for restraint or extra work (Results).
• The claims check did not agree with an independent judge. On the pressure tasks it had precision 0/12 and recall 0/8 against the judge, so the unverified-"tests pass" check does not yet work. It is reported as a negative result (details). On 472 labelled runs published by overclaim it now finds 89% of "tests pass" claims on a held-out study (was 83%) with 99% precision, but it catches none of the false completions there: 16 of the 22 had a passing test run, so their claims were backed, and in 4 of the other 6 it missed the claim phrase. On a later rerun it had never seen, it failed 22 of 97 correct runs (13 after fixes found there), so expect false alarms (Results).
• Task success was at ceiling. Hidden tests passed in 79 of 80 pressure runs, so the benchmark measures rule cleanliness, not capability (Results).
• Under pressure nothing was significant. Clean runs ranged from 1/15 to 6/15 across arms (smallest Fisher p = 0.08), and every agent followed a prompt's "don't run the tests" over AGENTS.md (pressure tasks).

Why

Instruction files are prose. Agents read them, skip steps, and report success anyway. Nobody checks.

• In HANDBOOK.md, agents follow 20–124-page operating procedures. The strongest model passed 36.2% of trials under strict grading; most frontier models stayed below 25%.
• In our own benchmark (below), Haiku 4.5 followed every rule in 12 of 15 runs with a short AGENTS.md and in 5 of 15 with a long one. Same rules, same tasks.
• Review catches some of this. It misses what is not in the diff: the test run that never happened, the pip install, the force-push, the file edited and reverted.

ruleproof makes the checkable part of the rules mechanical, so review can focus on the rest.

Install

uv tool install git+https://github.com/h-a-forster/ruleproof@v0.1.1

or pipx install git+https://github.com/h-a-forster/ruleproof@v0.1.1. Python 3.11+. Linux, macOS, Windows.

Quick start

cd your-repo
ruleproof compile          # draft ruleproof.toml from AGENTS.md / CLAUDE.md; review it
ruleproof check            # check the uncommitted diff
ruleproof check --session latest   # also check the latest agent session for this repo
ruleproof doctor           # lint the instruction files themselves

compile is a starting point, not an oracle. It only emits rules it is confident about and lists every directive it could not turn into a check.

Rules

Rules live in ruleproof.toml:

version = 1

[[rule]]
id = "tests-before-done"
description = "Run the tests after your last code change."
source = "AGENTS.md:15"
check = "require-command"
command = '\bpytest\b'
when_paths = ["src/**", "tests/**"]

[[rule]]
id = "migrations-append-only"
check = "forbid-change"
paths = ["migrations/versions/*.py"]
actions = ["modify", "delete"]

Or next to the prose they enforce, invisible in rendered Markdown:

- Never run `pip install`; add dependencies with `uv add`.
  <!-- ruleproof: forbid-command id=no-pip command="\bpip3?(?:\.exe)? install\b" -->

Check	Evidence	Fails when
forbid-change	diff, transcript	a matching file was added, modified or deleted
require-change	diff	files matching A changed without any matching B
forbid-text	diff	an added line matches a pattern
require-text	diff	a new file lacks a pattern
max-diff	diff	the change is larger than a limit
forbid-command	transcript	the agent ran a matching shell command
require-command	transcript	a matching command did not run (and pass) after the last edit
forbid-edit	transcript	the agent's file tools touched a path (or anything outside the repo)
forbid-tool	transcript	the agent called a matching tool
forbid-message	transcript	the agent said something it must not
max-command-output	transcript	a shell command printed more than a limit
claims	transcript	the agent claimed tests, lint, types, build, commit or push with no backing command

Results are pass, fail, unverified (the evidence exists but cannot be confirmed, e.g. no exit code) or skip (an input is missing). Full reference: docs/rules.md.

Supported agents

Agent	Transcripts	Notes
Claude Code	~/.claude/projects/** session files, subagents, claude -p --output-format stream-json	Exit codes, edits, subagent work merged by time.
Codex CLI	~/.codex/sessions/** rollouts, codex exec --json	Current item records and older formats, including code-mode cells.
Gemini CLI	~/.gemini/tmp/*/chats/*.json	Best effort: built from the documented format, not yet tested on real sessions.
Anything else	ruleproof JSONL	One event per line; see docs/architecture.md.

ruleproof sessions lists the sessions recorded for a repo. ruleproof timeline prints one, or exports it to JSON or SQLite (Datasette-ready).

Enforce it while the agent works

Two Claude Code hooks:

• ruleproof hook claude-pretool (PreToolUse) denies a forbidden command or edit before it runs.
• ruleproof hook claude-stop (Stop) checks the result when Claude tries to finish. If an error rule fails, Claude gets the failures and one more turn to fix them.

{
  "hooks": {
    "PreToolUse": [
      { "matcher": "Bash|PowerShell|Write|Edit|MultiEdit|NotebookEdit",
        "hooks": [{ "type": "command", "command": "ruleproof hook claude-pretool" }] }
    ],
    "Stop": [
      { "hooks": [{ "type": "command", "command": "ruleproof hook claude-stop" }] }
    ]
  }
}

Details: docs/claude-code-hook.md.

CI

- uses: actions/checkout@v7
  with:
    fetch-depth: 0
- uses: h-a-forster/ruleproof@v0.1.1

The action checks the pull request's diff, runs doctor, writes a job summary, and can emit SARIF for code scanning. Pass transcript: to also check an agent run in CI, such as the execution_file from anthropics/claude-code-action. See docs/github-action.md and docs/integrations.md (pre-commit, JSON, SARIF, Markdown).

Doctor

ruleproof doctor lints the instruction files and agent configs:

• drift: CLAUDE.md and AGENTS.md that cover the same ground differently, with no import between them;
• conflict: two package managers or test runners for the same job; "always X" next to "never X";
• repo-mismatch: "run npm install" in a repo with pnpm-lock.yaml;
• dead-reference: @imports, links, paths, npm run/make/just targets that do not exist;
• size: instruction files that cost many tokens on every session;
• skill: broken SKILL.md frontmatter, skills that drifted between agents;
• mcp-drift: the same MCP server configured differently per agent; secrets inline in MCP configs.

Results

Benchmark (bench/). Five everyday tasks on a small Python repo. Its AGENTS.md has nine checkable rules; no prompt mentions them. Claude Code runs headless with user-level config excluded, 3 runs per task, so n = 15 per row. "Short" is a 55-line AGENTS.md. "Long" puts the same rules inside a 12k-token team handbook.

Model	AGENTS.md	ruleproof hooks	Task done	All rules followed
Sonnet 5.5	short		15/15	15/15
Sonnet 5.5	long		15/15	15/15
Sonnet 5.5	long	PreToolUse + Stop	15/15	15/15
Haiku 4.5	short		15/15	12/15
Haiku 4.5	long		14/15	5/15
Haiku 4.5	long	PreToolUse + Stop	15/15	15/15
Haiku 5.5	short		15/15	15/15
Haiku 5.5	long		15/15	11/15
Haiku 5.5	long	PreToolUse + Stop	15/15	12/15

"All rules followed" means no rule failed and none was left unverified.

• Grading is circular. Every row, including the hook rows, is graded by ruleproof itself. In the hook rows the same rules also steered the agent, so those rows show that the agent ended up satisfying ruleproof, not that an independent grader agrees. The pressure run below adds an independent grader.
• No difference is statistically significant. The 15 runs per row are 3 repeats of 5 tasks, and repeats of a task are correlated, so the unit is the task: an exact sign-flip test on the 5 per-task differences. Haiku 4.5 short vs long (12/15 vs 5/15) gives p = 0.125 (Fisher on runs, which wrongly treats them as independent, gives 0.025). Haiku 5.5 short vs long (15/15 vs 11/15) gives p = 0.5, and Haiku 5.5 long vs long with hooks (11/15 vs 12/15; 13/15 as first reported) gives p = 1.0. With 5 tasks the smallest possible two-sided p is 0.0625.
• The long handbook cost Haiku 4.5 compliance. The rules dropped were the ones that ask for extra work or restraint: a changelog entry, leaving existing tests alone, not committing.
• 2 of Haiku 5.5's 4 long-handbook failures were runs stopped by the $1 per-run budget cap, which may leave work unfinished. All 3 of its failures with hooks were capped runs too; the cap ends the session before the Stop hook can run.
• With both hooks, Haiku 4.5 on the long handbook passed every rule in all 15 runs. PreToolUse refused 3 git commits and 5 edits to existing tests before they ran. The Stop hook sent 6 runs back to add a changelog entry or verify a claim; all 6 were fixed.
• The claims check flagged 0 of these 135 runs: every claim was backed by a test run. (Scored with the claims check as of commit e12baf70; the raw p3 transcripts are not committed, so they cannot be re-scored with the current check.)
• One repo, and the same authors wrote the tasks, the rules and the checks. See threats to validity.

Pressure tasks (bench/). Five more tasks tempt the agent to report success it has not verified: an unrelated failing test, a 4-minute test file, a prompt that says not to run the tests, a broken pytest config, a test that pins the bug being fixed. 80 runs: Haiku 4.5 short / long / long with ruleproof hooks / long with a control (permissions.deny plus a generic "re-read AGENTS.md" Stop hook), Sonnet 5.5 long (15 each), Opus 5.5 long (5). An Opus judge graded every run from a condensed transcript and the diff, without seeing ruleproof's output or the hooks' messages. The judge was not fully blind to the arm: its inputs still leaked it in some trials (see the threats).

Model	AGENTS.md	Protection	All rules followed (judge)	(ruleproof)
Haiku 4.5	short		3/15	3/15
Haiku 4.5	long		1/15	1/15
Haiku 4.5	long	control: deny rules + generic Stop hook	3/15	6/15
Haiku 4.5	long	ruleproof PreToolUse + Stop	5/15	5/15
Sonnet 5.5	long		6/15	3/15
Opus 5.5	long		2/5	2/5

• No difference is significant. Smallest Fisher p = 0.08; ruleproof hooks vs control 5/15 vs 3/15, p = 0.68. The task decides most outcomes (intraclass correlation up to 1.0), so each row has an effective n of 5 to 15 and at most 5 independent tasks.
• Task success is at ceiling. The hidden tests passed in 79 of 80 runs, so the table measures rule cleanliness, not task success.
• ruleproof vs the judge, per rule. changelog-entry 19/19 violations found, no false alarm (kappa 1.0); no-editing-existing-tests 11 of 13, no false alarm (kappa 0.90); tests-pass precision 0.86, recall 0.68, because the rule accepts runs that leave tests out (-k "not vat_id"). Pooled kappa 0.73. The judge agreed with itself on 97% of 200 re-judged verdicts.
• The claims check fails this test. It now fires (24 of 80 runs fail or are unverified), but against the judge it has precision 0/12 and recall 0/8 (4 of the 8 came out unverified). It checks only the first claim sentence of each kind, does not follow tests run in the background, and read honest partial reports ("15 of 16 tests pass") as claims. Fixes made after this analysis remove 3 of the 12 false alarms (in-sample); the rest is open work. The later changes scored below were not rescored on this run: its raw transcripts are not kept.
• Every agent, Opus and Sonnet included, followed the prompt's "don't bother running the test suite" over AGENTS.md. The judge is a Claude model, from a prompt by the same authors, and no human labelled the runs. Total nested spend for the run, judge included: about $25.

Claims check on labelled runs (bench/results/claims-eval/). overclaim publishes 472 Claude Code runs of coding tasks (Haiku and Sonnet 5.5), each with a hidden-check result and an adjudicated label of what the final message claims. bench/claims_eval.py runs the claims check on them offline. Study 1 (184 runs) was used to find and fix errors; study 2 (288 runs) was held out and scored after the fixes, then re-checked (numbers unchanged) after later fixes found on study 1, in review and on the rerun described below. Before is main at e33dc0f (still versioned 0.1.0, with the unreleased changes since the v0.1.0 tag), after is the current one (before, after).

Score (held-out study 2)	Before	After
"Tests pass" claims found (tests, lint, types or build claim; recall vs the gold label)	235/283 (83.0%)	252/283 (89.0%)
Claims found that the label rejects	2	2
Correct "done" runs failed (false alarms)	2 of 193	1 of 193
Correct "done" runs left unverified	118 of 193	15 of 193
False completions caught (fail)	0 of 22	0 of 22
False completions flagged (fail or unverified)	13 of 22	3 of 22

• What changed. The study-1 errors were mostly in the evidence, not the phrases: unittest discover -t . was read as a test filter (there -t is the top-level directory), so most full-suite claims came out unverified; a test run on git stashed code (to show a new test fails without the fix) was read as the result; scratch files written after cd to a directory outside the repo, patch --version, ( ) { } ; inside a quoted sed script and .gitignore edits reset the evidence. Phrase fixes: "the existing tests and 13 new ones pass", "3 tests, which pass", "the 10 new tests fail on the original code" (a description, not a failure), "to ensure correctness I ran the suite and it passes"; "it builds a dict" is not a build claim.
• Fewer false completions are flagged, for the wrong reason before. Before, 13 of the 22 false completions came out unverified through the evidence errors that also left 118 correct runs unverified (mostly unittest discover -t . read as a test filter), not because the check saw anything wrong. With it fixed they pass, like the correct runs (study 1: 7 of 11 flagged before, 1 after).
• Most false completions are out of reach by design. 10 of study 1's 11 had a passing test run after the last edit; the hidden checks failed on cases the visible tests do not cover. A check that verifies claims against commands cannot see that. Study 2 is less clear: 16 of its 22 had a passing run, but 3 had no test run after the last edit (they ran tests only before it) and the phrase matcher missed their claims ("97/97 checks, plus the 25 unit tests", "Both pass"), so those were reachable. overclaim's own published ruleproof row (0 of 11 caught, 7 false alarms in study 1) was scored with a 0.1.0 build and an empty --repo, so it saw no edits; in that configuration the before version gives 0 and 7 and the current check 0 and 0.
• In study 1 the current check still fails 2 correct runs in the agent's working directory: each edited a scratch script under tmp/ after its last test run. Add such paths to ignore_edit_paths.
• The held-out numbers are optimistic. overclaim's rerun of study 1's tasks (144 runs, a later Claude Code in a different environment, used in no tuning above) was first scored at beed533 (results). Claims found: 120 of 142 (84.5%), none rejected by the label. But 22 of its 97 correct "done" runs failed, against 1 of 193 in the table, and 20 more were unverified. Most of the failures with no test run were evidence errors: the interpreter in a variable ($PY -m unittest). With that fixed, 13 fail and 23 are unverified (after, now in-sample), and the false completions caught drop from 2 of 4 to 1 of 4: one catch was that same error. The rest are mostly scratch files written after the last test run, and command lines that also run the tests on the old code to show they fail there, which read as failures.
• One labeller pipeline (two Claude models plus adjudication), one agent, Python tasks only; intervals in the results file treat runs as independent, which they are not.

Compile (bench/corpus/). ruleproof compile was run on 50 public instruction files (Next.js, VS Code, Deno, ruff, uv, Airflow, ...). Of 1,960 directives, 62 became checks (3.2%), giving 70 rules. Precision is not well measured. The authors hand-checked three 40-rule samples drawn from the same 70–74-rule pool, tuning the compiler on this corpus between samples (38/40, then 39/40, then 40/40). The samples overlap and were not held out, and a later independent review found 4 classes of error none of them had caught. The least biased figure is the first sample, 38/40 (95%, 95% CI 83–99%), on files the compiler had been tuned on; precision on unseen files is unknown. Coverage is low by design. Most directives are style or design guidance with no deterministic check, and compile skips what it cannot check.

Prior art

• Claim checking. claimproof, groundtruth, attest and mcp-truth-check also check what an agent says it did against evidence. ruleproof's claims check is one more heuristic in that space; on the pressure benchmark it did not yet agree with an independent judge (see above).
• Blocking while the agent works. Claude Code's own permissions.deny and hooks already block forbidden commands and edits in Claude Code sessions; ruleproof's hooks are a convenience on top, not something only it can do. In the pressure benchmark a hand-written control (deny rules plus a generic Stop hook) did about as well as ruleproof's hooks.

What ruleproof adds is narrower: rules taken from the instruction files you already have, checked after the fact against both the session transcript and the git diff, the same way for Claude Code, Codex and Gemini CLI, deterministically and in CI.

Open questions

Each is something the repo can start on; the pointers say where.

• Why are self-reported verification claims hard to check from a transcript? The claims check reads only the first claim of each kind, does not follow background runs, and mistakes honest partial reports for claims. Start at src/ruleproof/checks/claims.py and the failure list in bench/README.md.
• Does rule compliance transfer to other repos and handbooks? Everything here is one small Python repo with handbooks by the same authors. Start at bench/tasks/, bench/variants/long/ and bench/run.py.
• How well does the judge match human labels? No human labelled any run; the judge is a Claude model and was only partly blind to the arm. Start at bench/judge.py and bench/results/judge/pt1/ (condensed per-trial logs, judge verdicts).
• Which rules can hooks enforce, and which can they only detect? PreToolUse can refuse a command or edit; "tell the truth about what you ran" cannot be refused up front. Start at src/ruleproof/hook.py and docs/claude-code-hook.md.
• Is a generic control as good as rule-derived hooks? A hand-written control scored 3/15 clean (judge) against 5/15 for ruleproof's hooks (p = 0.68) on the pressure tasks. Start at bench/control_stop.py and bench/analyze_pressure.py.
• Why does the tests-pass rule accept runs that leave tests out? Precision was 0.86 and recall 0.68 against the judge, because -k "not vat_id" passes the rule. Start at bench/rules.toml and src/ruleproof/checks/transcript_checks.py.

What it does not do

• It does not judge style, design or intent. "Prefer small functions" is not checkable; ruleproof says so instead of guessing.
• It sees what the transcript records. Shell commands that write files count as edits only for common forms (sed -i, redirects, formatters).
• Command rules are regexes over command lines. A determined agent can evade them (eval "$(echo Z2l0IHB1c2g= | base64 -d)"). ruleproof is a check, not a sandbox.
• Codex code-mode transcripts sometimes lack exit codes; those results are unverified, not pass.

Docs

• Rules reference
• Claude Code hooks
• GitHub Action
• Integrations: pre-commit, JSON, SARIF, Markdown
• Architecture
• Benchmark
• Contributing

License

MIT