Decision Contract Audit

An Inspect AI extension that audits an LLM (or any) decision point against six reliability contracts by replaying a frozen decision log. No model calls, no API keys: every run goes through Inspect's mockllm and exits with a machine-checkable report.

The six contract families:

Family	Question it answers
validity	Is every replayed decision one of the declared options?
stability	Does the same state get the same decision across repeats?
invariant	Does any decision break a pre-registered forbidden-action rule?
monotonicity	Does strictly improved evidence ever move the decision later in the hesitancy order?
representation	Do two renderings of the same state diverge?
cage	Post-hoc certification: which caller-selected cases your cage rules cover, would override, or leave residual

Scope is deliberate: this is post-hoc certification of caller-selected cases, not runtime prevention. A residual is a supplied case outside every cage rule; it is not automatically unsafe unless the caller selected it on that basis.

Upstream contributions

The question these families ask — does the documented contract match actual behavior, and does the failure path close? — has also been put to other projects' shipped code. As of 2026-07-28, nine findings have been reported to nine published LLM evaluation and guardrail projects — six pull requests and four issues, Sponsio carrying both — each filed through that project's own process, with a keyless reproducer and, where one was offered, a patch and a regression test.

No patch offered has been merged. The one finding that reached a resolution, Sponsio, was fixed by the maintainers' own pull request, in a safer direction than the one proposed.

Reproduction records for all nine: docs/upstream.

Install

pip install decision-contract-audit

Quickstart

The runnable example lives in the repository, not the distribution:

git clone https://github.com/CTWalk/decision-contract-audit
cd decision-contract-audit
pip install .
python examples/custom_decision_point.py

The example replays a synthetic loan-approval triage log with planted violations in five of the six families and exits nonzero unless every one is detected — it doubles as the repository's keyless self-check (the same command CI runs).

Wire your own decision point

A decision point is: an ordered set of options (least → most hesitant), a set of states, decisions for those states (a replay log, a dict, or a callable), and the rules you want enforced.

from decision_contracts import run_contracts

report = run_contracts(
    decision_order=["approve", "review", "decline"],  # least → most hesitant
    states={"low": {"score": 1}, "high": {"score": 3}},
    decisions={"low": "review", "high": "approve"},   # or a log, or a callable
    invariant_rules=[{
        "name": "score floor",
        "predicate": {"field": "score", "op": "<", "value": 2},
        "forbidden": ["approve"],
    }],
    monotonic_pairs=[("low", "high")],                # optional
    cage_cases=[{"state_id": "high", "decision": "approve"}],
    cage_rules=[{                                     # optional
        "name": "hard floor",
        "predicate": {"field": "score", "op": "<", "value": 1},
        "forbidden": ["approve"],
        "fallback": "review",
    }],
)

print(report["passed"], "—", report["reason"])
for family, row in report["contracts"].items():
    print(f"{family:15} {'PASS' if row['passed'] else 'FAIL'} — {row['reason']}")

Expected output: every configured family passes except cage, which fails with 1 valid decision(s) were residual to all cage rules — the caller selected the high approval for certification, but the toy rule (score < 1) does not cover it. Decide whether that gap is acceptable or tighten the rule set.

report["contracts"]["cage"] carries the certification counts first-class: covered_count, overridden_count, and residual_count. Supplying cage_rules requires an explicit cage_cases set, so ordinary benign decisions are never silently relabeled as residuals.

Malformed inputs fail closed: an unknown state, an unresolvable pair, an unsupported rule operator, or a decision outside decision_order produces a failing score with a reason, never a silent pass.

For finer control (running inside your own Inspect pipeline), use build_contract_task(...), which returns a plain Inspect Task.

Provenance

• Built and verified against inspect-ai 0.3.248 (2026-07-20); requires Python ≥ 3.10, matching Inspect AI's floor (CI exercises 3.10 and 3.12). The scorers replay pre-recorded data through a no-generation solver, and the example fails unless Inspect reports zero model usage.
• Development was AI-assisted under human review — see NOTICE.

License

Apache-2.0 — see LICENSE and NOTICE.