Skip to content

fix(cli): the run record kept a tier the run had not achieved - #347

Merged
AminChirazi merged 1 commit into
mainfrom
claude/run-record-achieved-tier
Aug 1, 2026
Merged

fix(cli): the run record kept a tier the run had not achieved#347
AminChirazi merged 1 commit into
mainfrom
claude/run-record-achieved-tier

Conversation

@AminChirazi

Copy link
Copy Markdown
Contributor

The last of the three places that read a predicted tier instead of the achieved one. #345 fixed the printed report and the --json payload; the run record — the artifact an auditor actually keeps — still called agent_flow::containment(spec) for itself.

This one costs evidence, not just a label

containment is read twice in build_control_record: once as the record's own tier line, and once to decide whether the blocked lane is evidence at all. So the two tiers disagreeing decides whether the destinations a run refused are carried or discarded — and it fails in both directions:

  • an optimistic probe over a run that was not contained presents destinations nothing on this run blocked;
  • a pessimistic probe over a run that was contained throws away the only proof that run produced.

The achieved tier is now passed in. None for a step-engine flow, which has no agent run to ask.

The test asserts the pessimistic direction, and that is a retreat

Stating this plainly because the PR would otherwise look like it tests the wrong thing.

The interesting production case is the opposite one: a Windows run that was contained, over a probe predicting otherwise. It cannot be made falsifiable here. This suite runs on Linux, where the probe already answers Enforced — so a test asserting "the achieved Enforced won" passes identically when the achieved tier is ignored altogether.

I wrote it that way first. It survived the mutation, which is the definition of a test that proves nothing, and it would have shipped as a green tick that was never asked a question.

Turned around, it pins the half that can fail on this platform, and that half is the safety-critical one: an uncontained run must not inherit an optimistic probe's evidence. The test also asserts up front that the probe and the run genuinely disagree on this host, so it fails loudly rather than silently going vacuous somewhere the probe answers differently.

Mutation-checked in that direction: making build_control_record ignore the achieved tier fails it.

The fixture guard earned its keep again

The first draft wrote the cassette as [] rather than {"turns": []}. The lane failed to parse, so blocked came back empty — and the emptiness assertion would have passed for entirely the wrong reason. The guard that the neighbouring test added after exactly this bug caught it immediately.

Verification

  • cargo test --workspace --all-features — green
  • cargo fmt --check — clean
  • ratchets — diff size 117 <= 400, tests 775 -> 776, nothing silenced

With this, all three readers of the containment tier — stdout, --json, and the run record — report what the run achieved.


Generated by Claude Code

The last of the three places that read a predicted tier instead of the
achieved one. #345 fixed the printed report and the JSON; the run record
— the artifact an auditor actually keeps — still called
`agent_flow::containment(spec)` for itself.

This one costs more than a wrong label. `containment` is read twice in
`build_control_record`: once as the record's own tier line, and once to
decide whether the blocked lane is evidence at all. So the two tiers
disagreeing decides whether the destinations a run refused are carried
or discarded — in either direction. An optimistic probe over a run that
was not contained would present destinations nothing on this run
blocked; a pessimistic one over a run that was contained would throw
away the only proof it produced.

So the achieved tier is passed in, `None` for a step-engine flow, which
has no agent run to ask.

The test asserts the PESSIMISTIC direction, and that is a deliberate
retreat. The interesting production case is the opposite one — a Windows
run that was contained, over a probe predicting otherwise — and it
cannot be made falsifiable here. This suite runs on Linux, where the
probe already answers Enforced, so a test asserting "the achieved
Enforced won" passes identically when the achieved tier is ignored
altogether. It was written that way first and it survived the mutation,
which is the definition of a test that proves nothing.

Turned around it pins the half that can fail here, and it is the
safety-critical half: an uncontained run must not inherit an optimistic
probe's evidence. Mutation-checked in that direction.

The fixture guard on the blocked lane earned its keep again — the first
draft wrote the cassette in the wrong shape, the lane failed to parse,
and without the guard the emptiness assertion would have passed for the
wrong reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XdXrbksFKirm7yW6EDunur
Copilot AI review requested due to automatic review settings August 1, 2026 13:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the CLI’s run-record generation so containment tier and blocked-evidence attribution can be based on what the run actually achieved (rather than only the host probe prediction), addressing audit-evidence correctness.

Changes:

  • Adds an achieved: Option<&Containment> parameter to build_control_record and prefers it over agent_flow::containment(spec) when present.
  • Threads achieved containment into the single-spec agent run path (cmd_run) and updates other call sites with None.
  • Adds a regression test asserting that an uncontained achieved tier does not inherit blocked evidence from an optimistic probe/fixture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 1202 to 1206
let control = match spec {
Some(spec) if spec.control.is_some() => {
let (verdict, reason) = flowproof_replay::ControlVerdict::from_run_report(report);
build_control_record(spec_path, dir, spec, verdict, reason)
build_control_record(spec_path, dir, spec, verdict, reason, None)
}
@AminChirazi
AminChirazi merged commit 8d0235d into main Aug 1, 2026
10 checks passed
AminChirazi added a commit that referenced this pull request Aug 1, 2026
Records #345 and #347, which are user-visible and unreleased. The
Windows containment work they came out of is deliberately NOT written up
here: no Windows job has been confirmed to run its end-to-end test, and
an entry claiming a mechanism nobody has watched work is the kind of
sentence this file exists not to contain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XdXrbksFKirm7yW6EDunur
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants