fix(cli): the run record kept a tier the run had not achieved - #347
Merged
Conversation
The last of the three places that read a predicted tier instead of the achieved one. #345 fixed the printed report and the JSON; the run record — the artifact an auditor actually keeps — still called `agent_flow::containment(spec)` for itself. This one costs more than a wrong label. `containment` is read twice in `build_control_record`: once as the record's own tier line, and once to decide whether the blocked lane is evidence at all. So the two tiers disagreeing decides whether the destinations a run refused are carried or discarded — in either direction. An optimistic probe over a run that was not contained would present destinations nothing on this run blocked; a pessimistic one over a run that was contained would throw away the only proof it produced. So the achieved tier is passed in, `None` for a step-engine flow, which has no agent run to ask. The test asserts the PESSIMISTIC direction, and that is a deliberate retreat. The interesting production case is the opposite one — a Windows run that was contained, over a probe predicting otherwise — and it cannot be made falsifiable here. This suite runs on Linux, where the probe already answers Enforced, so a test asserting "the achieved Enforced won" passes identically when the achieved tier is ignored altogether. It was written that way first and it survived the mutation, which is the definition of a test that proves nothing. Turned around it pins the half that can fail here, and it is the safety-critical half: an uncontained run must not inherit an optimistic probe's evidence. Mutation-checked in that direction. The fixture guard on the blocked lane earned its keep again — the first draft wrote the cassette in the wrong shape, the lane failed to parse, and without the guard the emptiness assertion would have passed for the wrong reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XdXrbksFKirm7yW6EDunur
There was a problem hiding this comment.
Pull request overview
Updates the CLI’s run-record generation so containment tier and blocked-evidence attribution can be based on what the run actually achieved (rather than only the host probe prediction), addressing audit-evidence correctness.
Changes:
- Adds an
achieved: Option<&Containment>parameter tobuild_control_recordand prefers it overagent_flow::containment(spec)when present. - Threads achieved containment into the single-spec agent run path (
cmd_run) and updates other call sites withNone. - Adds a regression test asserting that an uncontained achieved tier does not inherit blocked evidence from an optimistic probe/fixture.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
1202
to
1206
| let control = match spec { | ||
| Some(spec) if spec.control.is_some() => { | ||
| let (verdict, reason) = flowproof_replay::ControlVerdict::from_run_report(report); | ||
| build_control_record(spec_path, dir, spec, verdict, reason) | ||
| build_control_record(spec_path, dir, spec, verdict, reason, None) | ||
| } |
AminChirazi
added a commit
that referenced
this pull request
Aug 1, 2026
Records #345 and #347, which are user-visible and unreleased. The Windows containment work they came out of is deliberately NOT written up here: no Windows job has been confirmed to run its end-to-end test, and an entry claiming a mechanism nobody has watched work is the kind of sentence this file exists not to contain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XdXrbksFKirm7yW6EDunur
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last of the three places that read a predicted tier instead of the achieved one. #345 fixed the printed report and the
--jsonpayload; the run record — the artifact an auditor actually keeps — still calledagent_flow::containment(spec)for itself.This one costs evidence, not just a label
containmentis read twice inbuild_control_record: once as the record's own tier line, and once to decide whether the blocked lane is evidence at all. So the two tiers disagreeing decides whether the destinations a run refused are carried or discarded — and it fails in both directions:The achieved tier is now passed in.
Nonefor a step-engine flow, which has no agent run to ask.The test asserts the pessimistic direction, and that is a retreat
Stating this plainly because the PR would otherwise look like it tests the wrong thing.
The interesting production case is the opposite one: a Windows run that was contained, over a probe predicting otherwise. It cannot be made falsifiable here. This suite runs on Linux, where the probe already answers
Enforced— so a test asserting "the achievedEnforcedwon" passes identically when the achieved tier is ignored altogether.I wrote it that way first. It survived the mutation, which is the definition of a test that proves nothing, and it would have shipped as a green tick that was never asked a question.
Turned around, it pins the half that can fail on this platform, and that half is the safety-critical one: an uncontained run must not inherit an optimistic probe's evidence. The test also asserts up front that the probe and the run genuinely disagree on this host, so it fails loudly rather than silently going vacuous somewhere the probe answers differently.
Mutation-checked in that direction: making
build_control_recordignore the achieved tier fails it.The fixture guard earned its keep again
The first draft wrote the cassette as
[]rather than{"turns": []}. The lane failed to parse, soblockedcame back empty — and the emptiness assertion would have passed for entirely the wrong reason. The guard that the neighbouring test added after exactly this bug caught it immediately.Verification
cargo test --workspace --all-features— greencargo fmt --check— cleandiff size 117 <= 400, tests775 -> 776, nothing silencedWith this, all three readers of the containment tier — stdout,
--json, and the run record — report what the run achieved.Generated by Claude Code