fix(cli): the report predicted a tier the run had already decided

[AminChirazi]

The behaviour half of the defect #344 documented.

#341 made the run's containment tier win over the probe's, and check_egress has honoured that since. The report did not. It called containment(spec) before the run started, printed that, and reused it for the --json payload.

On Linux the two agree by construction, so this was invisible. On Windows the run is the authority — seven steps can fail after the probe passes — so a run that was contained printed not contained beside a trace lane saying the opposite.

That is worse than either being wrong alone. An auditor holding two artifacts that describe one run differently has no way to tell which is lying, and the artifact that is right is the one nobody printed.

The shape

record and replay now return (Containment, Result<(), String>) — the tier they achieved, alongside their outcome.

The tier leaves by out-parameter into a thin wrapper rather than through the return type, because it becomes known part way through the body: every ? before that point would otherwise have to name a tier it does not have yet. A run that failed before it started reports the prediction, which is the honest answer for a run that achieved nothing. That is also why it is a tuple rather than Result<Containment, _> — a failed run still has a tier worth reporting.

achieved_tier is one function, used by the certification path and the reporting path alike. Those two computing it separately is how they came to disagree in the first place.

The corrected line prints only when the run's answer differs from the prediction, so nothing changes on Linux and no output grows a line for a distinction that platform does not have. --json always carries the achieved tier.

Verification

• cargo test --workspace --all-features — 728 tests green
• cargo fmt --check — clean
• ratchets — diff size 107 <= 400, tests 773 -> 775, nothing silenced
• the_reported_tier_is_the_one_the_run_achieved was mutation-checked: making achieved_tier ignore the run and return the prediction fails it

Both directions are asserted, and the second is not implied by the first: where the run decided no tier — a url: service, a flow engaging no egress, a platform with no mechanism — the prediction stands. A run that determined nothing must not read as one that achieved nothing; those are different sentences.

Not fixed here, and it is the same defect

The run record still stores the predicted tier. build_control_record is called where the run is not in scope, so threading it there is a larger change than this one and would bury the fix above. Flagging it rather than leaving it to be rediscovered.

---

Generated by Claude Code

[Copilot]

Pull request overview

Fixes the CLI’s containment reporting for app: agent flows so stdout and --json reflect the tier actually achieved by the run (when the run determines one), rather than only the pre-run probe prediction. This aligns reporting with the certification/verdict path (check_egress) and prevents a single run producing conflicting artifacts.

Changes:

• Change agent_flow::record / agent_flow::replay to return (Containment, Result<(), String>), carrying the achieved tier alongside the run outcome.
• Introduce achieved_tier(run, spec_tier) as the shared rule for “report what the run achieved, otherwise fall back to the prediction”.
• Update CLI command paths to print the predicted tier first, and reprint only if the run’s achieved tier differs; --json always includes the achieved tier.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
crates/flowproof-cli/src/lib.rs	Updates record/run/suite agent-flow paths to print/report the achieved containment tier (with minimal extra stdout).
crates/flowproof-cli/src/agent_flow.rs	Adds achieved-tier computation and changes record/replay APIs to return achieved tier + outcome; adds regression tests.

Suppressed comments (2)

crates/flowproof-cli/src/lib.rs:721

• Same as the single-run path: predicted is computed here for printing, but agent_flow::replay also computes containment(spec) internally, duplicating host probes and making the tier != predicted comparison depend on two independently constructed values. Consider plumbing the precomputed predicted into replay so the suite path and replay logic stay strictly consistent.

    let predicted = agent_flow::containment(spec);
    if !json {
        println!("{}", predicted.report_line());
    }
    let (tier, outcome) = agent_flow::replay(spec, trace_path);
    if !json && tier != predicted {

crates/flowproof-cli/src/lib.rs:1303

• This path computes predicted via containment(&spec) and then calls agent_flow::replay, which recomputes containment(spec) internally. That duplicates the Windows host probe and makes tier != predicted compare values built by different probe calls. Consider passing the already-computed predicted into replay to avoid redundant probing and guarantee the comparison reflects only run-vs-probe differences.

        let predicted = agent_flow::containment(&spec);
        if !json {
            println!("{}", predicted.report_line());
        }
        let (tier, outcome) = agent_flow::replay(&spec, &trace_path);
        if !json && tier != predicted {

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.