Skip to content

fix(core): a sandboxed probe non-pass is inconclusive, not a broken tool - #231

Open
Max17190 wants to merge 1 commit into
mainfrom
probe-honest-about-what-it-cannot-prove
Open

fix(core): a sandboxed probe non-pass is inconclusive, not a broken tool#231
Max17190 wants to merge 1 commit into
mainfrom
probe-honest-about-what-it-cannot-prove

Conversation

@Max17190

@Max17190 Max17190 commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Why

openmax --check --run-examples probes an unapproved tool in a sandbox that denies the network and any write outside its scratch dir (ADR-0011). A tool that legitimately needs either, a web or API tool or a tool that writes a real file, can never pass that probe, yet the failure was reported as err and exited the check nonzero, identical to a genuinely broken tool. That fails CI on the largest tool family and steers an agent to "fix" working code. Dogfooding a keyless web-search tool reproduced it: err example ... Could not resolve host and exit 1 on a correct tool; in another run the sandbox's write confinement even led the agent to delete a valid [example].

Summary

A passing probe approves nothing (ADR-0011), so a failing one condemns nothing: a sandboxed non-pass is inconclusive.

  • It now reports warn, names why (the sandbox denies network and non-scratch writes), and does NOT fail the check (both the terminal and the --json faces).
  • The approved host run is unchanged: it still runs with host authority and still errs on real breakage, which is the honest verdict for such a tool.
  • --spec tools states the limitation and points at approve-then-rerun for the real proof of life.

Tradeoff, stated plainly: a genuinely broken unapproved tool no longer errs at probe time either, because the sandbox cannot tell "broken" from "denied a capability it needs"; it surfaces at the approved host run instead. The false-err on every network tool is the larger harm, and the warn still shows the failure and its reason.

Test Plan

  • cargo test -p open-max-tui: run_examples_is_gated_and_reported_through_json updated to the new contract and shown red on the old exit-1 behavior; new an_unapproved_sandbox_non_pass_warns_and_does_not_fail_the_check.
  • cargo test --workspace, cargo clippy --workspace --all-targets at zero warnings.
  • Manual: a keyless web tool now reports warn example ... could not be proven in the sandbox and --check --run-examples exits 0; after --approve, the real run behaves as before.

Greptile Summary

This change treats unsuccessful unapproved sandbox probes as advisory warnings, while keeping approved examples that run with host authority as failing checks.

A real-binary check exercised a passing example and an example that exits nonzero through both terminal and JSON output. Unapproved sandbox runs reported warnings and exited successfully; after approval, the same failing command reported an error and exited nonzero. Focused CLI integration tests covering both paths also passed.

Confidence Score: 5/5

Safe to merge based on the exercised example-run contracts.

The exercised terminal and JSON flows preserve the intended distinction between an inconclusive sandbox probe and a failing approved host-authority run, with focused integration coverage passing.

Files Needing Attention: No additional files need attention.

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex ran a deterministic real-binary harness against the parent behavior and the PR change, using one passing example and one command that exits 7.
  • T-Rex observed that the unapproved failing example changed from a terminal error with exit 1 to a terminal and JSON warning with exit 0.
  • T-Rex approved the same example and observed terminal and JSON error output with exit 1, confirming that warning handling does not suppress host-authority failures.
  • T-Rex ran the focused CLI integration tests covering the JSON contract and terminal warning behavior; both tests passed.
  • T-Rex reviewed the base-commit and PR-change logs to confirm the evolution of behavior and verified the focused integration tests also passed.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (1): Last reviewed commit: "fix(core): a sandboxed probe non-pass is..." | Re-trigger Greptile

openmax --check --run-examples probes an unapproved tool in a sandbox that
denies the network and any write outside its scratch dir (ADR-0011). A tool
that needs either - a web or API tool, a tool that writes a real file - can
never pass that probe, yet the failure was reported as `err` and exited the
check nonzero, identical to a genuinely broken tool. That fails CI on the
largest tool family and steers an agent to "fix" working code.

A passing probe approves nothing, so a failing one condemns nothing: treat a
sandboxed non-pass as inconclusive. It now reports `warn`, names why (the
sandbox denies network and non-scratch writes), and does NOT fail the check.
The honest verdict for such a tool is its approved host run, which still errs
on real breakage - the approved path is unchanged. --spec tools states the
limitation and points at the approve-then-rerun proof.

Tradeoff, stated plainly: a genuinely broken UNAPPROVED tool no longer errs at
probe time either (the sandbox cannot tell "broken" from "denied a capability
it needs"); it surfaces at the approved host run instead. The false-err on
every network tool is the larger harm, and the warn still shows the failure.

Test plan: cargo test -p open-max-tui (run_examples_is_gated_and_reported_
through_json updated to the new contract and shown red on the old exit-1
behavior; new an_unapproved_sandbox_non_pass_warns_and_does_not_fail_the_check);
cargo clippy --workspace at zero warnings.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant