fix(core): a sandboxed probe non-pass is inconclusive, not a broken tool - #231
Open
Max17190 wants to merge 1 commit into
Open
fix(core): a sandboxed probe non-pass is inconclusive, not a broken tool#231Max17190 wants to merge 1 commit into
Max17190 wants to merge 1 commit into
Conversation
openmax --check --run-examples probes an unapproved tool in a sandbox that denies the network and any write outside its scratch dir (ADR-0011). A tool that needs either - a web or API tool, a tool that writes a real file - can never pass that probe, yet the failure was reported as `err` and exited the check nonzero, identical to a genuinely broken tool. That fails CI on the largest tool family and steers an agent to "fix" working code. A passing probe approves nothing, so a failing one condemns nothing: treat a sandboxed non-pass as inconclusive. It now reports `warn`, names why (the sandbox denies network and non-scratch writes), and does NOT fail the check. The honest verdict for such a tool is its approved host run, which still errs on real breakage - the approved path is unchanged. --spec tools states the limitation and points at the approve-then-rerun proof. Tradeoff, stated plainly: a genuinely broken UNAPPROVED tool no longer errs at probe time either (the sandbox cannot tell "broken" from "denied a capability it needs"); it surfaces at the approved host run instead. The false-err on every network tool is the larger harm, and the warn still shows the failure. Test plan: cargo test -p open-max-tui (run_examples_is_gated_and_reported_ through_json updated to the new contract and shown red on the old exit-1 behavior; new an_unapproved_sandbox_non_pass_warns_and_does_not_fail_the_check); cargo clippy --workspace at zero warnings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
openmax --check --run-examplesprobes an unapproved tool in a sandbox that denies the network and any write outside its scratch dir (ADR-0011). A tool that legitimately needs either, a web or API tool or a tool that writes a real file, can never pass that probe, yet the failure was reported aserrand exited the check nonzero, identical to a genuinely broken tool. That fails CI on the largest tool family and steers an agent to "fix" working code. Dogfooding a keyless web-search tool reproduced it:err example ... Could not resolve hostand exit 1 on a correct tool; in another run the sandbox's write confinement even led the agent to delete a valid[example].Summary
A passing probe approves nothing (ADR-0011), so a failing one condemns nothing: a sandboxed non-pass is inconclusive.
warn, names why (the sandbox denies network and non-scratch writes), and does NOT fail the check (both the terminal and the--jsonfaces).errs on real breakage, which is the honest verdict for such a tool.--spec toolsstates the limitation and points at approve-then-rerun for the real proof of life.Tradeoff, stated plainly: a genuinely broken unapproved tool no longer
errs at probe time either, because the sandbox cannot tell "broken" from "denied a capability it needs"; it surfaces at the approved host run instead. The false-erron every network tool is the larger harm, and thewarnstill shows the failure and its reason.Test Plan
cargo test -p open-max-tui:run_examples_is_gated_and_reported_through_jsonupdated to the new contract and shown red on the old exit-1 behavior; newan_unapproved_sandbox_non_pass_warns_and_does_not_fail_the_check.cargo test --workspace,cargo clippy --workspace --all-targetsat zero warnings.warn example ... could not be proven in the sandboxand--check --run-examplesexits 0; after--approve, the real run behaves as before.Greptile Summary
This change treats unsuccessful unapproved sandbox probes as advisory warnings, while keeping approved examples that run with host authority as failing checks.
A real-binary check exercised a passing example and an example that exits nonzero through both terminal and JSON output. Unapproved sandbox runs reported warnings and exited successfully; after approval, the same failing command reported an error and exited nonzero. Focused CLI integration tests covering both paths also passed.
Confidence Score: 5/5
Safe to merge based on the exercised example-run contracts.
The exercised terminal and JSON flows preserve the intended distinction between an inconclusive sandbox probe and a failing approved host-authority run, with focused integration coverage passing.
Files Needing Attention: No additional files need attention.
What T-Rex did
Reviews (1): Last reviewed commit: "fix(core): a sandboxed probe non-pass is..." | Re-trigger Greptile