Skip to content

add sandbox mode option - #8

Merged
andreidavid merged 6 commits into
mainfrom
sorelmitra/add-sandbox-mode-option
Jul 28, 2026
Merged

add sandbox mode option#8
andreidavid merged 6 commits into
mainfrom
sorelmitra/add-sandbox-mode-option

Conversation

@sorelmitra

Copy link
Copy Markdown
Collaborator

Codex reviews can fail when its sandbox is inside an already sandboxed Claude session.

Add sandbox configuration, with workspace-write as the default.
Add command to configure and show the setting.

sorelmitra and others added 6 commits July 24, 2026 18:21
Co-authored-by: OpenAI Codex <noreply@openai.com>
note: this currently fails, apparently due to a bug in codex where the
sandbox permissions given to main agent are not passed to sub-agents

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sorelmitra

Copy link
Copy Markdown
Collaborator Author

Proof of how this works: sandbox-mode-real-review-proof.md

@andreidavid andreidavid left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly what the #3 review asked for, and more carefully than asked: the array-based mode args, the fail-safe default, one command with an argument hint, the script in scripts/ with CI and CONTRIBUTING updated to match, the SessionStart warning, and a CHANGELOG entry politely left unversioned. Two things deserve specific credit: the flag-position discovery (--sandbox really is rejected after the review subcommand — I verified against codex 0.144.5) and the state-file mechanism with its install-channel rationale, which is better-reasoned than the pluginConfigs approach it replaces. The tests isolating CLAUDE_CONFIG_DIR from the developer's real state is the kind of detail most contributors miss.

Merging as-is. The INCONCLUSIVE detection has a few integration gaps on our side of the fence (the Stop hook and GC don't know the new verdict, and the prose heuristic can misfire on reviews that merely discuss sandbox failures — this repo's reviews do constantly); rather than bounce it back, I'll fold those fixes into a follow-up PR right behind this one and link it here. Thanks for another thorough round.

@andreidavid
andreidavid merged commit 8d6bef8 into main Jul 28, 2026
2 checks passed
@andreidavid

Copy link
Copy Markdown
Owner

One favor that would sharpen the follow-up: the fix replaces the prose regex with a structural signal (a review whose JSONL stream contains zero command_execution items never actually inspected anything). That's sound if your failing reviews genuinely run no commands — but if codex in your environment runs commands that all fail before it gives up, the detector would need a secondary signal. Your history file already holds the ground truth: the phantom PASSes were recorded. Could you paste one (redact anything sensitive)?

jq 'select(.verdict=="PASS") | {timestamp, short_sha, review_prose}' .git/codex-reviews.jsonl | tail -20

— specifically an entry from a run where you know the sandbox failed. And once the follow-up ships in a release, a quick "INCONCLUSIVE now fires correctly here" (or doesn't!) from your setup would close the loop.

@sorelmitra
sorelmitra deleted the sorelmitra/add-sandbox-mode-option branch July 28, 2026 16:53
@andreidavid

Copy link
Copy Markdown
Owner

Follow-up landed as #9 — the detector ended up structural (no successful execution activity in the JSONL stream, grounded in codex's exec serializer source) rather than prose-based, plus the Stop-hook/GC integration and remote-aware retry guidance. Your redacted phantom-PASS sample would still be valuable to confirm the detector catches your environment's exact failure shape — and once 1.8.0 is released, a quick yes/no on whether INCONCLUSIVE fires correctly on your setup closes the loop. Thanks again!

andreidavid added a commit that referenced this pull request Jul 28, 2026
#9)

Follow-up to #8's phantom-pass detection, fixing the integration gaps
found in review:

The detector is now structural instead of prose matching, and
deliberately minimal: INCONCLUSIVE fires only when the review's event
stream shows no successful execution activity at all -- no
command_execution item that exited 0 and no code-mode
dynamic_tool_call item. That is the one empirically observed failure
shape (a nested sandbox that refuses to initialize, so the review runs
nothing) and the only fact the stream states reliably. Verifying that
activity actually READ the repository is explicitly not attempted:
shell output under-determines it (compounds, pipelines, || fallbacks,
quoting, error-like content), and a far stricter seven-layer version
of this check minted a new counterexample per layer before being
retired. Partial sandboxes that permit some execution while denying
repo reads are a documented accepted residual. The prose regex the
detector replaces matched reviews that merely DISCUSS sandbox failures
-- which every review of this repo does. Legacy text-parser mode is
exempt (no event stream).

The Stop hook clears INCONCLUSIVE state like ERROR/TIMEOUT, and the GC
recognizes it as a known verdict and sweeps aged entries -- previously
it fell into the unknown-verdict bucket, which is preserved
indefinitely as corruption evidence, so abandoned INCONCLUSIVE state
was immortal.

The guidance message is remote-aware like the fix loop: an unpushed
commit pairs /codex-review-sandbox-mode danger-full-access with a
repo-anchored git commit --amend --no-edit (new sha, fresh review
under the new mode; a same-sha re-run is dedup-skipped); a pushed
commit is never amended and gets a one-off elevated review command
carrying the configured CODEX_BIN and shell-quoted repo path.

Stub modes emit realistic command_execution events; no_inspection,
failed_inspection, unrelated_command_pass (documented residual), and
dynamic_tool_pass cover the detector from all directions. README
documents the verdict. Version 1.8.0 covering #8 plus this hardening.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@sorelmitra

Copy link
Copy Markdown
Collaborator Author

One favor that would sharpen the follow-up:
[...]

Happy to. And the follow-up: I updated to 1.8.1 and ran the test — INCONCLUSIVE does not fire on my setup. Details below.

Question 1 — phantom-PASS samples. From a work repo, short_sha redacted, prose verbatim:

{"timestamp":"2026-07-22T12:37:51Z","short_sha":"<redacted>","review_prose":"The local filesystem sandbox failed, and the commit is not available through the GitHub API. The changes could not be inspected, so this verdict has very low confidence."}
{"timestamp":"2026-07-27T08:59:35Z","short_sha":"<redacted>","review_prose":"The commit could not be inspected: local commands failed because the sandbox could not start, and the commit is not available in the connected GitHub repository. No reliable correctness verdict or findings can be provided."}
{"timestamp":"2026-07-27T10:20:21Z","short_sha":"<redacted>","review_prose":"The commit could not be inspected because local shell access failed and the commit is not available on the connected GitHub repository. No reliable findings could be produced."}

5 of 8 PASSes in that history look like this — codex self-declares "sandbox failed" or "could not be inspected" in the prose, then the verdict is recorded PASS.

Question 2 — INCONCLUSIVE on my setup. I set sandbox_mode=workspace-write, made a throwaway commit in a fresh git repo on my Mac (macOS Darwin 25.5.0, codex-cli 0.146.0), and ran the hook. Verdict: still PASS. Trace shows the detector was reached (JSONL_MODE=true, HAS_ISSUES=false, REVIEW_EXIT=0) but EXEC_ACTIVITY=3. Raw JSONL, condensed:

{"type":"item.completed","item":{"type":"mcp_tool_call","tool":"list_mcp_resources","status":"completed"}}
{"type":"item.completed","item":{"type":"mcp_tool_call","tool":"list_mcp_resource_templates","status":"completed"}}
{"type":"item.completed","item":{"type":"mcp_tool_call","tool":"github.search_commits","status":"completed"}}
{"type":"item.completed","item":{"type":"agent_message","text":"The review could not be completed because the sandbox rejected all filesystem and Git commands with `sandbox_apply: Operation not permitted`. The commit diff was unavailable, so this verdict is not an assessment of the code."}}

Zero command_execution items; three MCP tool calls, none of which read the repo — two are codex introspecting its own MCP setup at startup, one is a GitHub lookup that completes on any answer (the commit was local-only, so it returned nothing). The agent's own final message spells out that no code was inspected. The detector counts any completed mcp_tool_call as activity, so the threshold is met and the review lands as PASS.

Not sure if you'd consider that the "partial sandbox / commands whose output the reviewer ignores" residual the docstring accepts by design, or something worth catching. If it's worth catching, filtering out codex's own introspection tools (list_mcp_resources, list_mcp_resource_templates) from the activity count would probably do it — those two fire on every run regardless of what the sandbox lets through, and the third case (a github.* tool that completes empty) is presumably rarer.

Happy to run any other probe or paste more of the stream if it helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants