Skip to content

fix(mxc): a finished sandbox command is reported as a timeout - #1659

Open
SebTardif wants to merge 3 commits into
openclaw:mainfrom
SebTardif:fix/f107-grace-drain-timeout
Open

SebTardif wants to merge 3 commits into
openclaw:mainfrom
SebTardif:fix/f107-grace-drain-timeout

Conversation

@SebTardif

@SebTardif SebTardif commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What Problem This Solves

Fixes: a sandboxed command that finishes while its output is draining is reported as a timeout.

User Impact

User impact: a completed command keeps its exit code. A command the executor marks as timed out is still reported as a timeout.

Why This Change Was Made

The host grace timer is the command timeout plus five seconds. After wxc-exec exits, the executor drains output and returns TimedOut false. The caller also treated a cancelled grace token as a timeout, so the exit code became -1.

Evidence

wxc-exec.exe is the @microsoft/mxc-sdk binary the tray build copies to tools\mxc\x64\. MxcAvailability.Probe on this host reported tier=base-container and can_system_run=True. DirectAppContainerExecutor.ExecuteAsync on head e4a7d6e19b56621f6067eec058ccee36e7d90d03 then ran three commands:

finished_exit=0 timed_out=False duration_ms=118 marker=True tag=mxc
slept_exit=0 timed_out=False duration_ms=2641 marker=True tag=mxc
over_timeout_exit=-1 timed_out=False duration_ms=2139 marker=False tag=mxc

finished is cmd /c echo mxc-finished with a 30s limit. slept is PowerShell Start-Sleep -Seconds 2 with a 30s limit, and the marker was printed. over_timeout is the same sleep for 8 seconds with a 2s limit. The process was stopped at about 2.1s, the marker was not printed, and the host grace (7s) had not fired, so TimedOut stayed false. The earlier helper check still holds: a finished result with a cancelled grace token reports reported_timeout=False, and an executor result with TimedOut true still reports a timeout.

.\scripts\validate-mxc-e2e.ps1 -NoBuild on this head exited 0. The throwaway distro listened on 127.0.0.1:27087 and was unregistered by that run. Ubuntu-24.04 stayed registered. 18 tests passed. The three pool proofs passed. Gateway node.invoke system.run of Write-Output returned exit 0, timedOut false, containment mxc, duration 510 ms, and the marker. The denied write returned exit 1, timedOut false, containment mxc, duration 302 ms, and the target file was absent.

Change Type

  • Bug fix
  • Feature
  • Refactor
  • Docs or instructions
  • Tests or validation
  • Security hardening
  • Chore or infrastructure

Scope

  • Tray or WinUI UX
  • Windows node capability
  • Local MCP or winnode
  • Gateway, connection, or pairing
  • Setup or onboarding
  • Permissions, privacy, or security
  • Tests, CI, or docs

Required proof pools

  • windows-wsl-mxc: .\scripts\validate-mxc-e2e.ps1 -NoBuild exited 0 on this head. MirroredWslSafeGatewayPort_IsListeningAndRecorded, RealGateway_SystemRun_ExecutesThroughWindowsNodeMxcSandbox, and RealGateway_SystemRun_BlocksWritesToTrayDataDirectoryInMxcSandbox passed. The successful system.run was exit 0, timedOut false, containment mxc.

Validation

  • ./build.ps1 exit 0 (Shared, Cli, WinNodeCli, SetupEngine, WinUI).
  • dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore: 4263 passed, 33 skipped, 0 failed, 4296 total. The skipped MXC integration tests are gated on OPENCLAW_RUN_INTEGRATION. The ExecuteAsync trace above used the same wxc-exec.exe.
  • dotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj: 3990 passed, 0 skipped, 0 failed.

Real behavior proof

  • Behavior or issue addressed: A finished sandbox command stays not-timed-out. A command stopped by the sandbox limit is not relabeled by the host grace token.
  • Real environment tested: Windows, .NET SDK 10.0.401, wxc-exec.exe from @microsoft/mxc-sdk, probe tier base-container. Head e4a7d6e19b56621f6067eec058ccee36e7d90d03.
  • Exact steps or command run after this patch: MxcAvailability.Probe, then DirectAppContainerExecutor.ExecuteAsync for echo mxc-finished, a 2 second PowerShell sleep, and an 8 second sleep with a 2 second limit. Then .\scripts\validate-mxc-e2e.ps1 -NoBuild.
  • Evidence after fix:
finished_exit=0 timed_out=False duration_ms=118 marker=True tag=mxc
slept_exit=0 timed_out=False duration_ms=2641 marker=True tag=mxc
over_timeout_exit=-1 timed_out=False duration_ms=2139 marker=False tag=mxc
gateway_system_run exit=0 timedOut=false durationMs=510 containment=mxc
gateway_denied_write exit=1 timedOut=false durationMs=302 containment=mxc file_absent=true
  • Observed result after fix: The echo and the 2 second sleep returned exit 0, TimedOut false, and the marker. The 8 second sleep was stopped at about 2.1 seconds, the marker was absent, and TimedOut stayed false because the 7 second host grace had not fired. The gateway system.run returned the marker with exit 0 and timedOut false. The denied write did not create the file.
  • What was not tested: The grace token was not forced to fire during the post-exit drain. wxc-exec stops the process at TimeoutMs, and the host grace is that limit plus five seconds. The post-exit drain cap is 500 ms, so a normal wxc-exec exit does not overlap the grace. The 8 second sleep returned in 2139 ms, before the 7 second grace.
  • Environment tested: Windows, .NET SDK 10.0.401.
  • PR head or commit tested: e4a7d6e19b56621f6067eec058ccee36e7d90d03
  • Exact steps or command run: ExecuteAsync with wxc-exec, then .\scripts\validate-mxc-e2e.ps1 -NoBuild.
  • Evidence after fix: echo and the 2 second sleep are timed_out=False with the marker. The 8 second sleep is exit -1, marker absent, timed_out=False.
  • Observed result: finished sandbox commands stay not-timed-out through ExecuteAsync.
  • Screenshot or artifact links verified? No
  • Not verified or blocked: the grace token firing during the post-exit drain. wxc-exec returns about five seconds before that deadline, and the drain cap is 500 ms.

Security Impact

  • New permissions or capabilities? No
  • Secrets or tokens handling changed? No
  • New or changed network calls? No
  • Command or tool execution surface changed? No
  • Data access scope changed? No
  • If any answer is Yes, explain the risk and mitigation:

Compatibility and Migration

  • Backward compatible? Yes
  • Config or environment changes? No
  • Migration needed? No
  • If yes, list the exact upgrade steps: none

Review Conversations

  • I replied to or resolved every bot review conversation addressed by this PR.
  • I left unresolved only conversations that still need maintainer judgment.

Host token on this head

wxc-exec still returns on its own script deadline, about 0.2 seconds later, so a normal sandbox command never reaches the host grace timer at limit plus 5 seconds. A start /b child does not bridge that gap either.

The production MxcExecutor was run with the sandbox deadline set to 60 seconds and the host token cancelled at 2 seconds, while a cmd loop was still running. Result: exitCode=-1; timedOut=True; durationMs=2414; error=Execution was cancelled.

Post-exit drain inside ExecuteAsync is still not shown. That drain is capped at 500 milliseconds, and wxc-exec exits about 5 seconds before the grace timer.

The grace timer can trip while stdout is drained after wxc-exec has
exited. The executor already returns TimedOut false for that result.
The caller was also treating the cancelled grace token as a timeout
and replacing the exit code with -1.

Signed-off-by: Sebastien Tardif <SebTardif@ncf.ca>
@clawsweeper

clawsweeper Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Oct 7, 2026
@clawsweeper

clawsweeper Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed October 7, 2026, 2:11 PM ET / 18:11 UTC (Revision 4).

ClawSweeper review

What this changes

The PR preserves sandbox command exit codes when the host grace token expires after completion and adds timeout-classification and real-executor cancellation tests.

Merge readiness

⛔ Blocked before merge - 2 items remain

This PR remains useful: current main and the latest release retain the old classification. The new host-cancellation trace addresses part of the previous review, but real proof of the changed post-exit branch remains outstanding.

Priority: P2
Reviewed head: b51954768a04f48df04d9edd32a692e4c2bee78f

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The focused patch appears correct, with useful real execution evidence, but its central timing scenario remains unproven.
Proof confidence 🦐 gold shrimp (3/6) Needs stronger real behavior proof before merge: Real Windows wxc-exec and Gateway traces provide positive completion and containment evidence, and the new MxcExecutor trace proves still-running host cancellation. DirectAppContainerExecutor's changed post-exit grace-expiry branch remains explicitly unexercised. Production code is unchanged from the earlier proven head; no stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: Real Windows wxc-exec and Gateway traces provide positive completion and containment evidence, and the new MxcExecutor trace proves still-running host cancellation. DirectAppContainerExecutor's changed post-exit grace-expiry branch remains explicitly unexercised. Production code is unchanged from the earlier proven head; no stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 7 items Verified introduced change: The pinned merge-base-to-head diff changes only timeout classification and two test files. Caller-cancellation precedence remains intact; production growth is 10 added and 3 removed lines.
Production completion contract: MxcExecutor marks cancellation while waiting for process exit as TimedOut=true. After observed process exit, it drains output with bounded cleanup and returns the process exit code with TimedOut=false. This supports trusting the executor result rather than the later token state.
Still necessary on main and release: Fetched main still computes result.TimedOut || linked.IsCancellationRequested. The same expression is present in latest-release commit 94006b3 for v2026.9.8; no merged fixing PR was established.
Findings None None.
Security None None.

How this fits together

Windows node sandbox execution launches commands through Microsoft's MXC launcher. Its completion, captured output, and timeout status become the system.run result returned to the agent.

flowchart TD
 A[Agent command] --> B[Windows node sandbox runner]
 B --> C[MXC launcher]
 C --> D[Process completion or cancellation]
 D --> E[Bounded output drain]
 E --> F[Timeout classification]
 F --> G[Agent command result]
Loading

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: Real Windows wxc-exec and Gateway traces provide positive completion and containment evidence, and the new MxcExecutor trace proves still-running host cancellation. DirectAppContainerExecutor's changed post-exit grace-expiry branch remains explicitly unexercised. Production code is unchanged from the earlier proven head; no stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • Complete next step (P2) - Provide real production-path evidence of grace expiry during post-exit draining with the completed exit code preserved. Redacted terminal output or logs count; screenshots or recordings are welcome when they show the behavior. Update the PR body to trigger re-review, or ask a maintainer to comment @clawsweeper re-review.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production and test LOC production +10/-3, tests +81/-0 The small production change has a stated timeout-classification purpose and predominantly test additions.

Technical review

Best possible solution:

Use process completion as the timeout authority while preserving caller cancellation, backed by direct evidence of the post-exit grace-expiry case.

Do we have a high-confidence way to reproduce the issue?

No high-confidence runtime reproduction is established. Source shows the misclassification if grace expires after process completion, but the supplied real runs explicitly did not reach that timing condition.

Is this the best way to solve the issue?

Yes, trusting MxcExecutor's completion result is a narrow repair consistent with its existing contract; the added helper test alone does not prove the production timing path.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 5e3fb40bcc89.

Labels

Label changes:

No label changes.

Label justifications:

  • P2: This is a bounded sandbox result-classification repair without evidence of an urgent widespread regression.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Real Windows wxc-exec and Gateway traces provide positive completion and containment evidence, and the new MxcExecutor trace proves still-running host cancellation. DirectAppContainerExecutor's changed post-exit grace-expiry branch remains explicitly unexercised. Production code is unchanged from the earlier proven head; no stored-data contract changes. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

What I checked:

  • Verified introduced change: The pinned merge-base-to-head diff changes only timeout classification and two test files. Caller-cancellation precedence remains intact; production growth is 10 added and 3 removed lines. (src/OpenClaw.Shared/Mxc/DirectAppContainerExecutor.cs:160, b51954768a04)
  • Production completion contract: MxcExecutor marks cancellation while waiting for process exit as TimedOut=true. After observed process exit, it drains output with bounded cleanup and returns the process exit code with TimedOut=false. This supports trusting the executor result rather than the later token state. (src/OpenClaw.Shared/Mxc/MxcExecutor.cs:208, b51954768a04)
  • Still necessary on main and release: Fetched main still computes result.TimedOut || linked.IsCancellationRequested. The same expression is present in latest-release commit 94006b3 for v2026.9.8; no merged fixing PR was established. (src/OpenClaw.Shared/Mxc/DirectAppContainerExecutor.cs:159, 94006b3672a0)
  • Positive real proof and remaining gap: The captured PR body reports real Windows wxc-exec completion traces and a successful non-skipping Gateway MXC pool on e4a7d6e. Its added host-token section reports the production MxcExecutor returning exitCode=-1, timedOut=True, durationMs=2414 after cancellation at two seconds. It explicitly excludes cancellation during post-exit draining. Captured context sourceRevision: aeb9b6944c8bdb2d829927c31fc33a30dd2d0bfcc5dfb8a832353e3a077daa19.
  • Re-review continuity: The production file is unchanged from the earlier reviewed head. The new integration test supplies the previously requested still-running host-cancellation control, but does not exercise DirectAppContainerExecutor's completed-result classification during grace expiry. (tests/OpenClaw.Shared.Tests/Mxc/MxcCommandRunnerIntegrationTests.cs:310, b51954768a04)
  • Related merged cleanup and routing history: GitHub verifies fix(mxc): bound termination and output drain cleanup #1387 (fix(mxc): bound termination and output drain cleanup) as merged at 0e34e20. Main history also records Scott Hanselman, Barbara Kudiess, and Vitor Cepeda Lopes working on the execution path. Blame and deeper follow-history inspection encountered unavailable historical objects, so exact source-line introduction remains unverified. (src/OpenClaw.Shared/Mxc/MxcExecutor.cs, 0e34e201cb05)

Likely related people:

  • shanselman: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Barbara Kudiess: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add redacted production-path output showing grace expiry during post-exit draining preserves the completed command's exit code.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (3 earlier review cycles)
  • reviewed 2026-10-07T00:20:39.057Z sha e4a7d6e :: needs real behavior proof before merge. :: none
  • reviewed 2026-10-07T00:37:19.707Z sha e4a7d6e :: needs real behavior proof before merge. :: none
  • reviewed 2026-10-07T01:13:04.134Z sha e4a7d6e :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Oct 7, 2026
wxc-exec returns on its own script deadline, several seconds before
the host grace timer. A sandbox command cannot hold it open that long.

Run the production executor with a 60 second sandbox deadline and
cancel the host token at 2 seconds. The long cmd loop is still
running, so the host reports TimedOut and "Execution was cancelled."
On this machine that returned in 2414 ms.

The post-exit drain overlap inside ExecuteAsync is still not covered.
That path exits about five seconds before the grace timer.

Signed-off-by: Sebastien Tardif <SebTardif@ncf.ca>
The UI job failed one onboarding case, Existing route choice 0,
on Assert.True(resultBar.IsOpen) after a cancelled check. The Remote
case in the same theory passed. The previous head's Build and Test
run was green, and this commit does not change that page.

GitHub refused gh run rerun without admin rights, so this empty
commit starts the lane again.

Signed-off-by: Sebastien Tardif <SebTardif@ncf.ca>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant