fix(core): let the service decide a resume past the suspended window (#597)

[laithalsaadoon]

What and why

Sandbox::resume refused with ERR_WINDOW_CLOSED once suspendedDurationSeconds had passed since the sandbox's own suspend(). It refused before any call to the service, in require_open_suspended_window (crates/microvms-app/src/sandbox.rs, STATE-12). The service didn't apply that window to an explicit suspend. On 2026-10-12 in us-east-1, two VMs with maxIdleDurationSeconds 60 and suspendedDurationSeconds 60, suspended with SuspendMicrovm, were still SUSPENDED 906 and 917 seconds later, and ResumeMicrovm brought both to RUNNING; a third, with a 3600-second idle limit, resumed after 1283 seconds. So the client refused resumes that the service would have honored. AWS's IdlePolicy reference describes the window for any suspended VM, so the explicit case is measured behavior AWS doesn't promise, and the client now depends on neither answer: the service decides. The window does end a VM that the idle policy suspended. That VM was TERMINATED about 60 seconds after it suspended, and ResumeMicrovm then returned a ValidationException saying the VM "has been terminated and its state cannot be changed."

Before this change, a resume past the window of the sandbox's own suspend failed with ERR_WINDOW_CLOSED before any call, even though the service would have resumed the VM. After it, every resume goes to the service, and ERR_WINDOW_CLOSED reports a resume the service refused for a VM that had already ended, whatever ended it.

• Sandbox::resume no longer records when it suspended the VM, and it doesn't check a window. The suspended_at field and require_open_suspended_window are removed.
• ControlPlane::resume (crates/microvms-app/src/control/microvm.rs) reads the VM with GetMicrovm when the service refuses a resume with a 400. If the VM is TERMINATING or TERMINATED, the error becomes ERR_WINDOW_CLOSED, its message names the state and stateReason, and the service's refusal is its source. The message names no cause and says to launch a new VM, because a VM a terminate ended gets the same refusal as one the service ended. For any other state, or when the read fails, the caller gets the service's own error. The check reads the VM's state and doesn't match the message text.
• The CLI resume command and both bindings' ControlPlane.resume call ControlPlane::resume, so they get the same mapping. microvm resume on an ended VM now exits 8 (ERR_WINDOW_CLOSED). Before this change it exited 9 (ERR_PLATFORM). The changelog fragment says so.
• After a WindowClosed refusal, the sandbox records the VM as TERMINATED and as terminated, so a second resume is refused before any call (STATE-11). A refusal for a VM that's still alive leaves the sandbox SUSPENDED, and the next resume goes to the service again.
• A Sandbox doesn't see a suspend the idle policy makes, so it still holds that VM as RUNNING and its resume is refused before any call (STATE-7), as on main. The docs say so, and say the suspend reaches a caller only through Sandbox::adopt or ControlPlane::resume. The Drop warning names the state as the one the sandbox last saw, and says only that a VM that's still live bills.
• STATE-12 in verify/spec/core.symspec.json now says the client issues the resume and reports the service's refusal for an ended VM as a window-closed error. The client model in crates/model/src/client.rs drops its clock. It gains a SuspendedVmEnded action, which is the platform ending a suspended VM, and a resume of that VM reaches the service and is refused. The model-conformance test replays the new paths against Sandbox.
• Every surface that stated the old rule now states the new one:
  • the --suspended-sec help on both commands, the resume command doc, and the ERR_WINDOW_CLOSED exit meaning and hint, which now says the VM ended before the resume (the suspended window, the maximum duration, or a terminate) and names no flag;
  • crates/microvms-cli/README.md and the WindowClosed doc in microvms-domain;
  • the WindowClosedError, ControlPlane.resume, and Sandbox docs in both bindings, with microvms.pyi and index.d.ts regenerated;
  • the module docs of crates/microvms-app/src/clock.rs, which described the window timing this change removes;
  • docs/manifest.json, docs/reference/cli.md, docs/insights/debugging-guide.md, and docs/insights/contract-map.md;
  • three pages under site/authored/learn/operations/ and the long-lived VM tutorial, which also say that ERR_WINDOW_CLOSED for an idle-suspended VM comes from microvm resume and ControlPlane.resume, and that a Sandbox refuses its own resume first (A Sandbox never sees a platform idle suspend, so resume refuses from RUNNING #603);
  • docs/behavior/processes.md, whose suspend and resume steps described the stamp and window check this change removes.

Verified offline only. No live run exercises this change.

Size: 841 changed lines of product code. 515 of them are unit tests, almost all in sandbox.rs and control/microvm.rs. They replace the STATE-12 tests that held the old rule and add the refusal tests. The other 326 lines are the fix and the doc comments that stated the old rule on each surface. If the docs were split from the fix, one pull request would ship docs that contradict the code.

Fixes #597

Evidence

• feat(example): Codex SDK in a Strands Box on a MicroVM, on bedrock-runtime #594 and docs: one current answer per platform fact, and a prose rule that holds it #598 merged as 944577f and f410ec2, and this branch is rebased onto main f410ec2 and targets main. On 4384885 (head), 2026-10-12:
  • mise run check exits 0: 1,804 Rust tests passed, 0 failed, 28 ignored; prose 0 errors, 4,084 warnings, under every ceiling; guards:list 1,333 faults.
  • mise run docs:check exits 0: 223 site tests, CSpell 0 issues.
  • 54af2d9 makes the help, CLI reference, changelog and site text say the explicit-suspend result is measured and not promised, and fixes processes.md. Because each FAIL_TO_PASS patch reverses every product hunk, 4384885 regenerates the six patches with fail-to-pass.py's fix_patch against f410ec2. guards:fire then fired 7 of 7 (the six entries and resume-window-closed-drops-the-refusal).
• The review fixes are 19bd74a (the ERR_WINDOW_CLOSED message and meaning), ecd4d60 (what a Sandbox sees of an idle suspend, and the Drop warning) and 0499612 (clock.rs's docs). On 0499612:
  • mise run check exits 0 in 301 s on 2026-10-12. cargo test --all reports 1804 passed, 0 failed, and 28 ignored. prose:check reports 0 errors and 4,080 warnings, under every ceiling, and guards:list reports 1333 faults registered.
  • mise run ci:mutants -- --base 3330dc6 -- --in-place tested the 12 mutants in the fixes' lines: 9 caught, 3 unviable, 0 missed. Over the whole diff, with --base origin/docs/one-current-fact, it tested 16: 12 caught, 4 unviable, 0 missed.
  • mise run guards:fire fires the new fault and the six FAIL_TO_PASS entries, 7 of 7; Guards has the output.
  • microvms.pyi, index.d.ts and docs/manifest.json are regenerated with mise run stubs, mise run dts and mise run manifest, and check runs stubs:check, dts:check and manifest:check.
  • bindings:py, bindings:js and docs:check weren't run again after the fixes. The fixes change the bindings' doc comments and the WindowClosedError docstring, and no binding code.

Before the review fixes:

• Mutation testing: the first CI run on fix(core): let the service decide a resume past the suspended window (#597) #601 found two survivors in classify's suggestion match (crates/microvms-cli/src/exit.rs, deleting the (Exit::LaunchDied, _) or (Exit::WindowClosed, _) arm). 3330dc6 adds a_died_launch_and_a_closed_window_each_carry_their_remedy. On 3330dc6, run locally against origin/docs/one-current-fact, mise run ci:mutants -- --in-place tested the diff's 13 mutants: 9 caught, 4 unviable, 0 missed. mise run check on 3330dc6: exit 0, 1,803 Rust tests, 0 failed.

• Before feat(example): Codex SDK in a Strands Box on a MicroVM, on bedrock-runtime #594 and docs: one current answer per platform fact, and a prose rule that holds it #598 merged, this was stacked on docs: one current answer per platform fact, and a prose rule that holds it #598, which was stacked on feat(example): Codex SDK in a Strands Box on a MicroVM, on bedrock-runtime #594, all three rebased on main 78e5f1d. docs: one current answer per platform fact, and a prose rule that holds it #598 rewrites docs/PLATFORM.md and docs/SUSPEND-RESUME.md to the 2026-10-12 suspended-window fact and edits the same three operations pages. This branch keeps its client wording in those pages, and it rewrites the two sentences of site/authored/learn/tutorial/long-lived-vm.md that described the old up-front refusal. On the stacked tree at c3f7bc1, mise run check exits 0 (1,802 Rust tests, 0 failed; prose 0 errors, 4,080 warnings, with docs: one current answer per platform fact, and a prose rule that holds it #598's correction-trail rule active), and mise run docs:check exits 0 (223 site tests, CSpell 0 issues).

• c3f7bc1 is feat(agentd): an image-set default user for execs that omit user (#595) #600's ccdb6a4: ci:mutants builds the parity tunnel fixture first, because this change has mutants in microvms-cli, whose parity test spawns that fixture. The identical commit on both branches merges cleanly.

• mise run check exits 0 in 326 s on 2026-10-12. That run was before the review fixes, and before a one-line citation fix in docs/insights/debugging-guide.md. After the fix, prose:check, trace:check, fail-to-pass:check, changelog:check, and parity:check exit 0 again.

  • cargo test --all reports 1802 passed, 0 failed, and 28 ignored.
  • cargo fmt --all -- --check and cargo clippy --all-targets -- -D warnings pass in lint.
  • prose:check reports 0 errors and 4113 warnings, under every ceiling.
  • guards:list reports 1331 faults registered.

• mise run bindings:py reports 512 passed and 1 skipped.

• mise run bindings:js reports 382 tests, with 381 passing, 1 skipped, and 0 failing.

• mise run docs:check exits 0, and CSpell finds 0 issues in 22 files.

• mise run spec exits 0. mise run spec:core exits 1 on one error, the word "never" in TRAP-4. The same command on main's spec file at 78e5f1d fails with the same error, and STATE-12's new sentence raises no error.

• The model-conformance replay drives 940 of 1061 walked paths against Sandbox, with 7713 sandbox calls. It skips 121 paths where a completion arrives after another step, as before.

Parity

• No public name changed on any surface (core, CLI commands, microvms.pyi, index.d.ts).
• mise run parity:check passes. The resume and resume-by-id rows in verify/parity/capabilities.toml don't change, because the behavior lives in core's ControlPlane::resume, which every surface calls.

No parity case covers the refusal, because the Python and TypeScript parity runners can't script a control plane.

Guards

mise run fail-to-pass -- --emit resume-window ran against the merge base 78e5f1d. It proves six tests, which fail with the fix taken out and pass with it in. It wrote their entries into verify/guards/faults/resume-window.toml with their patches. Each patch reverses every product hunk of the branch, so the review fixes regenerate all six with the script's own fix_patch against 78e5f1d, and all six fire again on 0499612's tree.

fired: f2p-a-resume-of-a-vm-the-service-ended-exits-window-closed (26.5 s)
fired: f2p-every-walked-path-of-the-client-model-replays-against (9.8 s)
fired: f2p-a-resume-refused-for-a-vm-that-terminated-while (12.5 s)
fired: f2p-an-adopted-vm-resumes-past-the-reported-window-through (7.3 s)
fired: f2p-a-resume-past-the-window-of-an-explicit-suspend-reaches (5.6 s)
fired: f2p-a-resume-the-service-refuses-for-an-ended-vm-is-window (6.0 s)

The fire on 0499612, after the review fixes, with the new hand-written fault:

guards: tree 0499612 plus 0 uncommitted paths
fired: resume-window-closed-drops-the-refusal (10.5 s)
fired: f2p-a-resume-of-a-vm-the-service-ended-exits-window-closed (18.2 s)
fired: f2p-every-walked-path-of-the-client-model-replays-against (5.8 s)
fired: f2p-a-resume-refused-for-a-vm-that-terminated-while (8.8 s)
fired: f2p-an-adopted-vm-resumes-past-the-reported-window-through (4.0 s)
fired: f2p-a-resume-past-the-window-of-an-explicit-suspend-reaches (3.8 s)
fired: f2p-a-resume-the-service-refuses-for-an-ended-vm-is-window (3.8 s)
guards: 7 of 7 fired (78.8 s of faults)

• sandbox::tests::a_resume_past_the_window_of_an_explicit_suspend_reaches_the_service_and_resumes is the regression test for core: resume refuses an explicitly suspended VM past suspendedDurationSeconds, which the service still resumes #597. It suspends with a 60-second window and advances the clock 1283 seconds. The resume then reaches the service and succeeds.
• sandbox::tests::an_adopted_vm_resumes_past_the_reported_window_through_the_service replaces the test that refused an adopted VM's resume past its reported window.
• sandbox::tests::a_resume_the_service_refuses_for_an_ended_vm_is_window_closed_and_recorded checks the ERR_WINDOW_CLOSED error, the TERMINATED state, and that a second resume is refused before any call.
• control::microvm::tests::a_resume_refused_for_a_vm_that_terminated_while_suspended_is_window_closed checks the mapping for both TERMINATING and TERMINATED.
• control::microvm::tests::a_resume_refused_for_a_vm_a_terminate_ended_names_no_cause_and_keeps_the_refusal checks a VM a terminate ended (stateReason "Success."): the exact message, which names no cause and doubles no period, and the service's refusal as the error's source.
• guards::history::a_resume_of_a_vm_the_service_ended_exits_window_closed_and_records_no_resume checks that the CLI exits 8 and records no resume.
• every_walked_path_of_the_client_model_replays_against_the_sandbox replays the model's new STATE-12 paths.

Three new tests pass with the fix taken out, so FAIL_TO_PASS doesn't claim them. They are a_resume_refused_for_a_live_or_unreadable_vm_keeps_the_services_answer, a_resume_denied_for_its_credentials_reads_no_state, and a_resume_refused_for_a_live_vm_leaves_the_sandbox_suspended_and_resumable. They hold behavior that main already had.

Five hand-written faults, each fired with mise run guards:fire -- --only <id>:

• model-client-state-12-refused-resume-unrecorded is new in verify/guards/faults/agentd-model.toml. It leaves the model's VM SUSPENDED after the service refuses a resume for an ended VM, and the property "STATE-12 a resume refused for an ended VM leaves it TERMINATED and recorded terminated" fails. It printed fired: model-client-state-12-refused-resume-unrecorded.
• model-client-state-12-resume-refused-locally is new in the same file. It puts the removed check back into the model, so the client refuses a resume from SUSPENDED for a VM the platform ended. The property "STATE-12 a resume from SUSPENDED reaches the service" fails. It printed fired: model-client-state-12-resume-refused-locally.
• model-conformance-client-resume-resends-payload in verify/guards/faults/model-conformance-client.toml now anchors on the new resume block. The fault sends the run hook's payload again after a resume, and the replay reports "run-hook payloads: sandbox 2, model 1". It printed fired: model-conformance-client-resume-resends-payload.
• model-conformance-client-skips-moved in the same file now anchors on the planner's remaining skip, because the planner no longer skips a closed window. The fault stops skipping a completion that arrives after another step, and the planner fails on the pinned skip set before it replays anything. It printed fired: model-conformance-client-skips-moved.
• resume-window-closed-drops-the-refusal is new in verify/guards/faults/resume-closed-window.toml. It removes the .with_source(refused) that attaches the service's refusal, and the terminate test fails on "the refusal is the source". It printed fired: resume-window-closed-drops-the-refusal.

After each fire, the tool printed "restored, every command passes again". verify/guards/unregistered.txt loses two lines. One is the replaced an_adopted_vm_refuses_a_resume_past_the_reported_window. The other is the_lifecycle_watch_follows_every_transition, which had a STATE-12 note attached by mistake and no longer carries it.

Follow-ups

• A live conformance check for STATE-12 would suspend a VM through the idle policy, wait past the window, and expect ERR_WINDOW_CLOSED with the state from GetMicrovm. It would also resume an explicit suspend past its window. verify/spec/traced/STATE.toml waives the live check until the suite runs one.
• A parity case for the window-closed refusal can be added once the Python and TypeScript parity runners can script a control plane.
• mise run spec:core fails on main because of the word "never" in TRAP-4. This change doesn't touch TRAP-4.
• A Sandbox whose VM the idle policy suspended still holds it as RUNNING, so Sandbox::resume refuses from RUNNING with ERR_INVALID_ARG (STATE-7) and never asks the service. That's main's behavior, and this change only documents it; A Sandbox never sees a platform idle suspend, so resume refuses from RUNNING #603 tracks it.

Platform claims

This changes claims about AWS behavior in the CLI help, the CLI README, the binding docs, the debugging guide, three site operations pages, and the long-lived VM tutorial. docs/PLATFORM.md carries the same measurement through #598, merged as f410ec2. The claims come from the measurement in #597. It was taken on 2026-10-12 in us-east-1 against API version 2025-09-09, with one VM per case and autoResumeEnabled set to false.

• An idle-triggered suspend, with maxIdleDurationSeconds 60 and suspendedDurationSeconds 60, left the VM TERMINATED about 60 seconds after it suspended. Its stateReason was "MicroVM exceeded maximum lifetime." ResumeMicrovm then returned a ValidationException saying the VM "has been terminated and its state cannot be changed."
• An explicit SuspendMicrovm within 10 seconds of RUNNING, on two VMs with the idle case's own policy (maxIdleDurationSeconds 60, suspendedDurationSeconds 60), left both SUSPENDED 906 and 917 seconds later with stateReason null. ResumeMicrovm was accepted, and both were RUNNING within 20 seconds. A review of docs: one current answer per platform fact, and a prose rule that holds it #598 asked for this run, because the first explicit-suspend VM had maxIdleDurationSeconds 3600; that VM was SUSPENDED 1283 seconds later and resumed too.
• AWS's IdlePolicy reference describes suspendedDurationSeconds as the longest a MicroVM can remain suspended, with no exception for an explicit suspend, so the help and docs call the explicit case measured behavior AWS doesn't promise.
• About 4 hours later, GetMicrovm still answered TERMINATED for the idle-suspended VM and for one ended with microvm terminate (stateReason "Success."), and ResumeMicrovm refused both with the same ValidationException. So the refusal doesn't say what ended a VM, and ERR_WINDOW_CLOSED names no cause.
• The 2026-09-23 sample in docs/PLATFORM.md agrees. That VM was explicitly suspended with a 60-second window and was still SUSPENDED at 240 seconds.

Scope

• This is not an orchestrator or a fork implementation
  (see docs/STRATEGY.md).

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.