Repository navigation
Self-hosted Embed: failed pause or checkpoint loses the sandbox #3658
Description
On self-hosted Embed, a failed pause or checkpoint can stop the original sandbox and leave no snapshot the SDK can restore. Its files and process state become inaccessible.
Reproduce
Pause: disk sync failure
- Create a sandbox and write a file inside it.
- Call
sandbox.pause(keep_memory=True). - Use
straceto makefsyncreturnEIOfor the rootfs diff file during the pause.
Pause returns 500: Error pausing sandbox; the orchestrator reports failed to sync file: input/output error. Reproduced 2/2 times. Without the injected error, pause/resume preserves files and process memory.
Fault injection command
Attach after the diff file appears and before it is synced (about 1–2 seconds). Replace the path with the actual diff file path; a watcher was used to catch this window.
strace -f -p "$(pgrep -x orchestrator)" -qq -yy \
-P "/orchestrator/build/<build-id>-rootfs.ext4-<random-suffix>" \
-e trace=fsync,fdatasync \
-e inject=fsync:error=EIO:when=1 \
-o /tmp/pause-fsync.straceThe original incident followed repeated host No space left on device errors. Those logs do not establish the cause of the later fsync error.
Checkpoint: memory allocation failure
Exhaust the host's reserved hugepage memory pool, then call create_snapshot(). Starting the replacement sandbox fails with mmap memfd: cannot allocate memory. Reproduced 1/1 time with real memory exhaustion on the same release binary. Normal checkpoint/restore works with memory available.
Actual result
In both cases:
- The original sandbox's Firecracker process stops;
get_inforeturns “not found.” - The snapshot build is marked
failed; restoring it returns404: template not found.
The pause failure deletes the local snapshot artifacts. The checkpoint failure leaves local files, but they cannot be restored through the SDK.
Expected result
A failed pause or checkpoint should leave the sandbox recoverable: either the original sandbox remains available or a usable snapshot can be restored.
Environment
- Single-node Embed, local storage (
file:///var/lib/e2b/storage/templates) - Ubuntu 26.04 ARM64 VM with nested virtualization
- Orchestrator
v0.16.202609130627-59497eb9134(unmodified release binary) - API
v0.14.202609170000-908833e4c12; Python SDK2.51.0 - Compose revision:
a065a4ddb3f2c6a4149634d9acb14b62f65839ac
Related: #3659 covers a snapshot marked successful that crashes on restore.
Activity
[-]Self-hosted Embed: failed rootfs diff fsync during Pause leaves the sandbox unaddressable with no tested recovery path (v0.16.202609130627-59497eb9134)[/-][+]Self-hosted Embed: sandbox lost when pause fails[/+][-]Self-hosted Embed: sandbox lost when pause fails[/-][+]Self-hosted Embed: failed pause or checkpoint loses the sandbox[/+]docs: E2B PoC qualification — causal closures, root causes, gate updates
fix(orchestrator): persist checkpoint snapshot when resume-fresh fails
fix(pause): preserve sandbox when snapshot fails, behind a flag
svalleru commentedon Sep 29, 2026
Thanks for the detailed report.
This is the intended fail-closed behavior. A pause or checkpoint only produces a restorable snapshot once every artifact has been written and synced to disk. When the
fsyncof the rootfs diff fails withEIO, which usually happens when the underlying disk, controller, or filesystem is faulty, the kernel discards the unwritten pages, so the diff cannot be trusted; we discard the whole snapshot instead of leaving something restorable that may be corrupt. The checkpoint case ends the same way: the snapshot was never persisted, so its build is marked failed, and the original sandbox had already been stopped as part of taking it. The current default on a failed snapshot is to delete.Closing as working as intended. Closing #3666 and #3668 as well.
AdaAibaby commentedon Sep 30, 2026
Thanks — agreed that a corrupt snapshot must be discarded, and that's not in dispute. LocalDiffFile.CloseToDiff already removes the partial diff on an fsync failure, and neither PR tries to retain a possibly-corrupt snapshot.
The point I'd push back on is narrower: discarding the bad artifact and destroying the sandbox are two separate actions, and only the first is forced by the fault.
On the pause path the guest isn't "stopped as part of taking the snapshot" — pauseVM issues PatchVM{State: Paused}, a suspend. At the moment fsync fails, the VM is still frozen and intact; the failing fsync is on an exported diff copy, not the live overlay. Destroying
it is a choice made in the error path (stopSandboxAsync), not a consequence of the EIO. That's the only thing #3668 changes: on snapshot failure it resumes the already-suspended guest in place and returns FailedPrecondition, corrupt artifact still discarded. It's
flag-gated, default off — byte-identical to today unless a self-hoster opts in.
I take the fail-closed philosophy seriously, and I think the fair question isn't "discard or keep the snapshot" (discard — you're right) but "when the guest is demonstrably still alive at failure time, is destroying it the safe default or the aggressive one?" For a
transient trigger like the reported hugepage ENOMEM, resuming in place reuses memory the guest already holds — it needs no fresh allocation, so it's lighter than destroy-and-recreate, not riskier.