[tiago-mitralab]

Rebase of experimental onto the new stable (#16, upstream 1cea50fcb), with the cherry-pick series curated against what upstream merged in the last five weeks. 13 commits become 7, plus one Mitra fix on top (below).

How to review. The series on top of the new stable: sync/2026-10-19/stable...sync/2026-10-19/experimental

How it lands. Force-push after approval, like stable (backup tag experimental-before-upstream-sync-2026-10-19 at afd964b72).

Dropped, and why

Commit	What	Why it leaves
9aa6edd8f	envd tag-based process lookup (cherry-pick of e2b-dev#3099)	absorbed: upstream 67b318240 carries the identical change to service.go (diffed line by line), even though e2b-dev#3099 is still open
1330d8768	race in Cleanup (Add after run drops the function)	superseded: upstream 9cc9ca757 (2026-10-07) keeps the state check and the registration under the same lock as Run, and runs late cleanups through runLateCleanup outside the lock. That covers the TOCTOU this commit fixed
ace2e0a61	release the lock before running the cleanup in the double-check path	superseded by the same commit: the late cleanup already runs outside the lock
607329576	t.Parallel() in cleanup_test.go	belonged to the test file 1330d8768 added
03dd79359	CODEOWNERS (#13)	already in stable (#12), same content
afd964b72	E2B_DEBUG for the sandbox logger (#15)	already in stable (#14), same content

The four removed cherry-picks stay reachable from the backup tag.

Kept, with the check that says they are still needed

Cherry-pick	Evidence in upstream 1cea50fcb
uffd: log the error, drop the stale TODO	the TODO: If the handle function fails, we should kill the sandbox is still at uffd.go:132
typo to to	still at block/cache.go:480
t.Parallel() in uffd_test.go	goes with the uffd test above
TTL on the node via WaitForExit, plus its tests	no WaitForExit upstream; e2b-dev#3194 was declined there ("keep the sandbox TTL handling on API level only"), kept on purpose because a cell has a single API
Docker image ENV vars reach runtime processes	upstream parses Context.EnvVars at build (phases/base/builder.go) but never merges them into the sandbox config
nsenter instead of ip netns exec (e2b-dev#3037)	fc/process.go still uses ip netns exec; e2b-dev#3037 still open

Mitra fix on top of the cherry-picks

fix(orchestrator): node TTL is a backstop past the end time, stopping as killed is our own commit, on top of the seven so they stay byte-identical for patch-id absorption:

• The node-side TTL (WaitForExit cherry-pick) fired at the end time, the same moment the API evictor acts, and called Stop with no stop reason. So a sandbox the API meant to pause (auto-pause) could be killed first, since the node does not know about auto-pause, and every expiry read as a crash (ERROR, crashed label, per-team crash counter from 44932b67e).
• The lifecycle now waits nodeTTLGrace (2 min) past the end time: with the API up, the evictor decides pause or kill on time and the node only acts when the API never did. WaitForExit returns ErrSandboxTTLExpired, the lifecycle records StopReasonKilled and logs at Info, and when the timer and the Firecracker exit are ready together the exit wins, so a crash never reads as a TTL. Tests pin the expiry to end time plus grace and fail with the grace removed or doubled.
• The uffd handler logged its ERROR before upstream 394ed9413 classified a stop during accept or handshake as benign. The log now follows the classification.

Open decision: the grace against the pause-refusal path. When the node refuses a pause (disk or dedup in flight), the API puts the sandbox back to running and retries every 10 s, and only degrades to a filesystem-only pause after AutoPauseOverstayBudgetMs (120 s by default) from the first refusal. A 2 min grace from the end time can kill it just before that. This needs two opt-ins our fleet does not have today: no host sets PAUSE_REFUSAL_RESTORE (default off), and no mitra-sandbox caller asks for pause on timeout (code_runner_pool.py:569 passes kill; native pause left with mitra-sandbox#380). Options: keep 2 min and revisit when either is turned on; raise the grace to cover budget, retries and pauseTimeout (about 5 min); or have the node push its own deadline while it is answering a pause with ResourceExhausted. The last one is the correct fix, and a bigger change than this PR.

Left as is, on purpose. env_vars_test.go tests a copy of the merge, not production code, and the merge exists twice (server/sandboxes.go, sandbox/reboot.go); uffd_test.go has an empty assertion (SetsHandlerError). Fixing them means editing the cherry-picks; better sent upstream to e2b-dev#3148 and e2b-dev#3132.

Proof

Run on tiago-dev (x86_64, kernel 7.0) with the fleet toolchain, Go 1.26.8 (GOTOOLCHAIN=go1.26.8), on the experimental tip, which contains the whole stable series:

• go build ./... and go vet ./... clean in all 12 modules of go.work; go vet also clean on the stable tip alone (orchestrator, api, client-proxy, shared, envd). go mod tidy leaves no diff in orchestrator, shared and api on the stable tip.
• go test -count=1 ./... in every module, against local postgres:18-alpine and redis:8-alpine exported as PG_TEST_DATABASE_URL and REDIS_TEST_DATABASE_URL (what upstream CI does since 2026-10-18), with TZ=UTC. The userfaultfd and core/oci suites rerun under sudo -E, as they need userfaultfd and unshare. All green, except three packages that need what this machine does not have:
  • orchestrator/pkg/sandbox/rootfs (TestUblk_*): needs root and ublk_drv. Deliberately not run here, because loading ublk on a kernel 7.0 host that serves other work is exactly what hung sf-2 in October. These run on e2b-dev before the rollout.
  • envd/internal/services/filesystem (two FUSE tests): bindfs not installed here.
  • orchestrator/pkg/server TestUpdateTracksWorkWhileLockedAndPublishing fails only outside UTC (the host is on America/Sao_Paulo); passes with TZ=UTC. Upstream test, not touched by this series.

Not done yet, and gated on an explicit go: build and the ublk suites on e2b-dev, bootstrap, template rebuild with envd 0.10.0 and the load test ladder from mitra-dev.

🤖 Generated with Claude Code