Cold-node gVisor release extract (~20s, silent, uncancellable) exceeds the router resume budget — first resume after a release bump always fails

[dberkov]

Symptom

On a node whose gVisor asset cache is cold, the first ResumeActor reliably fails with:

workflow failed at step CallAteletRestore: while restoring durable snapshot:
rpc error: code = Canceled desc = context canceled

and atelet's Restore reports (with newly added pull logging):

while creating "counter" OCI bundle: in imageCache.EnsureImage:
in remote.Image (registry "gcr.io", gcp_auth=true, after 0s, ctx err: context canceled):
Get "https://gcr.io/v2/": context canceled     elapsed-time: 19.9s

The actor image is not the problem — zero bytes of it were fetched (after 0s); the context was already dead when the pull began.

Timeline (GKE, demo/my-counter-4, trace 96d4b7e35eee327b1ec90a0dc3950df6)

t	event
15:51:01.42–01.44	snapshot files downloaded + zstd-decompressed (5.2MB total, <60ms)
~15:51:06	caller (atenet-router, ~5s resume deadline) cancels the context
15:51:01.44–15:51:21.17	19.7s silent gap: ensureSandboxAssets downloads gs://gvisor/releases/release/20260803/x86_64/gvisor.tar.bz2 and extracts it via extractTarBz2
15:51:21.17	prep leg reaches the OCI image pull; first HTTP request dies instantly on the canceled context

Root cause

cmd/atelet/sandbox_assets.go fetchGVisorRelease → extractTarBz2 (introduced in #684, when the gVisor asset became a release tarball instead of a bare runsc binary):

• Silent: no log lines for the download or the extraction — a 20s hole in the trace.
• Uncancellable: the bzip2+tar extraction loop has no ctx checks (Go's stdlib bzip2 is single-threaded and slow), so it runs ~15s past the caller's cancellation.
• Fleet-wide cold: Update gVisor release which supports multiple durable-dirs #787 bumped the pinned release, invalidating the content-addressed cache (StaticFilesDir, hostPath) on every node at once — so every node's first resume after the rollout hits this.

#684's benchmarks measure warm-asset-cache steady state (which got faster); the per-node, per-release cold extraction cost was invisible to them.

Interacting existing issues: #606 (router retry budget), #646 (Canceled strands actors in RESUMING/SUSPENDING).

Mitigating behavior (why it self-heals)

The extraction completes and renames into the content-addressed release dir even though the RPC's caller is gone, so the node pays the cost once; the next resume on that node finds the assets cached and succeeds. Each node fails exactly one restore per release bump (more if the actor isn't retried — the actor stays RESUMING until another request retries it).

Candidate fixes (roughly independent)

1. Observability: log start/size/duration of asset downloads and the tarball extraction in ensureSandboxAssets/fetchGVisorRelease, so the phase is attributable from logs (the restore's phase metrics do record SnapshotPhaseSandboxAssets, but nothing lands in the trace).
2. Pre-warm off the request path: have atelet pre-fetch/extract pinned gVisor releases at startup (from the node's SandboxConfigs), so a release bump costs the node at rollout time, not on the first user resume.
3. ctx checks in extractTarBz2/downloadVerified: fail fast on cancellation. Trade-off: today the cancelled extract still populates the cache, which is what makes the retry succeed — aborting early would make retries pay the full cost again unless the extraction is detached from the request context instead (see 4).
4. Detach node-local cache population from the caller's deadline: run asset extraction (and image pulls, cf. the layer pool) under a server-side context so a hung-up caller doesn't abort work whose product is a shared cache. Related to the broader deadline discussion in [P1] Actor lock TTL (30s) exceeds router retry budget (15s) — ateapi crash causes guaranteed 15–30s user-visible outage #606/[P2] Transient atelet unavailability defaults to codes.Internal — outside the documented retryable set; strands actor in SUSPENDING and surfaces as http/500 #646.

🤖 Generated with Claude Code

[BenTheElder]

How long did it take before with the bare binary? Should we just revert?

Warming only sort of works. We need to be able to fetch the version pinned to an actor. There may be many versions.

[dberkov]

Benjamin Elder (@BenTheElder) What if, at atelet startup, we pre-download all possible gVisor and microVM images for every registered SandboxConfig?
Additionally, it could continuously monitor for changes and automatically download images whenever a new SandboxConfig appears.

Since this data is essentially mandatory in any case, proactively caching it would be well worth it.

[BenTheElder]

FYI Zoe Zhao (@zoez7) Haven Xia (@HavenXia)
I dunno, sounds complex and fragile. How does that work if you have lots of sandbox versions?

Are we downloading direct from gvisor release instead of staging to the substrate instance owned object store?

[zoez7]

What if, at atelet startup, we pre-download all possible gVisor and microVM images for every registered SandboxConfig?

I think an optimization is we can have each Workerpool specify the SandboxConfigVersion it needs, and schedule the ones with the same SandboxConfigVersion the same node, so that atelet can download sandbox runtime assets based on the workerpool config. Each ActorTemplateVersion also need to specify their SandboxConfigVersion, and we select matching workers during scheduling.

[BenTheElder]

I think an optimization is we can have each Workerpool specify the SandboxConfigVersion it needs, and schedule the ones with the same SandboxConfigVersion the same node

I think we want to break the connection between workerpool and sandbox config ... and we need to support existing snapshots which may be at older config ... if everytime a node first schedules a snapshot it takes this long ...

[zoez7]

we want to break the connection between workerpool and sandbox config .

We do, but I don't think we can completely break the connection, since each SandboxConfig will need to know "all ateom versions supporting this SandboxConfigVersion".

[BenTheElder]

We do, but I don't think we can completely break the connection, since each SandboxConfig will need to know "all ateom versions supporting this SandboxConfigVersion".

I don't think sandboxconfig needs to know this. Probably not the right issue to discuss it in though.

For the purpose of this issue ... I think we need to drive down the real cost to fetch it to a node.

[ygao-g]

Heads up that the "why it self-heals" section above is no longer accurate on main. #863 added the ctx.Err() check to the extraction loop — candidate fix 3 from this issue — without fix 4, so the aborted extract discards its temp dir rather than renaming into the content-addressed dir. Retries are now fresh cache misses and never converge. Filed the steady-state consequence as #1238.

[BenTheElder]

Pre-warm off the request path: have atelet pre-fetch/extract pinned gVisor releases at startup (from the node's SandboxConfigs), so a release bump costs the node at rollout time, not on the first user resume.

Zoe Zhao (@zoez7) I know you've been thinking about how to manage sandboxConfig, this is something we should consider -- if we could identify which sandbox config are relevant for the ateoms, and therefore the atelet, we could do this pre-warming. It does assume the cardinality of configs will be low but the more I think about it the more that has to be true, or downloading has to be extremely cheap/efficient. Otherwise the system devolves anyhow.