You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
atelet: a cold actor-image cache wedges request-driven resume the same way a cold sandbox-asset cache does, and per-node pre-warming does not fix it #1383
A request-driven resume onto a worker whose node has not yet unpacked the actor image should do one
of three things: pull the image off the request path so the resume succeeds; fail fast and
distinguishably so the caller can back off; or fail slowly but converge, so that retries
accumulate progress and a later attempt succeeds.
Relatedly: pre-warming a node should make it warm. If the documented way to prepare a pool reports
full coverage, a request-driven resume of a real actor onto that pool should work.
Actual Behavior
None of the three. On a node whose actor image is not yet unpacked — sandbox assets already
cached, so #811 is not in play — the first request-driven ResumeActor fails, and every retry
fails identically, with no cumulative progress:
ResumeActor → workflow failed at step CallAteletRestore: while restoring durable snapshot:
while creating "agent" OCI bundle: in imageCache.EnsureImage:
while unpacking layer sha256:77286e45…:
while writing contents of "usr/libexec/gcc/x86_64-linux-gnu/14/lto1" from tar stream:
context deadline exceeded
Six attempts, each dying in the same large layer at ~5.5s elapsed — the router's 5s
parked-request budget (DefaultParkedRequestBudget, cmd/atenet/internal/router/ingress/parking.go:29). The gateway returned HTTP 500 after 23.8s and
the actor was left stranded in ACTOR_STATE_RESUMING.
The same image on the same nodes warms fine through the control plane, which carries no short
client deadline: ten concurrent kubectl-ate resume actor of the real template covered all ten
workers in 15.9s, after which the identical request-driven turn succeeded in 12.2s and a
follow-up in 3.2s. So the unpack is not broken — its interaction with a short caller deadline
is. That is the same shape as #811, reached through a different cache.
And the pre-warm expectation fails silently: our coverage artefact reported 10/10 workers
warmed immediately before the wedge above, because it was produced with a light stand-in
template. See "The pre-warm guidance trap" below.
Steps to Reproduce the Problem
Bring up a pool of workers on nodes that have never unpacked the actor image under test — a
fresh cluster, or an existing one after a node-pool replacement. (Ours: 10 workers, 10 fresh
nodes.)
Create an ActorTemplate whose image has at least one layer larger than the router's 5s
parked-request budget can unpack. Ours is a real agent image; the failing layer contains a
full gcc toolchain. Any image with a single fat layer will do.
Create an actor from that template and suspend it, so a durable snapshot exists and the next
turn takes the resume path.
Send a request that routes to the suspended actor, i.e. a request-driven resume through atenet-router, not kubectl-ate resume. It fails at CallAteletRestore inside imageCache.EnsureImage at ~5.5s.
Retry, repeatedly. Every attempt dies in the same layer at the same elapsed time. Check the
node: the layer's temp dir (.tmp-<diffid>-* alongside the layer pool) is gone after each
attempt, and the layer dir was never created. No progress accumulates, so the node never warms.
The actor sits in ACTOR_STATE_RESUMING.
Contrast, on the same node: kubectl-ate resume actor for the same template. It carries no
short deadline, completes, and leaves the unpacked layers behind. Every subsequent
request-driven resume on that node then succeeds.
Why CI does not catch it: e2e suites use small images whose every layer unpacks well inside the
budget, and single-node kind shares one image pool across all workers, so the second test in a
suite is always warm. The bug needs a fat layer and a node that has not seen it.
Platform: GKE, gVisor sandbox class, 10 workers on 10 fresh nodes. Reproduced 27 Aug 2026.
Evidence: pre-warm coverage artefacts for both templates (the light one and the real one),
the atelet log on atelet-7gblv, and the control-plane comparison above.
Root cause
internal/imagecache/imagecache.go, on main at 23633f57. Four mechanisms, and the first is the
one that makes it permanent.
1. Progress is preserved per completed layer only.pull writes the image record before any
unpack so every layer is referenced, and cachedImage re-pulls only what is missing — the code's
comment calls an interrupted pull's record "just resumable progress," and at layer granularity it
is. But unpackLayerToPool (:577) unpacks into a temp dir and renames at the end, cleaning the
temp dir on any error including cancellation, so that "a layer dir either exists complete or not at
all" (its own comment, :575). A single layer that cannot finish inside the caller's budget
therefore restarts from zero on every attempt and never converges. Six attempts on our cluster
made no cumulative progress whatsoever.
This is the same non-convergence Yuan Gao (@ygao-g) identified for sandbox assets in #811 (comment)
after #863 — except here it was never self-healing to begin with, because the rename is at layer
granularity by design rather than by regression.
2. One expiring layer cancels its siblings.pull runs the layers under errgroup.WithContext with layerPullConcurrency = 4 (:490, :96), so the first layer to hit
the deadline cancels the other three in flight, discarding their partial temp dirs too. A pull that
was 3-of-4 layers along loses the in-flight work as well as the failing one.
3. The singleflight propagates one caller's dying deadline to every waiter.EnsureImage
collapses concurrent pulls of the same digest onto the winner (:372), and the comment is
explicit: "The winning call's ctx governs the pull; if it is cancelled the waiters fail too."
Under a burst onto a cold node — several actors of one template landing at once, which is exactly
what a fleet does — one caller's 5s budget expiring takes down every other caller waiting on the
same image, including any that had a longer budget.
4. The caller's context reaches all of this unmodified.ensureAteletRestored
(cmd/ateapi/internal/controlapi/workflow_resume.go) passes the inbound ctx straight into the
atelet Restore RPC, and cmd/atelet/oci.go:114 hands it to EnsureImage. Nothing along that
path gives the unpack a budget of its own.
Why the existing issues and the fix in flight do not cover this
The gVisor zstd work (google/gvisor#14528) makes this more visible rather than less: taking the
asset extract to ~2s removes the failure everyone is currently looking at and leaves this one, on a
slower cache, as the remaining way a fresh node wedges.
The pre-warm guidance trap — worth fixing before any guidance ships
There are two cold caches, and warming with a light stand-in workload fills only the first. We
pre-warmed our own pool with a minimal actor on the reasoning that warming does not need an agent
and a lighter actor warms faster. The coverage artefact reported 10/10 workers warmed. The very
next request-driven turn wedged, because the pool was warm for nothing anyone would actually run.
A coverage claim made with the wrong template is not incomplete, it is confidently wrong, and it
cost us a day of misdiagnosis. So:
Any pre-warm guidance must say per image, and must not suggest a light stand-in workload.
Any coverage artefact must record which template it was made with.
As of 23633f57 there is no pre-warm guidance anywhere in docs/ — a grep across docs/
and every *.md in the tree returns only vendored files. So there is currently nothing for atelet: pre-download sandbox assets from SandboxConfigs #1358 to amend, and a fresh install meets both wedges with nothing to find.
Recovery, for anyone hitting this today
Resume the real template through the control plane once per node — kubectl-ate resume actor
carries no short deadline, completes, and leaves the unpacked layers behind for every later resume
on that node. Ten concurrent calls covered ten workers in 15.9s for us. Per template, not per
node. (For an actor already stranded in ACTOR_STATE_RESUMING, DeleteActor with any_state set
now works — per Zoe Zhao (@zoez7) on #1238.)
Candidate fixes
Pre-warm by image, off the request path — the equivalent of atelet: pre-download sandbox assets from SandboxConfigs #1358 for the image cache, keyed
on the ActorTemplates a node's WorkerPools can schedule. Ideally the same mechanism, so there is
one answer to "is this node warm" rather than two.
Make an interrupted unpack resumable within a layer, so retries accumulate progress instead
of resetting. Weakest of the three — it fights the "complete or not at all" invariant, which
exists for good reasons.
Fail fast and distinguishably when the image is not yet local — an "image not yet local,
pull in progress" error the router can recognise and back off on, instead of burning the whole
budget on an unpack that will be cancelled. Pairs well with 1: the fast failure covers the
window before pre-warm completes.
Do not let the singleflight propagate the winner's deadline to the waiters — decouple the
pull's context from any single caller's, so a fleet burst does not have one short-budget request
killing the pull for everyone. Small and independently useful.
Expected Behavior
A request-driven resume onto a worker whose node has not yet unpacked the actor image should do one
of three things: pull the image off the request path so the resume succeeds; fail fast and
distinguishably so the caller can back off; or fail slowly but converge, so that retries
accumulate progress and a later attempt succeeds.
Relatedly: pre-warming a node should make it warm. If the documented way to prepare a pool reports
full coverage, a request-driven resume of a real actor onto that pool should work.
Actual Behavior
None of the three. On a node whose actor image is not yet unpacked — sandbox assets already
cached, so #811 is not in play — the first request-driven
ResumeActorfails, and every retryfails identically, with no cumulative progress:
Six attempts, each dying in the same large layer at ~5.5s elapsed — the router's 5s
parked-request budget (
DefaultParkedRequestBudget,cmd/atenet/internal/router/ingress/parking.go:29). The gateway returned HTTP 500 after 23.8s andthe actor was left stranded in
ACTOR_STATE_RESUMING.The same image on the same nodes warms fine through the control plane, which carries no short
client deadline: ten concurrent
kubectl-ate resume actorof the real template covered all tenworkers in 15.9s, after which the identical request-driven turn succeeded in 12.2s and a
follow-up in 3.2s. So the unpack is not broken — its interaction with a short caller deadline
is. That is the same shape as #811, reached through a different cache.
And the pre-warm expectation fails silently: our coverage artefact reported 10/10 workers
warmed immediately before the wedge above, because it was produced with a light stand-in
template. See "The pre-warm guidance trap" below.
Steps to Reproduce the Problem
fresh cluster, or an existing one after a node-pool replacement. (Ours: 10 workers, 10 fresh
nodes.)
light template through the ActorTemplate reconciler and let
ensureSandboxAssetsrun tocompletion on every node. Confirm
static-filesis populated. From here on the gVisor asset isa cache hit and out of the picture.
parked-request budget can unpack. Ours is a real agent image; the failing layer contains a
full gcc toolchain. Any image with a single fat layer will do.
turn takes the resume path.
atenet-router, notkubectl-ate resume. It fails atCallAteletRestoreinsideimageCache.EnsureImageat ~5.5s.node: the layer's temp dir (
.tmp-<diffid>-*alongside the layer pool) is gone after eachattempt, and the layer dir was never created. No progress accumulates, so the node never warms.
The actor sits in
ACTOR_STATE_RESUMING.kubectl-ate resume actorfor the same template. It carries noshort deadline, completes, and leaves the unpacked layers behind. Every subsequent
request-driven resume on that node then succeeds.
Why CI does not catch it: e2e suites use small images whose every layer unpacks well inside the
budget, and single-node kind shares one image pool across all workers, so the second test in a
suite is always warm. The bug needs a fat layer and a node that has not seen it.
Specifications
4c1b37d(the base ofrelease-0.1-rc, and it contains atelet: support dynamic asset extraction based on the format and fail-fast context checks #863 /c9bccde5).Confirmed still unfixed on
mainat23633f57by reading the code —internal/imagecache/hasexactly one commit since
4c1b37d(85d404a8) and it edits a README.the atelet log on
atelet-7gblv, and the control-plane comparison above.Root cause
internal/imagecache/imagecache.go, onmainat23633f57. Four mechanisms, and the first is theone that makes it permanent.
1. Progress is preserved per completed layer only.
pullwrites the image record before anyunpack so every layer is referenced, and
cachedImagere-pulls only what is missing — the code'scomment calls an interrupted pull's record "just resumable progress," and at layer granularity it
is. But
unpackLayerToPool(:577) unpacks into a temp dir and renames at the end, cleaning thetemp dir on any error including cancellation, so that "a layer dir either exists complete or not at
all" (its own comment,
:575). A single layer that cannot finish inside the caller's budgettherefore restarts from zero on every attempt and never converges. Six attempts on our cluster
made no cumulative progress whatsoever.
This is the same non-convergence Yuan Gao (@ygao-g) identified for sandbox assets in
#811 (comment)
after #863 — except here it was never self-healing to begin with, because the rename is at layer
granularity by design rather than by regression.
2. One expiring layer cancels its siblings.
pullruns the layers undererrgroup.WithContextwithlayerPullConcurrency = 4(:490,:96), so the first layer to hitthe deadline cancels the other three in flight, discarding their partial temp dirs too. A pull that
was 3-of-4 layers along loses the in-flight work as well as the failing one.
3. The singleflight propagates one caller's dying deadline to every waiter.
EnsureImagecollapses concurrent pulls of the same digest onto the winner (
:372), and the comment isexplicit: "The winning call's ctx governs the pull; if it is cancelled the waiters fail too."
Under a burst onto a cold node — several actors of one template landing at once, which is exactly
what a fleet does — one caller's 5s budget expiring takes down every other caller waiting on the
same image, including any that had a longer budget.
4. The caller's context reaches all of this unmodified.
ensureAteletRestored(
cmd/ateapi/internal/controlapi/workflow_resume.go) passes the inbound ctx straight into theatelet
RestoreRPC, andcmd/atelet/oci.go:114hands it toEnsureImage. Nothing along thatpath gives the unpack a budget of its own.
Why the existing issues and the fix in flight do not cover this
imageCache.EnsureImage, butit rules the image out and is right to:
after 0s, ctx err: context canceled, zero bytesfetched, the context already dead before the pull began. Ours is the inverse — assets warm,
deadline dies inside the unpack. A reader of Cold-node gVisor release extract (~20s, silent, uncancellable) exceeds the router resume budget — first resume after a release bump always fails #811 concludes the image cache is fine.
Warming a node with a golden does nothing for a different template's image.
SandboxConfig. The actor image is named by theActorTemplate, not the SandboxConfig, so atelet: pre-download sandbox assets from SandboxConfigs #1358 structurally cannot reach it. A node atelet: pre-download sandbox assets from SandboxConfigs #1358 has
fully pre-warmed is still cold for every real agent image.
The gVisor zstd work (google/gvisor#14528) makes this more visible rather than less: taking the
asset extract to ~2s removes the failure everyone is currently looking at and leaves this one, on a
slower cache, as the remaining way a fresh node wedges.
The pre-warm guidance trap — worth fixing before any guidance ships
There are two cold caches, and warming with a light stand-in workload fills only the first. We
pre-warmed our own pool with a minimal actor on the reasoning that warming does not need an agent
and a lighter actor warms faster. The coverage artefact reported 10/10 workers warmed. The very
next request-driven turn wedged, because the pool was warm for nothing anyone would actually run.
A coverage claim made with the wrong template is not incomplete, it is confidently wrong, and it
cost us a day of misdiagnosis. So:
23633f57there is no pre-warm guidance anywhere indocs/— a grep acrossdocs/and every
*.mdin the tree returns only vendored files. So there is currently nothing foratelet: pre-download sandbox assets from SandboxConfigs #1358 to amend, and a fresh install meets both wedges with nothing to find.
Recovery, for anyone hitting this today
Resume the real template through the control plane once per node —
kubectl-ate resume actorcarries no short deadline, completes, and leaves the unpacked layers behind for every later resume
on that node. Ten concurrent calls covered ten workers in 15.9s for us. Per template, not per
node. (For an actor already stranded in
ACTOR_STATE_RESUMING,DeleteActorwithany_statesetnow works — per Zoe Zhao (@zoez7) on #1238.)
Candidate fixes
on the ActorTemplates a node's WorkerPools can schedule. Ideally the same mechanism, so there is
one answer to "is this node warm" rather than two.
of resetting. Weakest of the three — it fights the "complete or not at all" invariant, which
exists for good reasons.
pull in progress" error the router can recognise and back off on, instead of burning the whole
budget on an unpack that will be cancelled. Pairs well with 1: the fast failure covers the
window before pre-warm completes.
pull's context from any single caller's, so a fleet burst does not have one short-budget request
killing the pull for everyone. Small and independently useful.
Related: #811, #1238, #1358, #863. None of them fix this.