ResumeActor returns Aborted "concurrent update conflict" when concurrent resumes pick the same worker

[sairajp-rewind]

Expected Behavior

ResumeActor should absorb transient optimistic-concurrency conflicts on the worker-assignment step and return either a resumed actor or a terminal error. A caller resuming an actor that nothing else is touching should not have to implement its own retry loop for a conflict ateapi created internally.

Actual Behavior

Under sustained suspend/resume load on a 1,000-actor fleet, ResumeActor returns codes.Aborted with the message "concurrent update conflict, please retry" throughout the run. The bounded backoff in ensureWorkerAssigned exhausts its five attempts and returns a bare store.ErrVersionConflict, which RPCService.ResumeActor maps to codes.Aborted.

The conflict is between concurrent resumes competing for the same worker. There is no reservation between the moment the scheduler picks a worker out of the cache and the moment the claim is written, so two calls can select the same worker at the same cached version and both write a conditional update predicated on it. One wins and the other is rejected.

This happens with the worker pool fully provisioned, so it is distinct from codes.ResourceExhausted, "no free workers available", which did not occur at all in the runs below. The scheduler always found a worker to pick; the failure is in claiming it.

Workload

• 1,000 concurrent actors on a GKE cluster, worker pool sized 1:1, two ate-api-server replicas, PostgreSQL-backed store.
• Each actor loops for 10 minutes: SuspendActor over gRPC, then ResumeActor over gRPC called directly against the control plane, then three HTTP requests through the router (read 64 MiB, read it again warm, overwrite 64 MiB), each read verified with a SHA-256 digest.
• The ActorTemplate declares a durableDir volume with no external volume backing it, and snapshotsConfig onPause: Full, onCommit: Full.
• gVisor sandbox class.

The direct ResumeActor call is what makes this visible. ActorResumer always treats codes.Aborted as retryable regardless of parking mode, and rides it out for a 15 s budget when request parking is disabled, or for the parking budget when it is enabled. Either way a resume driven through the router absorbs the conflict. Clients that call the control-plane API directly, including kubectl-ate, get the raw error.

Evidence

Two consecutive 10-minute runs on one cluster, same commit and same image digest, with the two ate-api-server pods never restarted between them. The worker pool started the first run still filling and the second run already warm. Both runs produced aborts continuously once load was established, and the fleet did not converge out of the condition while load ran. Counts come from ate-api-server structured request logs over each run's 10-minute load window.

	Run 1	Run 2
Worker pool when load started	filling, 1 → 1,000	warm, 1,000 ready
ResumeActor calls, all	75,963	77,838
of which are no-ops on already-running actors	56,684	58,106
resumes that reached worker assignment	19,279	19,732
Aborted, "concurrent update conflict"	52	59
Distinct actors affected	50	59
ResourceExhausted, "no free workers available"	0	0

The no-op row is the router resuming an actor that is already running. Those return from the pre-lease fast path in a median of 0.3 ms, never take the actor lease, and cannot reach worker assignment, so they are separated out rather than mixed into the totals.

The remaining observations come from "Picked worker" records over the same windows.

Two calls pick the same worker at the same cached version. Grouping "Picked worker" records by the worker name and cache version they carry, a two-minute sample of one run's steady state contains several hundred clusters in which two or three different ResumeActor calls picked the same worker at the identical version. The picks within a cluster land within tens of milliseconds of each other. Each of those calls then issues an UpdateWorker carrying the same precondition, and only one can succeed.

Calls that abort are contested far more often than calls in general. Of the 4,448 picks in that sample, 848 were contested in this sense: another call picked the same worker at the same version within 40 ms. Among the picks made by calls that ended in Aborted, 37 of 37 were contested, and in every one of those the rival picked first. Other two-minute windows over the same run give the same picture, with the occasional uncontested pick. Contention is common across the board and close to universal among the failures, which is what distinguishes it from an incidental correlation.

It is not only cross-replica cache divergence. Roughly half of those rival picks came from the other ate-api-server replica and roughly half from the same one. A same-replica collision means one process handed the same cached worker to two concurrent RPCs, so the two caches disagreeing with each other is not a sufficient explanation on its own.

Retries pick a different worker and collide again. Retries almost always selected a worker the call had not already tried. The failure is therefore not a single stale cache entry being re-read, but repeated contention across the free set.

A ready pool is not an idle pool. Polling kubectl get workerpool every 10 seconds through both load windows returned ready=1000 of spec=1000 on every sample, and neither run logged "no free workers available", the only route out of scheduling.ErrNoCapacity. That establishes a thousand healthy workers rather than a thousand available ones. With the pool sized 1:1, the set a resume can actually pick from holds only the workers whose actors are suspended at that instant, which is a small fraction of the fleet. Concurrent resumes are therefore drawing from a much smaller set than the pool size suggests.

Every abort terminates on the retry budget. All 111 aborts across the two runs completed in 168–283 ms server-side, with a p50 of 231 ms. That matches the ensureWorkerAssigned budget: with wait.Backoff{Steps: 5, Duration: 10ms, Factor: 2.0, Jitter: 1.0} the four sleeps are uniform over [10,20), [20,40), [40,80), and [80,160) ms, totalling 150–300 ms. The band was computed from the code before it was looked for in the data. Every aborted call also shows more than one "Picked worker" record, never exactly one, which is consistent with the retry loop being what produced them.

The Aborted string is returned from several places in this workflow and the error itself does not identify which one fired, so the above does not pin the failure to a single line. What it does establish is that the calls which fail are the ones whose worker picks were contested, and that they fail after exhausting a retry loop rather than on a first attempt.

Two notes on scope. An attempt that adopts a worker already carrying this actor's assignment skips the pick and logs nothing, so the pick records above undercount attempts. And a terminal Aborted means all five attempts lost; conflicts resolved on attempts two through five surface nowhere, so contention is more common than the failure count suggests. Separately, run 1 lost a single actor to a crash, which produced 15 FailedPrecondition responses on later resumes of that actor. That is a different failure mode and is not included above.

Observability Gap

Nothing on the server side counts this failure. schedulerRecordable excludes version conflicts from the scheduler metric by design, and the worker-claim branch logs nothing on conflict, so retried attempts leave no trace in ate.scheduler.assignment.duration. The Aborted string is shared by 19 sites in controlapi. Three of them can return it during a single ResumeActor, and one of those three covers two distinct conflicts, since both the worker claim and the actor claim escape ensureWorkerAssigned as a bare store.ErrVersionConflict and are mapped at the same place. The error does not identify which step failed. The worker cache is already listed under blind_spots in docs/metrics/substrate.yaml for this shape of failure. Reconstructing the race meant correlating "Picked worker" records across traces, because no counter or log line records a lost claim.

Steps to Reproduce the Problem

1. Bring up a cluster with a worker pool sized 1:1 with the actor count and more than one ate-api-server replica. Wait for the pool to report all workers ready before starting load.
2. Drive ~1,000 actors through a suspend/resume loop for 10 minutes, calling ResumeActor directly over gRPC rather than triggering resume through the router.
3. Collect ResumeActor failures carrying codes.Aborted with "concurrent update conflict, please retry".
4. Check the server-side elapsed times of the aborted calls: they cluster in the 150–300 ms band the retry budget predicts.
5. Collect "Picked worker" records over the same window and group them by worker name and the version in the record. Clusters containing more than one trace ID are concurrent calls competing for one worker. Compare how often picks are contested across all calls against how often they are contested among the calls that aborted.

Specifications

• Platform: GKE, gVisor (runsc) sandbox class, PostgreSQL-backed ateapi store
• Both runs were executed against 39df566d, and all code references are permalinked to that commit

[sairajp-rewind]

Bowei Du (@bowei) Julian Gutierrez Oschmann (@juli4n) PTAL!