Summary
When multiple workers finish registration close together, the gateway can keep an aggregate worker capacity that reflects only the first ready worker. The registry later reports every worker as healthy and ready, but priority admission remains permanently under-capacity until an unrelated lifecycle event triggers another snapshot.
Reproduction
This was reproduced against prod commit f6163ad9 with two Kimi K3 workers, each reporting max_running_requests=38:
- Start SMG with a replica-aware or static priority partition.
- Register or statically discover two workers concurrently.
- Let both registration workflows finish activation.
- Observe
/workers: both workers are ready, healthy, and report 38 slots.
- Observe scheduler capacity: it remains 38 instead of 76.
The race reproduced with both IGW dynamic registration and regular static worker startup. It was scheduling-dependent: one control startup reached 76, while later otherwise-identical startups remained at 38.
Root cause
ActivateWorkersStep calls worker.set_status(Ready) directly. That mutates the shared worker object without sending the registry's StatusChanged event. WorkerCapacity and the partition-capacity coordinator are edge-triggered by registry broadcasts, so their last snapshot can remain stale after the silent Pending -> Ready transition.
Expected behavior
Every worker lifecycle transition should go through WorkerRegistry::transition_status, and activating a pending worker should deterministically wake capacity subscribers. Two ready 38-slot workers must produce an aggregate capacity of 76 without requiring an unrelated later event.
Proposed regression coverage
- Pending activation emits exactly one
StatusChanged { Pending, Ready }; idempotent activation of an already-ready worker emits none.
- A deterministic stale-capacity setup reaches the legacy fallback while two 38-slot workers are pending, then reaches worker-reported capacity 76 immediately after activation.
Summary
When multiple workers finish registration close together, the gateway can keep an aggregate worker capacity that reflects only the first ready worker. The registry later reports every worker as healthy and ready, but priority admission remains permanently under-capacity until an unrelated lifecycle event triggers another snapshot.
Reproduction
This was reproduced against
prodcommitf6163ad9with two Kimi K3 workers, each reportingmax_running_requests=38:/workers: both workers areready, healthy, and report 38 slots.The race reproduced with both IGW dynamic registration and regular static worker startup. It was scheduling-dependent: one control startup reached 76, while later otherwise-identical startups remained at 38.
Root cause
ActivateWorkersStepcallsworker.set_status(Ready)directly. That mutates the shared worker object without sending the registry'sStatusChangedevent.WorkerCapacityand the partition-capacity coordinator are edge-triggered by registry broadcasts, so their last snapshot can remain stale after the silentPending -> Readytransition.Expected behavior
Every worker lifecycle transition should go through
WorkerRegistry::transition_status, and activating a pending worker should deterministically wake capacity subscribers. Two ready 38-slot workers must produce an aggregate capacity of 76 without requiring an unrelated later event.Proposed regression coverage
StatusChanged { Pending, Ready }; idempotent activation of an already-ready worker emits none.