Skip to content

Worker activation can leave aggregate capacity stale #56

Description

@DavidBellamy

Summary

When multiple workers finish registration close together, the gateway can keep an aggregate worker capacity that reflects only the first ready worker. The registry later reports every worker as healthy and ready, but priority admission remains permanently under-capacity until an unrelated lifecycle event triggers another snapshot.

Reproduction

This was reproduced against prod commit f6163ad9 with two Kimi K3 workers, each reporting max_running_requests=38:

  1. Start SMG with a replica-aware or static priority partition.
  2. Register or statically discover two workers concurrently.
  3. Let both registration workflows finish activation.
  4. Observe /workers: both workers are ready, healthy, and report 38 slots.
  5. Observe scheduler capacity: it remains 38 instead of 76.

The race reproduced with both IGW dynamic registration and regular static worker startup. It was scheduling-dependent: one control startup reached 76, while later otherwise-identical startups remained at 38.

Root cause

ActivateWorkersStep calls worker.set_status(Ready) directly. That mutates the shared worker object without sending the registry's StatusChanged event. WorkerCapacity and the partition-capacity coordinator are edge-triggered by registry broadcasts, so their last snapshot can remain stale after the silent Pending -> Ready transition.

Expected behavior

Every worker lifecycle transition should go through WorkerRegistry::transition_status, and activating a pending worker should deterministically wake capacity subscribers. Two ready 38-slot workers must produce an aggregate capacity of 76 without requiring an unrelated later event.

Proposed regression coverage

  • Pending activation emits exactly one StatusChanged { Pending, Ready }; idempotent activation of an already-ready worker emits none.
  • A deterministic stale-capacity setup reaches the legacy fallback while two 38-slot workers are pending, then reaches worker-reported capacity 76 immediately after activation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions