Skip to content

fix(worker): emit activation status events - #57

Merged
DavidBellamy merged 1 commit into
prodfrom
fix/56-worker-capacity-status-events
Aug 11, 2026
Merged

fix(worker): emit activation status events#57
DavidBellamy merged 1 commit into
prodfrom
fix/56-worker-capacity-status-events

Conversation

@DavidBellamy

@DavidBellamy DavidBellamy commented Aug 7, 2026

Copy link
Copy Markdown

Description

Problem

Worker registration activation changed each shared worker object from Pending to Ready directly. That bypassed the registry event stream used by aggregate worker capacity and partition admission. Two 38-slot workers could therefore remain exposed as only 38 scheduler slots even though both were healthy and ready.

Activation also has to fail closed if the workflow worker is replaced between registration and activation. URL or revision checks alone are insufficient because same-ID replacement shares mutable runtime state and its revision with the old Arc.

Solution

Resolve the registered worker ID by URL, then use WorkerRegistry::apply_if_revision and Arc::ptr_eq inside the registry per-ID mutation lock. Only the exact worker instance registered by this workflow may transition to Ready.

The canonical StatusChanged { Pending, Ready } event now wakes worker-capacity and partition-capacity consumers. Activation remains idempotent for the exact already-ready worker. Missing, removed, recreated, or replaced workers fail closed without activating the replacement or emitting a false event.

Closes #56.

Changes

  • Make activation status transition and exact-instance verification atomic.
  • Emit the registry lifecycle event needed to refresh scheduler capacity.
  • Add pending, idempotency, and same-ID replacement regressions.
  • Add a deterministic aggregate-capacity regression requiring two 38-slot activations to produce 76.

Test Plan

Local verification on exact prod base 8cf8b1d0df752475bb561752c881ad7c3b6b5ca9:

  • Focused activation tests: 3 passed.
  • cargo test -p smg --lib --quiet: 1,499 passed, 5 ignored, 0 failed.
  • cargo +nightly fmt --all -- --check.
  • git diff --check.
  • cargo clippy -p smg --lib --no-deps completed with only the unchanged warning in model_gateway/src/middleware/concurrency.rs:213.

The same-ID replacement regression is intentionally stronger than a missing-worker check: it proves a stale workflow cannot activate a replacement that shares the original worker runtime and revision.

Checklist
  • Formatting and focused/full library tests pass.
  • cargo clippy --all-targets --all-features -- -D warnings passes. The unchanged base still reports existing warnings outside this one-file diff.
  • Optional documentation updated.

@DavidBellamy

Copy link
Copy Markdown
Author

M1 runtime validation from the corrected owner-pressure A/B (job 1191549)

This PR was common to both arms. Two independently registered Kimi K3 workers each reported 38 running-request slots. In both the control and treatment gateway processes, runtime telemetry advanced deterministically from one worker / 38 slots to two workers / 76 aggregate slots and remained complete for all 228 workload decisions.

The treatment additionally carried #55; the control carried only this PR. Both runtime-loaded compiled-binary hashes and embedded provenance were checked before traffic, so this validation does not rely on the earlier stale build artifact.

Both arms completed 228/228 turns with zero terminal failures, 429 retries, retractions, or CUDA OOMs. This is positive runtime evidence for the activation-event fix, but the PR remains draft pending code-owner review. Production Comet was not changed.

Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
@DavidBellamy
DavidBellamy force-pushed the fix/56-worker-capacity-status-events branch from 63b022b to 43c03c7 Compare August 11, 2026 07:28
@DavidBellamy
DavidBellamy marked this pull request as ready for review August 11, 2026 07:39
@DavidBellamy
DavidBellamy requested a review from a team August 11, 2026 07:39
@DavidBellamy
DavidBellamy enabled auto-merge (squash) August 11, 2026 07:46
@DavidBellamy
DavidBellamy merged commit 03898f2 into prod Aug 11, 2026
11 of 12 checks passed
@DavidBellamy
DavidBellamy deleted the fix/56-worker-capacity-status-events branch August 11, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant