fix(worker): emit activation status events - #57
Conversation
|
M1 runtime validation from the corrected owner-pressure A/B (job This PR was common to both arms. Two independently registered Kimi K3 workers each reported 38 running-request slots. In both the control and treatment gateway processes, runtime telemetry advanced deterministically from one worker / 38 slots to two workers / 76 aggregate slots and remained complete for all 228 workload decisions. The treatment additionally carried #55; the control carried only this PR. Both runtime-loaded compiled-binary hashes and embedded provenance were checked before traffic, so this validation does not rely on the earlier stale build artifact. Both arms completed 228/228 turns with zero terminal failures, 429 retries, retractions, or CUDA OOMs. This is positive runtime evidence for the activation-event fix, but the PR remains draft pending code-owner review. Production Comet was not changed. |
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
63b022b to
43c03c7
Compare
Description
Problem
Worker registration activation changed each shared worker object from
PendingtoReadydirectly. That bypassed the registry event stream used by aggregate worker capacity and partition admission. Two 38-slot workers could therefore remain exposed as only 38 scheduler slots even though both were healthy and ready.Activation also has to fail closed if the workflow worker is replaced between registration and activation. URL or revision checks alone are insufficient because same-ID replacement shares mutable runtime state and its revision with the old
Arc.Solution
Resolve the registered worker ID by URL, then use
WorkerRegistry::apply_if_revisionandArc::ptr_eqinside the registry per-ID mutation lock. Only the exact worker instance registered by this workflow may transition toReady.The canonical
StatusChanged { Pending, Ready }event now wakes worker-capacity and partition-capacity consumers. Activation remains idempotent for the exact already-ready worker. Missing, removed, recreated, or replaced workers fail closed without activating the replacement or emitting a false event.Closes #56.
Changes
Test Plan
Local verification on exact
prodbase8cf8b1d0df752475bb561752c881ad7c3b6b5ca9:cargo test -p smg --lib --quiet: 1,499 passed, 5 ignored, 0 failed.cargo +nightly fmt --all -- --check.git diff --check.cargo clippy -p smg --lib --no-depscompleted with only the unchanged warning inmodel_gateway/src/middleware/concurrency.rs:213.The same-ID replacement regression is intentionally stronger than a missing-worker check: it proves a stale workflow cannot activate a replacement that shares the original worker runtime and revision.
Checklist
cargo clippy --all-targets --all-features -- -D warningspasses. The unchanged base still reports existing warnings outside this one-file diff.