Skip to content

Filter stale worker addresses during cleanup via scheduler-authoritative data#1049

Open
garvit-gupta wants to merge 1 commit into
ArroyoSystems:masterfrom
garvit-gupta:garvit-gupta/pipe-588-cleanup-via-scheduler-placements
Open

Filter stale worker addresses during cleanup via scheduler-authoritative data#1049
garvit-gupta wants to merge 1 commit into
ArroyoSystems:masterfrom
garvit-gupta:garvit-gupta/pipe-588-cleanup-via-scheduler-placements

Conversation

@garvit-gupta
Copy link
Copy Markdown
Collaborator

@garvit-gupta garvit-gupta commented Apr 28, 2026

Context

Follow-up to #1045. Cleanup paths (Recovering, Failing, Stopping) can iterate stale gRPC channels targeting addresses recycled to other workers. While #1045 prevents misrouted messages at the receiver, this change prevents them at the sender by filtering the in-memory worker map against the scheduler's authoritative view before sending.

Changes

  • Add rpc_address tracking to WorkerStatus so cached channels can be matched against live addresses.
  • Add worker_addresses_for_job to the Scheduler trait (default None; opt-in for schedulers with address recycling).
  • Before cleanup graceful stops, filter the worker map against the scheduler's response; skip entries whose addresses are no longer live.
  • Clear stale handles explicitly at every state transition where workers are known to be gone.
  • Short-circuit wait_for_finish when the worker map is empty.

Follows: #1045 (receiver-side worker identity validation).

@garvit-gupta garvit-gupta force-pushed the garvit-gupta/pipe-588-cleanup-via-scheduler-placements branch from 0384f12 to 03e87d0 Compare April 29, 2026 14:47
@garvit-gupta garvit-gupta changed the title Use scheduler-authoritative addresses for cleanup graceful stop Filter stale worker addresses during cleanup via scheduler-authoritative data Apr 29, 2026
@garvit-gupta garvit-gupta force-pushed the garvit-gupta/pipe-588-cleanup-via-scheduler-placements branch from 03e87d0 to ad4c5db Compare April 29, 2026 15:43
@garvit-gupta garvit-gupta force-pushed the garvit-gupta/pipe-588-cleanup-via-scheduler-placements branch from ad4c5db to 244a617 Compare May 4, 2026 19:00
@garvit-gupta garvit-gupta marked this pull request as ready for review May 4, 2026 19:04
@garvit-gupta garvit-gupta requested review from QnJ1c2kNCg and mwylde and removed request for mwylde May 4, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant