Skip to content

cluster: redial robustness follow-ups (latency, suspect-churn, per-dial timeout, address gossip) #149

Description

@fabracht

Non-blocking robustness follow-ups from the redial half (#146 / PR #148), on the #140 mesh/reconnection track.

  • reconnection latency: a crashed peer leaves alive_nodes after the ~15s heartbeat timeout and is re-dialled only on the 60s mesh-check tick, so worst-case reconnection is ~75s. consider driving a re-dial directly off death-detection, or a shorter/adaptive tick.
  • suspect-window churn: redial_unlinked keys on strict Alive, so a peer briefly in Suspected (healthy but ~7.5s late) is eligible for re-dial, which replaces a working connection with an equivalent fresh one. bounded (once per 60s tick) and safe, but could gate on Dead/Unknown only.
  • hard-crash removal latency: remove_peer_generation fires on a send-side write_all error, which for a hard crash (no clean close) waits on quinn's ~30s idle timeout. removal is not load-bearing for reconnection (redial keys on liveness and connect_to_peer replaces stale entries), but the map/direct_peers view lags for that window.
  • no per-dial timeout: connect_to_peer has no dial timeout and redial_unlinked dials sequentially, so a black-hole peer stalls that tick's remaining redials up to quinn's handshake timeout (the redial runs detached, so the event loop is unaffected).
  • address-change handling: peer_addrs is populated only from --peers; a peer that restarts at a new address cannot be re-dialled to it. proper discovery needs address gossip, which is cluster: nodes never dial peers they discover, never retry failed dials, and never drop dead peers #140.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions