Skip to content

speed up and harden cluster peer redial - #150

Merged
fabracht merged 1 commit into
mainfrom
fix/149-redial-robustness
Sep 22, 2026
Merged

fabracht merged 1 commit into
mainfrom
fix/149-redial-robustness

Conversation

@fabracht

@fabracht fabracht commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • redial the mesh sooner and more robustly (follow-ups from redial disconnected cluster peers and remove dead ones #148/cluster: congestion-drop reconciliation and stale-peer redial after writer-task death #146, tracked in cluster: redial robustness follow-ups (latency, suspect-churn, per-dial timeout, address gossip) #149)
  • drive a re-dial off death-detection, not just the 60s mesh tick: a peer that is reachable at death-detection time (a transient link loss, or a peer already back up) reconnects in ~15s (the heartbeat dead-detect) instead of waiting up to the 60s tick. a peer that stays down longer still recovers on the next mesh tick.
  • gate redial on Dead/Unknown peers, skipping the transient Suspected window, so a briefly-late but still-live peer is not churned. tradeoff: up to (dead - suspect) extra latency before redialing a link that is broken but still only Suspected.
  • dial peers concurrently with a per-dial timeout, so one unreachable (black-hole) peer cannot serialize reconnection of the others (a per-dial timeout alone did not fix this: a sequential dial of a non-listening peer blocks for the full timeout and makes the real peer miss the one-shot death-trigger window, falling back to the 60s tick).
  • no model change: this refines the already-modelled redial. guarded ABA-safe removal and convergence are unchanged; Dead/Unknown gating is a strict subset of the modelled trigger. concurrent same-peer dials can leave the surviving peer-map entry with a non-maximal generation, which is safe because removal is keyed on exact-generation equality with a monotonic in-memory counter, so no live connection is ever dropped.

Test plan

  • cargo make clippy (pedantic, all feature combos)
  • cargo make test (1203) + new alive_or_suspected_excludes_dead_and_unknown unit test
  • existing dead_peer_is_removed_and_redialled integration test
  • live 5-node E2E: immediate accept-only restart heals in ~15-17s (down from ~75s), consistent across victims 2/3/5
  • live 5-node E2E with a permanent black-hole peer configured: healing stays ~16-17s (concurrent dials); a per-dial timeout alone left an ~86s stall
  • 3-agent quorum review

Addresses #149 (latency, suspect-window churn, black-hole stall). Address gossip / discovery of non-configured peers remains #140.

@fabracht
fabracht merged commit 5249b85 into main Sep 22, 2026
9 checks passed
@fabracht
fabracht deleted the fix/149-redial-robustness branch September 22, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant