Skip to content

fix raft election safety defects from #155 - #158

Merged
fabracht merged 2 commits into
mainfrom
raft-election-safety
Oct 4, 2026
Merged

fabracht merged 2 commits into
mainfrom
raft-election-safety

Conversation

@fabracht

@fabracht fabracht commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes three of the four Raft election defects in #155.

  • A candidate that steps down to a same-term leader keeps its vote. Before, become_follower cleared voted_for on every step-down, so the node could grant a second vote in the same term.
  • The election timer starts on the first tick. Before, last_heartbeat_time = 0 against wall-clock milliseconds meant every node's timer was already expired at startup, so joiners campaigned within ~100 ms.
  • The first election timeout is randomized per node like every later one. With the timer starting correctly, a fixed 3000 ms first timeout would make nodes started together campaign together.
  • Startup grace applies again. The coordinator constructor called tick(0) to replay committed entries, which set startup_time = 0, so a node without peers elected itself immediately. It now uses take_committed().
  • Not fixed here: quorum is still computed over each node's own peer list, with no committed membership. Tracked in cluster raft: committed membership with joint-consensus changes #157. Starting every node with the full peer list remains the guard.

Full-mesh startup now waits one election timeout (3–5 s) before the first election, so settle time goes from 8–9 s to 11–14 s.

Test plan

  • cargo make clippy
  • cargo make test
  • New unit tests fail on the old code and pass now: same-term vote retention, timer start on first tick, distinct first timeouts, startup grace with storage
  • election_timeout_is_redrawn_for_every_election pins re-randomization on every election (review finding: an earlier revision of this branch dropped it)
  • 5-node partial mesh, 16 fresh starts: 16/16 settled in 8–10 s, one leader each (main: 2/16 never converged, two leaders; median settle 64 s)
  • 5-node full mesh, 16 fresh starts: 16/16 settled in 11–14 s, one leader each (main: 16/16, 8–9 s)
  • 5-node full mesh, simultaneous start, 12 runs: 12/12 settled in 13–14 s, one leader each
  • 5-node full mesh, kill the leader after the map settles, 10 runs interleaved with main: 10/10 elect one new leader in term 2 and resettle in 38–42 s (main: 10/10, 39–41 s)
  • 4-node full mesh, then a 5th node joins, 12 runs: 12/12, settles again about 29 s after the join (main: 12/12)
  • 5-node mqdb dev test ownership, sharing and presence: pass, twice each. --all: passes twice. Its constraints step is timing-dependent: it can hit the existing 503 "unique reservation not yet durable" window, which lasts 5–143 s on main

@fabracht
fabracht merged commit 958122e into main Oct 4, 2026
9 checks passed
@fabracht
fabracht deleted the raft-election-safety branch October 4, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant