Skip to content

fix: make failover safe under partitions - #452

Merged
kacy merged 4 commits into
mainfrom
fix/failover-safety
Nov 12, 2026
Merged

kacy merged 4 commits into
mainfrom
fix/failover-safety

Conversation

@kacy

@kacy kacy commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Fixes four cluster items from the audit, and adds the self-fencing primary we discussed after removing Raft.

Election quorum. Elections needed a majority of the other primaries. With two primaries, a replica cut off together with the second primary won with one vote, while its old primary kept serving on the other side of the partition. The quorum is now a majority of all primaries, the failed one included, as in Redis Cluster. A replica that can't reach that many voters logs that automatic failover needs more primaries, and doesn't start an election.

Two winners. A primary voted once per epoch, but each retry round uses a new epoch, so a second replica of the same failed primary could win a later round. A primary now votes for only one replica per failed primary for two failure timeouts. The same replica can still ask again after a round falls short. A candidate also stops its rounds once another node owns the failed primary's slots.

Fencing. A primary that can't reach a majority of the primaries, itself included, rejects writes with CLUSTERDOWN. Reads still work. Without the fence, a cut-off primary kept acknowledging writes that were lost once the majority promoted its replica.

Roles for nodes that join later. The Welcome reply had no replica field, and the sender always called itself a primary. A node that joined later counted replicas among the voters, which could make the quorum unreachable. MemberInfo now carries replicates, and the Welcome sender reports its real role (restored from nodes.conf on startup). The receiver emits RoleChanged for replicas it learns about.

Incarnation. Refuting a Dead claim set the incarnation to the claim's value plus one, which lowered it whenever the claim was stale. It now never goes down.

Tests

  • Gossip unit tests: a Welcome reports the sender's replica role, a joiner learns roles, and a stale Dead claim doesn't lower the incarnation.
  • Coordinator unit tests: the fence at 1/1, 3/3, 1/3 and 2/3 reachable primaries; one vote per failed primary, the same candidate re-asking, and a different failed primary.
  • New integration test: in a 3-primary cluster, killing two primaries makes the third reject writes with CLUSTERDOWN and keep serving reads.
  • The automatic failover integration test now has two slotless voters instead of one, because one primary plus one voter is exactly the unsafe two-primary setup.

Dropped during development: a check that refused votes when the failed primary owned no slots in the voter's view. A node that met the primary through its Join doesn't know its slots until their next exchange. The failover test caught a voter refusing for that reason. The per-candidate cooldown already prevents a second winner, as in Redis.

Performance: SET and GET in a single-node cluster are unchanged within noise (292k–300k against 288k–308k SET/s). Cluster mode runs at about 300k requests per second, against 870k+ standalone, on both main and this branch. That gap is separate, and I'll look at it next.

Docs: the production checklist now states that automatic failover needs at least three primaries, removes a claim that the most up-to-date replica wins (the stagger is fixed), and describes fencing. The troubleshooting section on split-brain is updated.

@kacy
kacy merged commit 2bee03d into main Nov 12, 2026
12 checks passed
@kacy
kacy deleted the fix/failover-safety branch November 12, 2026 23:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant