Repository navigation
fix: make failover safe under partitions - #452
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes four cluster items from the audit, and adds the self-fencing primary we discussed after removing Raft.
Election quorum. Elections needed a majority of the other primaries. With two primaries, a replica cut off together with the second primary won with one vote, while its old primary kept serving on the other side of the partition. The quorum is now a majority of all primaries, the failed one included, as in Redis Cluster. A replica that can't reach that many voters logs that automatic failover needs more primaries, and doesn't start an election.
Two winners. A primary voted once per epoch, but each retry round uses a new epoch, so a second replica of the same failed primary could win a later round. A primary now votes for only one replica per failed primary for two failure timeouts. The same replica can still ask again after a round falls short. A candidate also stops its rounds once another node owns the failed primary's slots.
Fencing. A primary that can't reach a majority of the primaries, itself included, rejects writes with
CLUSTERDOWN. Reads still work. Without the fence, a cut-off primary kept acknowledging writes that were lost once the majority promoted its replica.Roles for nodes that join later. The Welcome reply had no replica field, and the sender always called itself a primary. A node that joined later counted replicas among the voters, which could make the quorum unreachable.
MemberInfonow carriesreplicates, and the Welcome sender reports its real role (restored fromnodes.confon startup). The receiver emitsRoleChangedfor replicas it learns about.Incarnation. Refuting a Dead claim set the incarnation to the claim's value plus one, which lowered it whenever the claim was stale. It now never goes down.
Tests
CLUSTERDOWNand keep serving reads.Dropped during development: a check that refused votes when the failed primary owned no slots in the voter's view. A node that met the primary through its Join doesn't know its slots until their next exchange. The failover test caught a voter refusing for that reason. The per-candidate cooldown already prevents a second winner, as in Redis.
Performance: SET and GET in a single-node cluster are unchanged within noise (292k–300k against 288k–308k SET/s). Cluster mode runs at about 300k requests per second, against 870k+ standalone, on both main and this branch. That gap is separate, and I'll look at it next.
Docs: the production checklist now states that automatic failover needs at least three primaries, removes a claim that the most up-to-date replica wins (the stagger is fixed), and describes fencing. The troubleshooting section on split-brain is updated.