Skip to content

fix: let replicas keep up with the primary under write load - #455

Merged
kacy merged 5 commits into
mainfrom
fix/replication-throughput
Nov 14, 2026
Merged

kacy merged 5 commits into
mainfrom
fix/replication-throughput

Conversation

@kacy

@kacy kacy commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Under write load every replica fell into a resync loop. Measured with 1.26M keys loaded and 50 clients writing at a pipeline of 16, on one 32-core machine:

  • the replica applied one key per shard round trip and loaded the full sync in 13.5 s, at about 93k keys a second
  • meanwhile 9.5M writes passed through a broadcast channel that holds 65,536 events, so the primary sent MSG_RESYNC
  • the replica flushed its data and started another full sync, which overran in the same way. it ended with about half the keys

Changes:

  • per-replica queue: a forwarding task moves events from the broadcast channel into each replica's own queue, capped at 256 MiB (REPLICA_QUEUE_LIMIT). writes during a full sync wait there. past the cap the replica gets MSG_RESYNC at once, without the stale backlog. the writer flushes only when the queue is empty
  • one event per burst: shards collect records during a burst and send them as one ReplicationEvent, as the AOF does. every broadcast send locks the channel, which held the primary to about 600k writes a second with a replica attached. the wire protocol is unchanged
  • batched applies: the replica reads through a buffer, collects records into one batch per shard, sends each batch as one message, and acknowledges once it is applied. snapshot keys go the same way
  • WAIT: a new replica starts at offset 0 and acknowledges the snapshot offsets after loading them. WAIT counted it as soon as the snapshot was sent
  • INFO replication: connected_replicas counts the replicas streaming from the primary. it counted those gossip had reported, which read 0 while one streamed. a new sync_full field counts full syncs

Same load, before and after:

main this branch
full sync of 1.26M keys 13.5 s 1.0 s
full syncs during 30M writes 2+, replica left with ~half the keys 1, replica matches
primary SET rate with a replica, P=16 928k (replica mostly resyncing) 1.12M

A cluster without replicas is unchanged or slightly faster: 2.43M SET at P=16, against 2.28M.

The new integration test shrinks the broadcast channel to 1,024 events and writes 100,000 INCRs during a full sync. Against main's server it fails: the replica is missing the counter after WAIT returns 1.

@kacy
kacy merged commit f412ee7 into main Nov 14, 2026
12 checks passed
@kacy
kacy deleted the fix/replication-throughput branch November 14, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant