Skip to content

fix: authenticate both ends of replication and time out stalled peers - #456

Merged
kacy merged 3 commits into
mainfrom
fix/replication-hardening
Nov 14, 2026
Merged

kacy merged 3 commits into
mainfrom
fix/replication-hardening

Conversation

@kacy

@kacy kacy commented Nov 14, 2026

Copy link
Copy Markdown
Owner

The remaining replication items from the audit:

  • mutual authentication: only the primary checked the replica. A replica took data from anything that answered at the primary's address, after flushing its own. Now the replica sends a nonce too, and checks the primary's tag before it flushes anything. Each side signs the other's nonce after a label for its role, so one side's answer can't be relayed as the other's. Protocol version 3
  • handshake timeout and cap: the primary drops a connection that hasn't finished the handshake within 10 s, and runs at most 16 handshakes at once. A silent connection held a task forever
  • idle timeout and heartbeat: both sides wrap the socket so a read or write that makes no progress for 60 s fails, as Redis's repl-timeout. The primary sends MSG_PING after 10 s with nothing to send, and the replica answers, so a healthy idle connection stays open in both directions
  • lengths: snapshot and record bodies are read with take(len).read_to_end, so the buffer grows as bytes arrive instead of allocating the header's length, up to 4 GiB, before reading anything
  • shard mismatch: the primary flushes its status byte before closing. It stayed in the write buffer, so the replica saw only a closed connection

Tests:

  • each side rejects the other without the shared secret
  • the primary closes a silent connection after the handshake timeout
  • a replica drops a primary that goes quiet
  • heartbeats keep an idle connection open past the idle timeout. This test fails with the heartbeat interval set above the timeout

The three timing durations are shorter under cfg(test) so these run in real time. Paused tokio time gave false timeouts with real sockets.

Throughput with a replica attached is unchanged: 1.25M SET/s at P=16, against 1.12M before, within noise.

Only the handshake is authenticated. Records after it carry no MAC and the stream isn't encrypted. The production checklist now says to keep replication traffic on a trusted network.

@kacy
kacy merged commit be09858 into main Nov 14, 2026
12 checks passed
@kacy
kacy deleted the fix/replication-hardening branch November 14, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant