Skip to content

fix: make slot migration work end to end - #451

Merged
kacy merged 5 commits into
mainfrom
fix/migrate-resharding
Nov 12, 2026
Merged

kacy merged 5 commits into
mainfrom
fix/migrate-resharding

Conversation

@kacy

@kacy kacy commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Resharding didn't work. This fixes the audit's MIGRATE bug and the orphaned-keys bug, plus three more problems found along the way.

What was wrong

  • No ASKING: MIGRATE sent RESTORE straight to the target. The target doesn't own a slot it is importing, so it answered MOVED, and every MIGRATE of a key failed.
  • No AUTH: MIGRATE had no way to log in to a target with a password.
  • COPY ignored: COPY was parsed but the local key was deleted anyway.
  • Orphaned keys: the source sent ASK only for keys that MIGRATE had moved. A key created while the slot was migrating stayed on the source, and was left behind when the slot was reassigned.
  • CLI reshard: if a key failed to move, GETKEYSINSLOT listed it again forever. If the loop did end, the CLI still reassigned the slot that held the key.

What changed

  • MIGRATE pipelines [AUTH] ASKING RESTORE to the target and reads every reply. It takes AUTH password and AUTH2 username password, as Redis does. It honors COPY, and replies NOKEY for a missing key, also as in Redis.

  • check_slot_with_migration returns a SlotCheck enum. For a migrating slot it returns AskIfMissing, and the connection layer then asks the shards which of the command's keys still exist:

    • all of them: serve the command here;
    • none: send ASK;
    • some: send TRYAGAIN, as Redis Cluster does.

    This replaces the per-slot set of migrated keys, which grew with every key moved. The migration now keeps only a count.

  • ember-cli cluster reshard passes -a to MIGRATE as AUTH, treats NOKEY as moved, and stops with an error on a key it can't move, leaving that slot with the source.

Tests
New integration tests:

  • a slot moved by hand: a key moved by MIGRATE, a key still on the source, a key written during the move (ASK, then ASKING on the target), a multi-key command split across the nodes (TRYAGAIN), and SETSLOT NODE. Against main it fails at the first MIGRATE with MOVED.
  • MIGRATE ... COPY keeps the local key.
  • MIGRATE to a target with a password: NOAUTH without AUTH, and the key stays put; it succeeds with AUTH pw.
  • ember-cli cluster reshard moves 5 slots with 10 keys and leaves none behind. Against main, this loop never ends.

The coordinator's unit tests are updated for SlotCheck.

Not covered: MIGRATE's KEYS form for several keys, which redis-cli --cluster uses. Only ember-cli reshard is supported.

@kacy
kacy merged commit 1795955 into main Nov 12, 2026
12 checks passed
@kacy
kacy deleted the fix/migrate-resharding branch November 12, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant