Raft snapshot restore can delete recently-discovered instances across the entire cluster after a rolling restart / leader change

[ahmetsoguksu]

Bug Report

Orchestrator version

v4.31.1

Backend

SQLite (BackendDB: "sqlite"), one local .sqlite3 file per node (not shared). Raft is enabled for cross-node consensus/replication of commands (RaftEnabled: true), with a separate boltdb-backed raft log/snapshot store (RaftDataDir), independent from the SQLite backend.

Configuration

{
  "Debug": true,
  "ListenAddress": ":3000",
  "MySQLTopologyUser": "orchestrator",
  "MySQLTopologyPassword": "*****",
  "MySQLTopologySSLSkipVerify": true,
  "BackendDB": "sqlite",
  "SQLite3DataFile": "/path/to/orchestrator.sqlite3",
  "MySQLConnectTimeoutSeconds": 2,
  "MySQLConnectionLifetimeSeconds": 300,
  "DefaultInstancePort": 3307,
  "DiscoverByShowSlaveHosts": true,
  "InstancePollSeconds": 5,
  "UnseenInstanceForgetHours": 240,
  "InstanceBulkOperationsWaitTimeoutSeconds": 10,
  "HostnameResolveMethod": "none",
  "MySQLHostnameResolveMethod": "none",
  "ReasonableReplicationLagSeconds": 10,
  "AuthenticationMethod": "",
  "PowerAuthUsers": ["*"],
  "RecoverMasterClusterFilters": ["*"],
  "RecoverIntermediateMasterClusterFilters": ["*"],
  "FailureDetectionPeriodBlockMinutes": 1,
  "RecoveryPeriodBlockSeconds": 60,
  "ApplyMySQLPromotionAfterMasterFailover": true,
  "DetachLostSlavesAfterMasterFailover": true,
  "RaftEnabled": true,
  "RaftDataDir": "/path/to/raft-data",
  "RaftBind": "node1:10008",
  "DefaultRaftPort": 10008,
  "RaftNodes": ["node1:10008", "node2:10008", "node3:10008"],
  "ClusterNameToAlias": {
    "node2:3307": "customer-a",
    "node1:3308": "customer-b",
    "node1:3309": "customer-c",
    "node1:3310": "customer-d",
    "node2:3312": "customer-e"
  }
}

3-node raft cluster (node1, node2, node3), each running its own SQLite backend, quorum size 3.

Topology

Example affected cluster (customer-d, 1 master + 3 replicas), from /api/cluster/customer-d right after discovery, before the restart:

node1:3310  (master)              server_id=301  read_only=false
node1:3311  -> master node1:3310  server_id=304  replicating=true  read_only=false
node2:3310  -> master node1:3310  server_id=302  replicating=true  read_only=false
node3:3310  -> master node1:3310  server_id=303  replicating=true  read_only=false

Two other newly-provisioned clusters (customer-c: 1 master + 2 replicas, customer-e: 1 master + 1 replica) were affected identically in the same incident. Two long-lived, previously-existing clusters (customer-a, customer-b) on the same raft cluster were not affected.

Steps to reproduce

1. Stand up a 3-node orchestrator raft cluster with SQLite backend, quorum healthy, leader elected.
2. Discover a new MySQL topology (master + replica(s)) that has never been seen by this cluster before, e.g. via PUT /api/discover/<host>/<port> for each instance. Confirm via GET /api/clusters that the new cluster now appears on all nodes.
3. Shortly after (within the raft snapshot interval, i.e. before a new raft snapshot has necessarily been taken that includes the freshly discovered instances), perform a rolling restart of the orchestrator process on all 3 nodes (one at a time is sufficient; a leader change during the restart reproduces it reliably).
4. After all 3 nodes are back up, call GET /api/clusters on any node.

Expected behavior

The newly discovered cluster (all of its instances) should still be present in /api/clusters and /api/cluster/<alias> after the restart, identical to pre-restart state — no instance data should be lost purely due to a process restart / leader change when the underlying MySQL topology has not changed.

Actual behavior

All instances belonging to the newly-discovered clusters vanished from /api/clusters on every node simultaneously (9 instances across 3 clusters in our case). Long-lived, previously-existing clusters were unaffected. The underlying MySQL replication topology was completely healthy and untouched throughout — this is purely an orchestrator-side data loss, not a real topology change. Re-running /api/discover/<host>/<port> for each lost instance against the current leader immediately restored them, confirming no data was actually lost outside orchestrator's own bookkeeping.

Root cause (identified via code review):

• PUT /api/discover only replicates the discovery request through raft (go/http/api.go, orcraft.PublishCommand("discover", instanceKey)). Each node then independently connects to the target MySQL and writes the resulting instance row to its own local backend (go/logic/orchestrator.go DiscoverInstance → inst.WriteInstance). This write is a local side effect, not itself a raft-replicated command/state.
• A raft snapshot is created from whichever node is leader at snapshot time, by dumping that node's local backend (go/logic/snapshot_data.go, CreateSnapshotData → inst.ReadAllMinimalInstances()). It is a point-in-time, best-effort, non-authoritative view — not a synchronized replicated state machine of instance data.
• On process startup (and on InstallSnapshot), raft calls Restore() on the FSM (go/logic/snapshot_data.go), which reconciles the node's local database_instance table against the snapshot: instances present in the snapshot but missing locally are added, but — critically — instances present locally but absent from the snapshot are unconditionally deleted via inst.ForgetInstance, with no recency/age check whatsoever.
• If a snapshot was taken (or is being restored from disk on restart) before a freshly-discovered instance had propagated into it — which is entirely plausible given normal snapshot timing and buffered writes — Restore() treats the fresh instance as "should not exist" and deletes it. Because this restore logic runs independently on every node at startup, the same freshly-discovered instances get purged on all 3 nodes near-simultaneously, producing the observed cluster-wide, correlated disappearance.
• Older/long-lived instances are unaffected because they have long since been captured in every node's snapshot history.

We reviewed the git history of this logic (Restore()'s destructive branch was introduced in a 2017 commit) and found no comment, commit message, or design rationale anywhere that justifies deleting locally-known instances purely because they're missing from a point-in-time snapshot. We believe this is an unintentional bug in the snapshot-restore reconciliation, not a deliberate design choice — the deletion has no recency guard at all, whereas orchestrator already has a dedicated, properly age-gated mechanism for forgetting genuinely stale instances (ForgetLongUnseenInstances(), gated by UnseenInstanceForgetHours, default 240h) that runs independently of snapshot restore.

We have a minimal, low-risk proposed fix (guard the restore-time delete on the same UnseenInstanceForgetHours recency window already used by ForgetLongUnseenInstances, so restore's deletes become a strict subset of what that mechanism would remove anyway — no new config, no behavior change for genuinely stale/decommissioned instances) and intend to open a PR with it referencing this issue.

Logs

(orchestrator was run without --debug --stack during the incident; happy to reproduce with full verbosity and attach logs if useful — the sequence of events was reconstructed via /api/clusters polling and code review rather than log inspection)

Additional context

• OS: Linux (Debian-based lab VMs), MySQL variant: MariaDB.
• Mode: raft-enabled, 3-node quorum, SQLite backend on each node (not MySQL backend).
• This is specifically a raft + snapshot-restore interaction; we have not tested whether a MySQL-backend (shared DB) raft deployment is affected the same way, though the Restore() logic itself is backend-agnostic so we'd expect the same issue to reproduce there too, just with a shared DB reducing the visible divergence between nodes' pre-restart state (the destructive restore-vs-snapshot race would still apply).
• Impact is significant for any HA/failover use case (e.g. Kubernetes Operator managed clusters) where orchestrator is expected to be restarted (rolling upgrades, pod restarts, node maintenance) shortly after new topologies are onboarded — currently this can silently blind orchestrator to real, healthy replication topologies until they're manually re-discovered.
• We will follow up with a PR implementing the fix described above.

[coderabbitai]

🔗 Related PRs

#36 - Add built-in ProxySQL failover hooks [merged]
#60 - Fix pre-existing lint issues (partial) [closed]
#75 - Multi-cluster support: MySQL and PostgreSQL simultaneously [closed]
#99 - feat(postgresql): graceful primary switchover [closed]
#108 - fix: prevent false DeadMaster and drain MariaDB relay logs (#106) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!