Orchestrator graceful takeover fails when rpl_semi_sync_source_wait_for_replica_count is more than 1

[yunushaikh]

Hello Team,

We got one scenario where the failover gets stuck when rpl_semi_sync_source_wait_for_replica_count is more than 1.

Here is my topology
`bash-5.1$ orchestrator-client -c topology -a ps84-semisync
ps84-master:3306 (ps84-master) [0s,ok,8.4.7-7,rw,ROW,>>,GTID,semi:master]

• ps84-replica1:3306 (ps84-replica1) [0s,ok,8.4.7-7,ro,ROW,>>,GTID,semi:replica]
• ps84-replica2:3306 (ps84-replica2) [0s,ok,8.4.7-7,ro,ROW,>>,GTID,semi:replica]
• ps84-replica3:3306 (ps84-replica3) [0s,ok,8.4.7-7,ro,ROW,>>,GTID]`

My semi sync configuration on mysql
rpl_semi_sync_source_enabled=ON rpl_semi_sync_replica_enabled=ON rpl_semi_sync_source_wait_no_replica=ON rpl_semi_sync_source_timeout=1000000000000000000 rpl_semi_sync_source_wait_for_replica_count=2

And my orchestrator configuration
bash-5.1$ cat /etc/orchestrator/orchestrator.conf.json { "Debug": true, "ListenAddress": ":3000", "BackendDB": "sqlite", "SQLite3DataFile": "/var/lib/orchestrator/orc.db", "MySQLTopologyUser": "orchestrator", "MySQLTopologyPassword": "orcpass", "MySQLTopologyCredentialsConfigFile": "", "MySQLTopologySSLSkipVerify": true, "MySQLConnectTimeoutSeconds": 2, "DefaultInstancePort": 3306, "DiscoverByShowSlaveHosts": true, "InstancePollSeconds": 5, "UnseenInstanceForgetHours": 240, "HostnameResolveMethod": "none", "MySQLHostnameResolveMethod": "@@report_host", "ReasonableReplicationLagSeconds": 10, "CandidateInstanceExpireMinutes": 60, "RemoveTextFromHostnameDisplay": ":3306", "ReadOnly": false, "PowerAuthUsers": ["*"], "DetectClusterAliasQuery": "SELECT 'ps84-semisync'", "DetectInstanceAliasQuery": "SELECT @@report_host", "DetectSemiSyncEnforcedQuery": "select priority from meta.semi_sync where cluster_member = @@hostname", "LockedSemiSyncMaster": true, "MasterWithTooManySemiSyncReplicas": true, "RecoverLockedSemiSyncMaster": true, "EnforceExactSemiSyncReplicas": true, "StatusEndpoint": "/api/status", "StatusSimpleHealth": true, "RaftEnabled": true, "RaftBind": "127.0.0.1", "RaftAdvertise": "127.0.0.1", "RaftDataDir": "/var/lib/orchestrator", "RaftNodes": ["127.0.0.1"], "ApplyMySQLPromotionAfterMasterFailover": true, "DetachLostReplicasAfterMasterFailover": true, "UseSuperReadOnly": true, "RecoverMasterClusterFilters": [".*"], "RecoverIntermediateMasterClusterFilters": [".*"], "RecoveryPeriodBlockSeconds": 60, "FailMasterPromotionOnLagMinutes": 0, "ProcessesShellCommand": "sh", "OnFailureDetectionProcesses": [ "echo 'Detected {failureType} on {failureCluster}. Affected replicas: {countSlaves}' >> /tmp/recovery.log" ], "PreFailoverProcesses": [ "echo 'Will recover from {failureType} on {failureCluster}' >> /tmp/recovery.log" ], "PostFailoverProcesses": [ "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Successor: {successorHost}:{successorPort}' >> /tmp/recovery.log" ], "PostMasterFailoverProcesses": [ "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log" ] }

When I try to do a orchestrator failover, it gets stuck and give me later inavalid connection

bash-5.1$ orchestrator-client -c graceful-master-takeover -d ps84-replica1 -a ps84-semisync invalid connection

On master, we can see that orchestrator trying to set read_only to 1
| 11617 | root | localhost | NULL | Query | 1285 | Waiting for semi-sync ACK from replica | insert into test.t1 values (NULL) | 1285210 | 0 | 0 | | 11618 | orchestrator | 172.24.0.6:50696 | NULL | Query | 1285 | Waiting for global read lock | set global read_only = 1 | 1285116 | 0 | 0 | +-------+-----------------+--------

And topology stuck at
`ps84-master:3306 (ps84-master) [0s,ok,8.4.7-7,rw,ROW,>>,GTID,semi:master]

• ps84-replica1:3306 (ps84-replica1) [0s,ok,8.4.7-7,ro,ROW,>>,GTID,semi:replica]
  • ps84-replica2:3306 (ps84-replica2) [0s,ok,8.4.7-7,ro,ROW,>>,GTID,semi:replica]
  • ps84-replica3:3306 (ps84-replica3) [0s,ok,8.4.7-7,ro,ROW,>>,GTID]`

Since there is no more than one immediate replica, it gets stuck waiting for semi-sync ACK because rpl_semi_sync_source_wait_for_replica_count is set to 2

Let me know if any more information is needed.
Regards,
Yunus Shaikh.