CoMasterRecoveryMustPromoteOtherCoMaster: false has no effect when co-master is writable (active-active M-M deadlock)

[mailankitkm]

Summary

Setting CoMasterRecoveryMustPromoteOtherCoMaster: false is completely ineffective when the surviving co-master has read_only=OFF. Recovery always fails with:

RecoverDeadCoMaster: could not manage to promote other-co-master X; was only able to promote Y;
mustPromoteOtherCoMaster is true (either CoMasterRecoveryMustPromoteOtherCoMaster is true, or co-master
is writeable), therefore failing

Root Cause

In go/logic/topology_recovery.go, RecoverDeadCoMaster unconditionally overrides the config:

mustPromoteOtherCoMaster := config.Config.CoMasterRecoveryMustPromoteOtherCoMaster  // false
if !otherCoMaster.ReadOnly {
    AuditTopologyRecovery(...)
    mustPromoteOtherCoMaster = true  // unconditionally overrides the config setting!
}

Use Case: Active-Active M-M across two data centers

Site A (DC1):  master-a (writable) ↔ master-b (writable)  :Site B (DC2)
                   |                                              |
               replica-a (read-only)                       replica-b (read-only)

Config:

{
  "CoMasterRecoveryMustPromoteOtherCoMaster": false,
  "PreventCrossDataCenterMasterFailover": true
}

When master-a fails:

1. We want replica-a promoted (same DC).
2. CoMasterRecoveryMustPromoteOtherCoMaster: false should allow this.
3. BUT: master-b is writable → mustPromoteOtherCoMaster is forced to true.
4. PreventCrossDataCenterMasterFailover: true blocks cross-DC promotion.
5. Result: recovery always fails — a deadlock between the ReadOnly override and the cross-DC prevention.

Both masters must stay writable in active-active setups; setting read_only=ON on the co-master is not viable.

Fix
Respect the config as authority. Only set mustPromoteOtherCoMaster = true from the ReadOnly check when the config doesn't already say false:

mustPromoteOtherCoMaster := config.Config.CoMasterRecoveryMustPromoteOtherCoMaster
if !otherCoMaster.ReadOnly {
    if mustPromoteOtherCoMaster {
        AuditTopologyRecovery(topologyRecovery, fmt.Sprintf("RecoverDeadCoMaster: other co-master %+v is writeable hence has to be promoted", otherCoMaster.Key))
    } else {
        // CoMasterRecoveryMustPromoteOtherCoMaster=false: admin explicitly opted out.
        // Respect the config rather than forcing promotion.
        AuditTopologyRecovery(topologyRecovery, fmt.Sprintf("RecoverDeadCoMaster: other co-master %+v is writeable; CoMasterRecoveryMustPromoteOtherCoMaster=false overrides writeable check, will not force promotion", otherCoMaster.Key))
    }
}

When CoMasterRecoveryMustPromoteOtherCoMaster=true (the default): behavior is identical to before — no regression.

Reproduction

1. M-M replication: master-a ↔ master-b, replica-a replicates from master-a.
2. Both masters read_only=OFF (active-active).
3. Config: CoMasterRecoveryMustPromoteOtherCoMaster: false, PreventCrossDataCenterMasterFailover: true.
4. Kill master-a.
5. Observe: recovery fails with mustPromoteOtherCoMaster is true ... therefore failing.

Versions
Confirmed in v3.2.6 (openark) and v3.2.6-22 (percona fork).