Multi-controller probing with quorum-based failover decision

[superchee0526]

Right now only the master controller probes kvrocks nodes and decides failover. This can be fragile if the master has a partial view of the network

Proposal:

1. Let every controller probe all kvrocks nodes and write its probe results to the storage engine
2. The master controller periodically reads recent records (within a short window) and makes the decision using a quorum
3. There's a configuration to toggle between single-controller and multi-controller probing

Is this mechanism suitable to implement in a future version? If yes, I can draft a more detailed design/PR

[git-hulk]

@superchee0526 Thanks for raising this discussion.

Is this mechanism suitable to implement in a future version? If yes, I can draft a more detailed design/PR

Yes, we did plan to implement this, but we haven't started yet. And welcome to contribute.