Add workflow for v0.3.0

[ahrtr]

cc @nwnt @silentred

I will present this workflow in our next community meeting.

Some details (e.g. PV/PVC stuff) still need clarification. But this is just a high level workflow, it should be already good enough for now.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nwnt]

Just to put the comments here for the discussion during the meeting:

1. I think the reconciliation loop doesn't need to actively correct the ordinal, and instead should only be correcting when there's a need to replace the pod. The 'lazy' approach should in theory reduce the potential instability from replacing an existing pod, which might or might not be a leader.
2. The update config should be perhaps be done earlier than later (i.e. fix a failed member or create an additional pod when the expected count is higher). By doing so, I think it introduces less disruption by not having to go through pod creation and then later kill them because they're mismatched with the current content of EtcdCluster.
3. When the leader pod is about to be killed, we should also move the leader to the pod with the least possible ordinal as the higher ones are deemed to be killed first unless they are a leader:
   • In a 5 replica cluster, if the current leader is 3, then 4, 2, 1, 0 are killed and recreated first. The leadership would be transferred from 3 to 0, as 4 will be killed first and before 3 during a scale-in.
   • If the current leader is 0, it would still be killed last, but the leadership should then be moved to 1 and the operator will leave it there.

[silentred]

"failed member/POD" means members that cannot pass health check?

[ahrtr]

Basically yes

[silentred]

Fixing workflow seems to be a heavy work. It depends on cases, quorum hold vs not hold, data corrupted vs not corrupted.

[ahrtr]

Yes, we need to breakdown this when we work on the v0.3.0 roadmap tasks.

[silentred]

Currently, it is more like "ensuring replicas number". What is the difference with "Scale in & out" section?

Should we wait the pod, created in the previous loop and in the starting process, to be Ready and started in etcd cluster Members API?

[nwnt]

Comments from the community meeting:

1. Learner promotion maybe should be done at the fix failed member stage to avoid deadlocks from having multiple learners.
2. During exception handling that we need to remove pods, if the pod we're about to remove is the leader, we need to move it to other member first.
3. If the config of etcd cluster has been updated, the pod with highest ordinal will be replaced first and so on.

[silentred]

1. Learner promotion maybe should be done at the fix failed member stage to avoid deadlocks from having multiple learners.

A similar case in the meeting:
For a misconfigured 3-node cluster with one failed member pod, should we choose to fix the member or replace the old pod (user has updated EtcdCluster CR) first? New member may not be able join the cluster, due to the old wrong Spec.

[ahrtr]

Comments from the community meeting:

1. Learner promotion maybe should be done at the fix failed member stage to avoid deadlocks from having multiple learners.
2. During exception handling that we need to remove pods, if the pod we're about to remove is the leader, we need to move it to other member first.
3. If the config of etcd cluster has been updated, the pod with highest ordinal will be replaced first and so on.

Updated.

1. Learner promotion maybe should be done at the fix failed member stage to avoid deadlocks from having multiple learners.

A similar case in the meeting: For a misconfigured 3-node cluster with one failed member pod, should we choose to fix the member or replace the old pod (user has updated EtcdCluster CR) first? New member may not be able join the cluster, due to the old wrong Spec.

Yes

@nwnt @silentred this is just a high level workflow. There are two key points:

• We always try to fix any problems/issues first, before performing any normal operation.
• We need to ensure the workflow is idempotent

If there is no any objection, let's merge this for now. We can discuss more details separately when we work on the refactoring work and v0.3.0 roadmap tasks

[ahrtr]

cc @ArkaSaha30 @nwnt @silentred @ivanvc PTAL