HDDS-15874. Adding readinessProbe for helm HA deployments

[shuan1026]

What changes were proposed in this pull request?

This change adds a readinessProbe for HA OM/SCM deployments so pods are removed from Service endpoints until they have a stable Ratis role (LEADER or FOLLOWER).

The chart already has (or will have via [HDDS-14349]) an HTTP livenessProbe and a roles-based startupProbe. Startup answers “still initializing do not treat liveness failures as fatal.” Readiness answers “should this pod receive client traffic via Service endpoints now?”

A roles exec readinessProbe matches the [reviewer sample] and is stricter than UI httpGet: a process can respond on the UI port before OM/SCM has joined the HA ring in a follower/leader role.

In scope

• OM HA (replicas > 1): readinessProbe via ozone admin om roles, requiring LEADER|FOLLOWER on the line that matches $(hostname)
• SCM HA (replicas > 1): readinessProbe via ozone admin scm roles, with $(hostname -f) to match SCM roles output format
• Probe timings from kind testing: timeoutSeconds: 20 (sample 5 is too low while peers/DNS/Ratis are still forming)
• SCM HA headless DNS (if not already present): publishNotReadyAddresses: true on scm-service-headless when scm.replicas > 1 — same peer-DNS requirement as OM during bootstrap/readiness

Why add readiness (Things to confirm)

[HDDS-1668] warned against readiness on the OM HA example: Ready-gated endpoints can fight peer DNS needed during startup (OzoneManager.loadOMHAConfigs). That concern is may still valid if readiness is added without companions.

This PR follows the reviewer sample and addresses former by:

1. Relying on 14349 for startup + OM Parallel + OM headless publishNotReadyAddresses so peers are discoverable while pods are not Ready
2. Using readiness only to gate Service endpoints (traffic), not to block headless peer DNS once publishNotReadyAddresses is set

Trade-off: ozone admin * roles as a recurring readiness check has ongoing cost (fork JVM/CLI + RPC). Acceptable for HA correctness in this chart; a lighter httpGet readiness remains a possible follow-up if maintainers prefer.

Probe shape (illustrative; OM)

readinessProbe:
  exec:
    command:
      - sh
      - -c
      - |
        ozone admin om roles -id {{ .Values.clusterId }} 2>/dev/null \
          | grep $(hostname) | grep -qE 'LEADER|FOLLOWER' && exit 0 || exit 1
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 20
  failureThreshold: 3

SCM HA uses ozone admin scm roles with $(hostname -f) and the same LEADER|FOLLOWER grep.

Non-HA (replicas == 1) does not add roles exec readiness.

Startup ([HDDS-14349]) only checks membership in the roles list; readiness additionally requires LEADER|FOLLOWER.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15874

How was this patch tested?

1. kind — default install (OM HA)

On a fresh kind cluster:

helm upgrade --install ozone charts/ozone --wait --timeout 15m

Results:

• Release deployed; all pods Ready with no restarts
• During cold start, OM may show 1–2 Startup timed out after 20s events while peers/roles are not ready; within failureThreshold the probe succeeds and Ready is reached without repeated liveness Killing
• Green CI run and manually verified leader transfer and basic ozone commands

2. kind — SCM HA (scm.replicas=3, fresh install)

On a new release/cluster (do not scale an existing scm.replicas=1 in place):

helm upgrade --install ozone charts/ozone \
  --set scm.replicas=3 \
  --wait --timeout 20m

Same pass criteria: Ready, no restart storm.

Out of scope for this issue/PR: kubectl scale or helm upgrade --set scm.replicas=3 migrating an existing scm 1→3 deployment (known STS revision / addSCM issues; track separately).

[adoroszlai]

@dnskr please review if you have time

[chungen0126]

+1 LGTM