HDDS-14349. Adding startupProbe for helm HA deployments

[shuan1026]

What changes were proposed in this pull request?

This change adds a startupProbe for HA OM/SCM deployments so longer initialization is not treated as a liveness failure and does not restart the container.

The chart currently relies on an HTTP livenessProbe against the UI port. Under HA (replicas > 1), OM/SCM startup can take longer; a short liveness window can kill the process while it is still bootstrapping.

A startupProbe addresses that failure mode: until startup succeeds, kubelet does not apply liveness-based restarts for probe failures.

In scope

• OM HA (replicas > 1): startupProbe via ozone admin om roles, checking that the local hostname appears in the roles output
• SCM HA: startupProbe via ozone admin scm roles (same idea; hostname match follows SCM roles format)
• OM HA bootstrap wiring (required for roles startup to work):
  • podManagementPolicy: Parallel on the OM StatefulSet — avoids OrderedReady stalling at only om-0 while startup waits for peers
  • publishNotReadyAddresses: true on the OM headless Service — peer DNS must exist before startup passes; default headless only publishes Ready pods, and roles startup keeps pods NotReady until the probe succeeds
• Probe timings from kind testing: timeoutSeconds must be larger than the sample value of 5, because ozone admin * roles often blocks while peers / DNS / Ratis are not ready yet

Out of scope

• readinessProbe (moved to [HDDS-15874])

Why readiness is not in this PR

The Jira issue and [reviewer sample]) included both startupProbe and readinessProbe. After kind testing, readiness was split to [HDDS-15874]; this PR delivers startup only.

1. Different problem, different tool
   The pain is HA deployments take longer to initialize → avoid premature liveness restarts → startupProbe. A readinessProbe does not fix that; it controls Service endpoint membership, not “do not kill me while starting”.

2. Historical concern for quorum OM may still applies
   [HDDS-1668] suggested skipping readiness on the OM HA: Ready-gated Endpoints can fight peer DNS needed at OM HA startup (OzoneManager.loadOMHAConfigs).

3. Cost of roles-based readiness on kind
   Using ozone admin * roles as a recurring readiness probe has ongoing cost (fork JVM/CLI + RPC). That is acceptable for a bounded startup window; as a lifelong readiness check the cost is amplified.

Follow-up

• [HDDS-15874]: whether/how to add readiness (roles exec vs light httpGet, or document deferral).

Probe shape (illustrative; OM)

startupProbe:
  exec:
    command:
      - sh
      - -c
      - |
        ozone admin om roles -id {{ .Values.clusterId }} 2>/dev/null \
          | grep -q $(hostname) && exit 0 || exit 1
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 20          # sample value 5 is too low in practice
  failureThreshold: 30        # multi-minute bootstrap window

SCM HA uses ozone admin scm roles with a hostname form that matches SCM roles output ($(hostname -f) in testing).

Non-HA (replicas == 1) does not add a roles exec probe, to avoid assuming multi-node Ratis roles output.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14349

How was this patch tested?

1. kind — default install (OM HA)

On a fresh kind cluster:

helm upgrade --install ozone charts/ozone --wait --timeout 15m

Results:

• Release deployed; all pods Ready with no restarts
• During cold start, OM may show 1–2 Startup timed out after 20s events while peers/roles are not ready; within failureThreshold the probe succeeds and Ready is reached without repeated liveness Killing
• After Ready: time ozone admin om roles -id cluster1 is about 2s (below timeoutSeconds: 20)
• Green CI run and manually verified leader transfer and basic ozone commands

2. kind — SCM HA (scm.replicas=3, fresh install)

On a new release/cluster (do not scale an existing scm.replicas=1 in place):

helm upgrade --install ozone charts/ozone \
  --set scm.replicas=3 \
  --wait --timeout 20m

Same pass criteria: Ready, no restart storm.

Out of scope for this issue/PR: kubectl scale or helm upgrade --set scm.replicas=3 migrating an existing scm 1→3 deployment (known STS revision / addSCM issues; track separately).

[shuan1026]

PTAL @adoroszlai Thank you!

[adoroszlai]

@dnskr please review if you have time

[chungen0126]

Thanks @shuan1026 for working on this. I think we should also check if SCM leaves safe mode.

[shuan1026]

Thanks for the suggestion! I've updated the SCM startupProbe to also check that SCM has left safe mode, and increased failureThreshold so the probe has enough time before the pod is restarted.