Fix nsx-ncp liveness probe broken by nsx-ujo change 823215 - #320
Merged
Conversation
ggverma
force-pushed
the
fix/ncp-healthz-liveness-probe
branch
from
August 6, 2026 16:31
ecd6284 to
36d8584
Compare
NCP's liveness probe mechanism changed from an exec-based external script to an in-process httpGet probe served on :8086/healthz. The old exec probe command is no longer supported and now always fails, causing kubelet to restart-loop the nsx-ncp pod on every platform this operator manages. Switch the exec probe to httpGet on 127.0.0.1:8086/healthz (all three pods run hostNetwork: true, so kubelet and NCP share the host network namespace and loopback is reachable), keeping the same initialDelaySeconds/timeoutSeconds/periodSeconds/failureThreshold as before. Deliberately not declaring a containerPort for 8086: on a hostNetwork: true pod, declaring it makes the scheduler track that port as an exclusive per-node reservation. A rolling update needs two new-template replicas to be schedulable before any old replica is removed, and two pods can never simultaneously hold the same exclusive hostPort on one node -- when available nodes drop to or below the replica count, this deadlocks the rollout forever. E2E CI caught this directly: a single-node Kind cluster scaled to 2 replicas got stuck with both new-template pods Pending indefinitely, and the NcpInstall CRD never reported Available. The declaration isn't needed for the probe to work either -- hostNetwork means the process binds :8086 regardless, and kubelet's httpGet probe reaches it via host/port directly, not via the pod's ports field. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ggverma
force-pushed
the
fix/ncp-healthz-liveness-probe
branch
from
August 6, 2026 16:37
36d8584 to
08972e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
:8086/healthz, and removed the"nsx-ncp"handler fromcheck_pod_liveness's dispatch table entirely.ncp-openshift4.yaml,ncp-rhel.yaml,ncp-ubuntu.yaml) still invoke the removed exec probe (check_pod_liveness nsx-ncp 30). Since"nsx-ncp"is no longer a recognized target, the script falls through tonull_handler, which always reports failure -- so the liveness probe now always fails and kubelet restart-loops the NCP pod on every platform this operator manages (OpenShift, RHEL, Ubuntu Kubernetes).containerPort: 8086,host: 127.0.0.1(all three pods already runhostNetwork: true, so kubelet and NCP share the host network namespace and loopback is reachable), preserving the existinginitialDelaySeconds/timeoutSeconds/periodSeconds/failureThreshold.