OCP 4.21.9 NNO setup fails due to DTK incompability

[thameem-abbas]

What happened: On a cluster with NNO installed, when upgrading to OCP 4.21.9, the mofed driver build fails

What you expected to happen: Build and run fine

How to reproduce it (as minimally and precisely as possible): OCP 4.21.9 with NNO Operator 26.1.0

Anything else we need to know?:

Logs:

• NicClusterPolicy CR spec and state:
• -Spec:

spec:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node-role.kubernetes.io/worker
              operator: Exists
  nvIpam:
    enableWebhook: false
    image: nvidia-k8s-ipam
    imagePullSecrets: []
    repository: nvcr.io/nvidia/mellanox
    version: network-operator-v26.1.0
  ofedDriver:
    imagePullSecrets: []
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 30
    forcePrecompiled: false
    terminationGracePeriodSeconds: 300
    repository: nvcr.io/nvidia/mellanox
    livenessProbe:
      initialDelaySeconds: 30
      periodSeconds: 30
    env:
      - name: UNLOAD_STORAGE_MODULES
        value: 'true'
    upgradePolicy:
      autoUpgrade: true
      drain:
        deleteEmptyDir: true
        enable: true
        force: true
        podSelector: ''
        timeoutSeconds: 300
      maxParallelUpgrades: 2
      safeLoad: false
    startupProbe:
      initialDelaySeconds: 10
      periodSeconds: 20
    version: doca3.3.0-26.01-1.0.0.0-0
    image: doca-driver
  rdmaSharedDevicePlugin:
    config: '{"configList": [{"resourceName": "ens31f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens31f0np0"]}}, {"resourceName": "ens32f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens32f0np0"]}}, {"resourceName": "ens34f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens34f0np0"]}}, {"resourceName": "ens35f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens35f0np0"]}}, {"resourceName": "ens36f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens36f0np0"]}}, {"resourceName": "ens37f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens37f0np0"]}}, {"resourceName": "ens38f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens38f0np0"]}}, {"resourceName": "ens40f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens40f0np0"]}}, {"resourceName": "ens41f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens41f0np0"]}}, {"resourceName": "ens42f0np0_roce", "rdmaHcaMax": 160, "selectors": {"vendors": ["15b3"], "ifNames": ["ens42f0np0"]}}, {"resourceName": "shared_roce", "rdmaHcaMax": 100, "selectors": {"vendors": ["15b3"], "ifNames": ["ens31f0np0", "ens32f0np0", "ens34f0np0", "ens35f0np0", "ens36f0np0", "ens37f0np0", "ens38f0np0", "ens40f0np0", "ens41f0np0", "ens42f0np0"]}}]}'
    image: k8s-rdma-shared-dev-plugin
    imagePullSecrets: []
    repository: nvcr.io/nvidia/cloud-native
    version: v1.5.2

• Output of: kubectl -n nvidia-network-operator get -A:
• Network Operator version: 26.1.0
• Logs of Network Operator controller:
• Logs of the various Pods in nvidia-network-operator namespace:
• Helm Configuration (if applicable):
• Kubernetes' nodes information (labels, annotations and status): kubectl get node -o yaml:

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      k8s.ovn.org/host-cidrs: '["<REDACTED_IP>/24"]'
      k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","bridge-id":"br-ex","interface-id":"br-ex_<REDACTED_HOSTNAME>","mac-address":"<REDACTED_MAC>","ip-addresses":["<REDACTED_IP>/24"],"ip-address":"<REDACTED_IP>/24","next-hops":["<REDACTED_IP>"],"next-hop":"<REDACTED_IP>","node-port-enable":"true","vlan-id":"0"}}'
      k8s.ovn.org/layer2-topology-version: "2.0"
      k8s.ovn.org/node-chassis-id: <REDACTED>
      k8s.ovn.org/node-encap-ips: '["<REDACTED_IP>"]'
      k8s.ovn.org/node-id: "3"
      k8s.ovn.org/node-masquerade-subnet: '{"ipv4":"169.254.0.0/17","ipv6":"fd69::/112"}'
      k8s.ovn.org/node-primary-ifaddr: '{"ipv4":"<REDACTED_IP>/24"}'
      k8s.ovn.org/node-subnets: '{"default":["10.129.0.0/23"]}'
      k8s.ovn.org/node-transit-switch-port-ifaddr: '{"ipv4":"100.88.0.3/16"}'
      k8s.ovn.org/remote-zone-migrated: <REDACTED_HOSTNAME>
      k8s.ovn.org/zone-name: <REDACTED_HOSTNAME>
      machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
      machineconfiguration.openshift.io/currentConfig: rendered-worker-9c38ada62137adce550f4c98efd48dd2
      machineconfiguration.openshift.io/desiredConfig: rendered-worker-9c38ada62137adce550f4c98efd48dd2
      machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-worker-9c38ada62137adce550f4c98efd48dd2
      machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-worker-9c38ada62137adce550f4c98efd48dd2
      machineconfiguration.openshift.io/lastObservedServerCAAnnotation: "false"
      machineconfiguration.openshift.io/lastSyncedControllerConfigResourceVersion: "230864234"
      machineconfiguration.openshift.io/post-config-action: ""
      machineconfiguration.openshift.io/reason: ""
      machineconfiguration.openshift.io/state: Done
      nfd.node.kubernetes.io/feature-labels: cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AMXBF16,cpu-cpuid.AMXFP8,cpu-cpuid.AMXINT8,cpu-cpuid.AMXTILE,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.AVX512BF16,cpu-cpuid.AVX512BITALG,cpu-cpuid.AVX512BW,cpu-cpuid.AVX512CD,cpu-cpuid.AVX512DQ,cpu-cpuid.AVX512F,cpu-cpuid.AVX512FP16,cpu-cpuid.AVX512IFMA,cpu-cpuid.AVX512VBMI,cpu-cpuid.AVX512VBMI2,cpu-cpuid.AVX512VL,cpu-cpuid.AVX512VNNI,cpu-cpuid.AVX512VPOPCNTDQ,cpu-cpuid.AVXVNNI,cpu-cpuid.BHI_CTRL,cpu-cpuid.CETIBT,cpu-cpuid.CETSS,cpu-cpuid.CLDEMOTE,cpu-cpuid.CMPSB_SCADBS_SHORT,cpu-cpuid.CMPXCHG8,cpu-cpuid.ENQCMD,cpu-cpuid.FLUSH_L1D,cpu-cpuid.FMA3,cpu-cpuid.FSRM,cpu-cpuid.FXSR,cpu-cpuid.FXSROPT,cpu-cpuid.GFNI,cpu-cpuid.IA32_ARCH_CAP,cpu-cpuid.IA32_CORE_CAP,cpu-cpuid.IBPB,cpu-cpuid.IDPRED_CTRL,cpu-cpuid.LAHF,cpu-cpuid.MD_CLEAR,cpu-cpuid.MOVBE,cpu-cpuid.MOVDIR64B,cpu-cpuid.MOVDIRI,cpu-cpuid.MOVSB_ZL,cpu-cpuid.OSXSAVE,cpu-cpuid.PCONFIG,cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES,cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS,cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES,cpu-cpuid.PMU_FIXEDCOUNTER_TOPDOWN_SLOTS,cpu-cpuid.PSFD,cpu-cpuid.RRSBA_CTRL,cpu-cpuid.SERIALIZE,cpu-cpuid.SHA,cpu-cpuid.SPEC_CTRL_SSBD,cpu-cpuid.STIBP,cpu-cpuid.STOSB_SHORT,cpu-cpuid.SYSCALL,cpu-cpuid.SYSEE,cpu-cpuid.TME,cpu-cpuid.TSXLDTRK,cpu-cpuid.VAES,cpu-cpuid.VMX,cpu-cpuid.VPCLMULQDQ,cpu-cpuid.WAITPKG,cpu-cpuid.WBNOINVD,cpu-cpuid.X87,cpu-cpuid.XGETBV1,cpu-cpuid.XSAVE,cpu-cpuid.XSAVEC,cpu-cpuid.XSAVEOPT,cpu-cpuid.XSAVES,cpu-cstate.enabled,cpu-hardware_multithreading,cpu-model.family,cpu-model.id,cpu-model.vendor_id,cpu-pstate.status,cpu-pstate.turbo,kernel-config.NO_HZ,kernel-config.NO_HZ_FULL,kernel-selinux.enabled,kernel-version.full,kernel-version.major,kernel-version.minor,kernel-version.revision,memory-numa,network-sriov.capable,nvidia.com/cuda.driver-version.full,nvidia.com/cuda.driver-version.major,nvidia.com/cuda.driver-version.minor,nvidia.com/cuda.driver-version.revision,nvidia.com/cuda.driver.major,nvidia.com/cuda.driver.minor,nvidia.com/cuda.driver.rev,nvidia.com/cuda.runtime-version.full,nvidia.com/cuda.runtime-version.major,nvidia.com/cuda.runtime-version.minor,nvidia.com/cuda.runtime.major,nvidia.com/cuda.runtime.minor,nvidia.com/gfd.timestamp,nvidia.com/gpu.compute.major,nvidia.com/gpu.compute.minor,nvidia.com/gpu.count,nvidia.com/gpu.family,nvidia.com/gpu.machine,nvidia.com/gpu.memory,nvidia.com/gpu.mode,nvidia.com/gpu.product,nvidia.com/gpu.replicas,nvidia.com/gpu.sharing-strategy,nvidia.com/mig.capable,nvidia.com/mps.capable,nvidia.com/vgpu.present,pci-102b.present,pci-10de.present,pci-10de.sriov.capable,pci-14e4.present,pci-15b3.present,pci-15b3.sriov.capable,rdma.available,rdma.capable,storage-nonrotationaldisk,system-os_release.ID,system-os_release.OPENSHIFT_VERSION,system-os_release.OSTREE_VERSION,system-os_release.VERSION_ID,system-os_release.VERSION_ID.major,system-os_release.VERSION_ID.minor
      nvidia.com/gpu-driver-upgrade-enabled: "true"
      sriovnetwork.openshift.io/state: Idle
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2026-02-25T17:58:40Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      feature.node.kubernetes.io/cpu-cpuid.ADX: "true"
      feature.node.kubernetes.io/cpu-cpuid.AESNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXBF16: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXFP8: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXINT8: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXTILE: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX2: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512BF16: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512BITALG: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512BW: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512CD: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512DQ: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512F: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512FP16: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512IFMA: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI2: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VL: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VPOPCNTDQ: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVXVNNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.BHI_CTRL: "true"
      feature.node.kubernetes.io/cpu-cpuid.CETIBT: "true"
      feature.node.kubernetes.io/cpu-cpuid.CETSS: "true"
      feature.node.kubernetes.io/cpu-cpuid.CLDEMOTE: "true"
      feature.node.kubernetes.io/cpu-cpuid.CMPSB_SCADBS_SHORT: "true"
      feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8: "true"
      feature.node.kubernetes.io/cpu-cpuid.ENQCMD: "true"
      feature.node.kubernetes.io/cpu-cpuid.FLUSH_L1D: "true"
      feature.node.kubernetes.io/cpu-cpuid.FMA3: "true"
      feature.node.kubernetes.io/cpu-cpuid.FSRM: "true"
      feature.node.kubernetes.io/cpu-cpuid.FXSR: "true"
      feature.node.kubernetes.io/cpu-cpuid.FXSROPT: "true"
      feature.node.kubernetes.io/cpu-cpuid.GFNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.IA32_ARCH_CAP: "true"
      feature.node.kubernetes.io/cpu-cpuid.IA32_CORE_CAP: "true"
      feature.node.kubernetes.io/cpu-cpuid.IBPB: "true"
      feature.node.kubernetes.io/cpu-cpuid.IDPRED_CTRL: "true"
      feature.node.kubernetes.io/cpu-cpuid.LAHF: "true"
      feature.node.kubernetes.io/cpu-cpuid.MD_CLEAR: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVBE: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVDIR64B: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVDIRI: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVSB_ZL: "true"
      feature.node.kubernetes.io/cpu-cpuid.OSXSAVE: "true"
      feature.node.kubernetes.io/cpu-cpuid.PCONFIG: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_TOPDOWN_SLOTS: "true"
      feature.node.kubernetes.io/cpu-cpuid.PSFD: "true"
      feature.node.kubernetes.io/cpu-cpuid.RRSBA_CTRL: "true"
      feature.node.kubernetes.io/cpu-cpuid.SERIALIZE: "true"
      feature.node.kubernetes.io/cpu-cpuid.SHA: "true"
      feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD: "true"
      feature.node.kubernetes.io/cpu-cpuid.STIBP: "true"
      feature.node.kubernetes.io/cpu-cpuid.STOSB_SHORT: "true"
      feature.node.kubernetes.io/cpu-cpuid.SYSCALL: "true"
      feature.node.kubernetes.io/cpu-cpuid.SYSEE: "true"
      feature.node.kubernetes.io/cpu-cpuid.TME: "true"
      feature.node.kubernetes.io/cpu-cpuid.TSXLDTRK: "true"
      feature.node.kubernetes.io/cpu-cpuid.VAES: "true"
      feature.node.kubernetes.io/cpu-cpuid.VMX: "true"
      feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ: "true"
      feature.node.kubernetes.io/cpu-cpuid.WAITPKG: "true"
      feature.node.kubernetes.io/cpu-cpuid.WBNOINVD: "true"
      feature.node.kubernetes.io/cpu-cpuid.X87: "true"
      feature.node.kubernetes.io/cpu-cpuid.XGETBV1: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVE: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVEC: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVES: "true"
      feature.node.kubernetes.io/cpu-cstate.enabled: "true"
      feature.node.kubernetes.io/cpu-hardware_multithreading: "true"
      feature.node.kubernetes.io/cpu-model.family: "6"
      feature.node.kubernetes.io/cpu-model.id: "207"
      feature.node.kubernetes.io/cpu-model.vendor_id: Intel
      feature.node.kubernetes.io/cpu-pstate.status: passive
      feature.node.kubernetes.io/cpu-pstate.turbo: "true"
      feature.node.kubernetes.io/kernel-config.NO_HZ: "true"
      feature.node.kubernetes.io/kernel-config.NO_HZ_FULL: "true"
      feature.node.kubernetes.io/kernel-selinux.enabled: "true"
      feature.node.kubernetes.io/kernel-version.full: 5.14.0-570.99.1.el9_6.x86_64
      feature.node.kubernetes.io/kernel-version.major: "5"
      feature.node.kubernetes.io/kernel-version.minor: "14"
      feature.node.kubernetes.io/kernel-version.revision: "0"
      feature.node.kubernetes.io/memory-numa: "true"
      feature.node.kubernetes.io/network-sriov.capable: "true"
      feature.node.kubernetes.io/pci-10de.present: "true"
      feature.node.kubernetes.io/pci-10de.sriov.capable: "true"
      feature.node.kubernetes.io/pci-14e4.present: "true"
      feature.node.kubernetes.io/pci-15b3.present: "true"
      feature.node.kubernetes.io/pci-15b3.sriov.capable: "true"
      feature.node.kubernetes.io/pci-102b.present: "true"
      feature.node.kubernetes.io/rdma.available: "true"
      feature.node.kubernetes.io/rdma.capable: "true"
      feature.node.kubernetes.io/storage-nonrotationaldisk: "true"
      feature.node.kubernetes.io/system-os_release.ID: rhel
      feature.node.kubernetes.io/system-os_release.OPENSHIFT_VERSION: "4.21"
      feature.node.kubernetes.io/system-os_release.OSTREE_VERSION: 9.6.20260318-0
      feature.node.kubernetes.io/system-os_release.VERSION_ID: "9.6"
      feature.node.kubernetes.io/system-os_release.VERSION_ID.major: "9"
      feature.node.kubernetes.io/system-os_release.VERSION_ID.minor: "6"
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: <REDACTED_HOSTNAME>
      kubernetes.io/os: linux
      network.nvidia.com/operator.mofed.wait: "false"
      network.nvidia.com/operator.nic-configuration.wait: "false"
      node-role.kubernetes.io/worker: ""
      node.openshift.io/os_id: rhel
      nvidia.com/cuda.driver-version.full: 580.126.20
      nvidia.com/cuda.driver-version.major: "580"
      nvidia.com/cuda.driver-version.minor: "126"
      nvidia.com/cuda.driver-version.revision: "20"
      nvidia.com/cuda.driver.major: "580"
      nvidia.com/cuda.driver.minor: "126"
      nvidia.com/cuda.driver.rev: "20"
      nvidia.com/cuda.runtime-version.full: "13.0"
      nvidia.com/cuda.runtime-version.major: "13"
      nvidia.com/cuda.runtime-version.minor: "0"
      nvidia.com/cuda.runtime.major: "13"
      nvidia.com/cuda.runtime.minor: "0"
      nvidia.com/dra-kubelet-plugin: "true"
      nvidia.com/gfd.timestamp: "1776685099"
      nvidia.com/gpu-driver-upgrade-state: upgrade-done
      nvidia.com/gpu.compute.major: "10"
      nvidia.com/gpu.compute.minor: "0"
      nvidia.com/gpu.count: "8"
      nvidia.com/gpu.deploy.container-toolkit: "true"
      nvidia.com/gpu.deploy.dcgm: "true"
      nvidia.com/gpu.deploy.dcgm-exporter: "true"
      nvidia.com/gpu.deploy.device-plugin: "true"
      nvidia.com/gpu.deploy.driver: "true"
      nvidia.com/gpu.deploy.gpu-feature-discovery: "true"
      nvidia.com/gpu.deploy.mig-manager: "true"
      nvidia.com/gpu.deploy.node-status-exporter: "true"
      nvidia.com/gpu.deploy.nvsm: "true"
      nvidia.com/gpu.deploy.operator-validator: "true"
      nvidia.com/gpu.family: blackwell
      nvidia.com/gpu.machine: PowerEdge-XE9680L
      nvidia.com/gpu.memory: "183359"
      nvidia.com/gpu.mode: compute
      nvidia.com/gpu.present: "true"
      nvidia.com/gpu.product: NVIDIA-B200
      nvidia.com/gpu.replicas: "1"
      nvidia.com/gpu.sharing-strategy: none
      nvidia.com/mig.capable: "true"
      nvidia.com/mig.config: all-disabled
      nvidia.com/mig.config.state: success
      nvidia.com/mps.capable: "false"
      nvidia.com/ofed-driver-upgrade-state: upgrade-done
      nvidia.com/vgpu.present: "false"
      sriovnetwork.openshift.io/device-plugin: Disabled
    name: <REDACTED_HOSTNAME>
    resourceVersion: "232218932"
    uid: <REDACTED_UID>
  spec: {}
  status:
    addresses:
    - address: <REDACTED_IP>
      type: InternalIP
    - address: <REDACTED_HOSTNAME>
      type: Hostname
    allocatable:
      cpu: 223500m
      ephemeral-storage: "862137112786"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 2112083384Ki
      nvidia.com/gpu: "8"
      nvidia.com/roce: "0"
      openshift.io/ens31f0np0_roce: "16"
      openshift.io/ens32f0np0_roce: "16"
      openshift.io/ens34f0np0_roce: "16"
      openshift.io/ens35f0np0_roce: "16"
      openshift.io/ens36f0np0_roce: "16"
      openshift.io/ens37f0np0_roce: "16"
      openshift.io/ens38f0np0_roce: "16"
      openshift.io/ens39f1np1_roce: "0"
      openshift.io/ens40f0np0_roce: "16"
      openshift.io/ens41f0np0_roce: "16"
      openshift.io/ens42f0np0_roce: "16"
      pods: "250"
      rdma/ens31f0np0_roce: "0"
      rdma/ens32f0np0_roce: "0"
      rdma/ens34f0np0_roce: "160"
      rdma/ens35f0np0_roce: "0"
      rdma/ens36f0np0_roce: "0"
      rdma/ens37f0np0_roce: "0"
      rdma/ens38f0np0_roce: "0"
      rdma/ens40f0np0_roce: "0"
      rdma/ens41f0np0_roce: "0"
      rdma/ens42f0np0_roce: "160"
      rdma/roce_shared: "0"
      rdma/shared_roce: "100"
    capacity:
      cpu: "224"
      ephemeral-storage: 936643724Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 2113234360Ki
      nvidia.com/gpu: "8"
      nvidia.com/roce: "0"
      openshift.io/ens31f0np0_roce: "16"
      openshift.io/ens32f0np0_roce: "16"
      openshift.io/ens34f0np0_roce: "16"
      openshift.io/ens35f0np0_roce: "16"
      openshift.io/ens36f0np0_roce: "16"
      openshift.io/ens37f0np0_roce: "16"
      openshift.io/ens38f0np0_roce: "16"
      openshift.io/ens39f1np1_roce: "0"
      openshift.io/ens40f0np0_roce: "16"
      openshift.io/ens41f0np0_roce: "16"
      openshift.io/ens42f0np0_roce: "16"
      pods: "250"
      rdma/ens31f0np0_roce: "0"
      rdma/ens32f0np0_roce: "0"
      rdma/ens34f0np0_roce: "160"
      rdma/ens35f0np0_roce: "0"
      rdma/ens36f0np0_roce: "0"
      rdma/ens37f0np0_roce: "0"
      rdma/ens38f0np0_roce: "0"
      rdma/ens40f0np0_roce: "0"
      rdma/ens41f0np0_roce: "0"
      rdma/ens42f0np0_roce: "160"
      rdma/roce_shared: "0"
      rdma/shared_roce: "100"
    conditions:
    - lastHeartbeatTime: "2026-04-20T13:52:49Z"
      lastTransitionTime: "2026-04-20T06:24:37Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2026-04-20T13:52:49Z"
      lastTransitionTime: "2026-04-20T06:24:37Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2026-04-20T13:52:49Z"
      lastTransitionTime: "2026-04-20T06:24:37Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2026-04-20T13:52:49Z"
      lastTransitionTime: "2026-04-20T06:24:37Z"
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    features:
      supplementalGroupsPolicy: true
    nodeInfo:
      architecture: amd64
      bootID: <REDACTED>
      containerRuntimeVersion: cri-o://1.34.6-2.rhaos4.21.gitbca534a.el9
      kernelVersion: 5.14.0-570.99.1.el9_6.x86_64
      kubeProxyVersion: ""
      kubeletVersion: v1.34.5
      machineID: <REDACTED>
      operatingSystem: linux
      osImage: Red Hat Enterprise Linux CoreOS 9.6.20260318-0 (Plow)
      systemUUID: <REDACTED>
    runtimeHandlers:
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: nvidia-cdi
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: nvidia-legacy
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: crun
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: ""
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: runc
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: nvidia
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      capacity.topolvm.io/00default: "201614405140480"
      capacity.topolvm.io/vg1: "201614405140480"
      csi.volume.kubernetes.io/nodeid: '{"topolvm.io":"<REDACTED_HOSTNAME>"}'
      k8s.ovn.org/host-cidrs: '["<REDACTED_IP>/24"]'
      k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","bridge-id":"br-ex","interface-id":"br-ex_<REDACTED_HOSTNAME>","mac-address":"<REDACTED_MAC>","ip-addresses":["<REDACTED_IP>/24"],"ip-address":"<REDACTED_IP>/24","next-hops":["<REDACTED_IP>"],"next-hop":"<REDACTED_IP>","node-port-enable":"true","vlan-id":"0"}}'
      k8s.ovn.org/layer2-topology-version: "2.0"
      k8s.ovn.org/node-chassis-id: <REDACTED>
      k8s.ovn.org/node-encap-ips: '["<REDACTED_IP>"]'
      k8s.ovn.org/node-id: "2"
      k8s.ovn.org/node-masquerade-subnet: '{"ipv4":"169.254.0.0/17","ipv6":"fd69::/112"}'
      k8s.ovn.org/node-primary-ifaddr: '{"ipv4":"<REDACTED_IP>/24"}'
      k8s.ovn.org/node-subnets: '{"default":["10.128.0.0/23"]}'
      k8s.ovn.org/node-transit-switch-port-ifaddr: '{"ipv4":"100.88.0.2/16"}'
      k8s.ovn.org/remote-zone-migrated: <REDACTED_HOSTNAME>
      k8s.ovn.org/zone-name: <REDACTED_HOSTNAME>
      machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
      machineconfiguration.openshift.io/currentConfig: rendered-master-ec38943116eccbaa72c5dc90c9c709f9
      machineconfiguration.openshift.io/desiredConfig: rendered-master-ec38943116eccbaa72c5dc90c9c709f9
      machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-ec38943116eccbaa72c5dc90c9c709f9
      machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-ec38943116eccbaa72c5dc90c9c709f9
      machineconfiguration.openshift.io/lastObservedServerCAAnnotation: "false"
      machineconfiguration.openshift.io/lastSyncedControllerConfigResourceVersion: "230864234"
      machineconfiguration.openshift.io/post-config-action: ""
      machineconfiguration.openshift.io/reason: ""
      machineconfiguration.openshift.io/state: Done
      nfd.node.kubernetes.io/feature-labels: cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AMXBF16,cpu-cpuid.AMXFP8,cpu-cpuid.AMXINT8,cpu-cpuid.AMXTILE,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.AVX512BF16,cpu-cpuid.AVX512BITALG,cpu-cpuid.AVX512BW,cpu-cpuid.AVX512CD,cpu-cpuid.AVX512DQ,cpu-cpuid.AVX512F,cpu-cpuid.AVX512FP16,cpu-cpuid.AVX512IFMA,cpu-cpuid.AVX512VBMI,cpu-cpuid.AVX512VBMI2,cpu-cpuid.AVX512VL,cpu-cpuid.AVX512VNNI,cpu-cpuid.AVX512VPOPCNTDQ,cpu-cpuid.AVXVNNI,cpu-cpuid.BHI_CTRL,cpu-cpuid.CETIBT,cpu-cpuid.CETSS,cpu-cpuid.CLDEMOTE,cpu-cpuid.CMPSB_SCADBS_SHORT,cpu-cpuid.CMPXCHG8,cpu-cpuid.ENQCMD,cpu-cpuid.FLUSH_L1D,cpu-cpuid.FMA3,cpu-cpuid.FSRM,cpu-cpuid.FXSR,cpu-cpuid.FXSROPT,cpu-cpuid.GFNI,cpu-cpuid.IA32_ARCH_CAP,cpu-cpuid.IA32_CORE_CAP,cpu-cpuid.IBPB,cpu-cpuid.IDPRED_CTRL,cpu-cpuid.LAHF,cpu-cpuid.MD_CLEAR,cpu-cpuid.MOVBE,cpu-cpuid.MOVDIR64B,cpu-cpuid.MOVDIRI,cpu-cpuid.MOVSB_ZL,cpu-cpuid.OSXSAVE,cpu-cpuid.PCONFIG,cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES,cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS,cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES,cpu-cpuid.PMU_FIXEDCOUNTER_TOPDOWN_SLOTS,cpu-cpuid.PSFD,cpu-cpuid.RRSBA_CTRL,cpu-cpuid.SERIALIZE,cpu-cpuid.SHA,cpu-cpuid.SPEC_CTRL_SSBD,cpu-cpuid.STIBP,cpu-cpuid.STOSB_SHORT,cpu-cpuid.SYSCALL,cpu-cpuid.SYSEE,cpu-cpuid.TME,cpu-cpuid.TSXLDTRK,cpu-cpuid.VAES,cpu-cpuid.VMX,cpu-cpuid.VPCLMULQDQ,cpu-cpuid.WAITPKG,cpu-cpuid.WBNOINVD,cpu-cpuid.X87,cpu-cpuid.XGETBV1,cpu-cpuid.XSAVE,cpu-cpuid.XSAVEC,cpu-cpuid.XSAVEOPT,cpu-cpuid.XSAVES,cpu-cstate.enabled,cpu-hardware_multithreading,cpu-model.family,cpu-model.id,cpu-model.vendor_id,cpu-pstate.status,cpu-pstate.turbo,kernel-config.NO_HZ,kernel-config.NO_HZ_FULL,kernel-selinux.enabled,kernel-version.full,kernel-version.major,kernel-version.minor,kernel-version.revision,memory-numa,network-sriov.capable,nvidia.com/cuda.driver-version.full,nvidia.com/cuda.driver-version.major,nvidia.com/cuda.driver-version.minor,nvidia.com/cuda.driver-version.revision,nvidia.com/cuda.driver.major,nvidia.com/cuda.driver.minor,nvidia.com/cuda.driver.rev,nvidia.com/cuda.runtime-version.full,nvidia.com/cuda.runtime-version.major,nvidia.com/cuda.runtime-version.minor,nvidia.com/cuda.runtime.major,nvidia.com/cuda.runtime.minor,nvidia.com/gfd.timestamp,nvidia.com/gpu.compute.major,nvidia.com/gpu.compute.minor,nvidia.com/gpu.count,nvidia.com/gpu.family,nvidia.com/gpu.machine,nvidia.com/gpu.memory,nvidia.com/gpu.mode,nvidia.com/gpu.product,nvidia.com/gpu.replicas,nvidia.com/gpu.sharing-strategy,nvidia.com/mig.capable,nvidia.com/mps.capable,nvidia.com/vgpu.present,pci-102b.present,pci-10de.present,pci-10de.sriov.capable,pci-14e4.present,pci-15b3.present,pci-15b3.sriov.capable,rdma.available,rdma.capable,storage-nonrotationaldisk,system-os_release.ID,system-os_release.OPENSHIFT_VERSION,system-os_release.OSTREE_VERSION,system-os_release.VERSION_ID,system-os_release.VERSION_ID.major,system-os_release.VERSION_ID.minor
      nvidia.com/gpu-driver-upgrade-enabled: "true"
      sriovnetwork.openshift.io/state: Idle
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2026-02-11T03:32:26Z"
    finalizers:
    - topolvm.io/node
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      feature.node.kubernetes.io/cpu-cpuid.ADX: "true"
      feature.node.kubernetes.io/cpu-cpuid.AESNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXBF16: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXFP8: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXINT8: "true"
      feature.node.kubernetes.io/cpu-cpuid.AMXTILE: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX2: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512BF16: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512BITALG: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512BW: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512CD: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512DQ: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512F: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512FP16: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512IFMA: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI2: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VL: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVX512VPOPCNTDQ: "true"
      feature.node.kubernetes.io/cpu-cpuid.AVXVNNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.BHI_CTRL: "true"
      feature.node.kubernetes.io/cpu-cpuid.CETIBT: "true"
      feature.node.kubernetes.io/cpu-cpuid.CETSS: "true"
      feature.node.kubernetes.io/cpu-cpuid.CLDEMOTE: "true"
      feature.node.kubernetes.io/cpu-cpuid.CMPSB_SCADBS_SHORT: "true"
      feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8: "true"
      feature.node.kubernetes.io/cpu-cpuid.ENQCMD: "true"
      feature.node.kubernetes.io/cpu-cpuid.FLUSH_L1D: "true"
      feature.node.kubernetes.io/cpu-cpuid.FMA3: "true"
      feature.node.kubernetes.io/cpu-cpuid.FSRM: "true"
      feature.node.kubernetes.io/cpu-cpuid.FXSR: "true"
      feature.node.kubernetes.io/cpu-cpuid.FXSROPT: "true"
      feature.node.kubernetes.io/cpu-cpuid.GFNI: "true"
      feature.node.kubernetes.io/cpu-cpuid.IA32_ARCH_CAP: "true"
      feature.node.kubernetes.io/cpu-cpuid.IA32_CORE_CAP: "true"
      feature.node.kubernetes.io/cpu-cpuid.IBPB: "true"
      feature.node.kubernetes.io/cpu-cpuid.IDPRED_CTRL: "true"
      feature.node.kubernetes.io/cpu-cpuid.LAHF: "true"
      feature.node.kubernetes.io/cpu-cpuid.MD_CLEAR: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVBE: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVDIR64B: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVDIRI: "true"
      feature.node.kubernetes.io/cpu-cpuid.MOVSB_ZL: "true"
      feature.node.kubernetes.io/cpu-cpuid.OSXSAVE: "true"
      feature.node.kubernetes.io/cpu-cpuid.PCONFIG: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES: "true"
      feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_TOPDOWN_SLOTS: "true"
      feature.node.kubernetes.io/cpu-cpuid.PSFD: "true"
      feature.node.kubernetes.io/cpu-cpuid.RRSBA_CTRL: "true"
      feature.node.kubernetes.io/cpu-cpuid.SERIALIZE: "true"
      feature.node.kubernetes.io/cpu-cpuid.SHA: "true"
      feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD: "true"
      feature.node.kubernetes.io/cpu-cpuid.STIBP: "true"
      feature.node.kubernetes.io/cpu-cpuid.STOSB_SHORT: "true"
      feature.node.kubernetes.io/cpu-cpuid.SYSCALL: "true"
      feature.node.kubernetes.io/cpu-cpuid.SYSEE: "true"
      feature.node.kubernetes.io/cpu-cpuid.TME: "true"
      feature.node.kubernetes.io/cpu-cpuid.TSXLDTRK: "true"
      feature.node.kubernetes.io/cpu-cpuid.VAES: "true"
      feature.node.kubernetes.io/cpu-cpuid.VMX: "true"
      feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ: "true"
      feature.node.kubernetes.io/cpu-cpuid.WAITPKG: "true"
      feature.node.kubernetes.io/cpu-cpuid.WBNOINVD: "true"
      feature.node.kubernetes.io/cpu-cpuid.X87: "true"
      feature.node.kubernetes.io/cpu-cpuid.XGETBV1: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVE: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVEC: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT: "true"
      feature.node.kubernetes.io/cpu-cpuid.XSAVES: "true"
      feature.node.kubernetes.io/cpu-cstate.enabled: "true"
      feature.node.kubernetes.io/cpu-hardware_multithreading: "true"
      feature.node.kubernetes.io/cpu-model.family: "6"
      feature.node.kubernetes.io/cpu-model.id: "207"
      feature.node.kubernetes.io/cpu-model.vendor_id: Intel
      feature.node.kubernetes.io/cpu-pstate.status: passive
      feature.node.kubernetes.io/cpu-pstate.turbo: "true"
      feature.node.kubernetes.io/kernel-config.NO_HZ: "true"
      feature.node.kubernetes.io/kernel-config.NO_HZ_FULL: "true"
      feature.node.kubernetes.io/kernel-selinux.enabled: "true"
      feature.node.kubernetes.io/kernel-version.full: 5.14.0-570.99.1.el9_6.x86_64
      feature.node.kubernetes.io/kernel-version.major: "5"
      feature.node.kubernetes.io/kernel-version.minor: "14"
      feature.node.kubernetes.io/kernel-version.revision: "0"
      feature.node.kubernetes.io/memory-numa: "true"
      feature.node.kubernetes.io/network-sriov.capable: "true"
      feature.node.kubernetes.io/pci-10de.present: "true"
      feature.node.kubernetes.io/pci-10de.sriov.capable: "true"
      feature.node.kubernetes.io/pci-14e4.present: "true"
      feature.node.kubernetes.io/pci-15b3.present: "true"
      feature.node.kubernetes.io/pci-15b3.sriov.capable: "true"
      feature.node.kubernetes.io/pci-102b.present: "true"
      feature.node.kubernetes.io/rdma.available: "true"
      feature.node.kubernetes.io/rdma.capable: "true"
      feature.node.kubernetes.io/storage-nonrotationaldisk: "true"
      feature.node.kubernetes.io/system-os_release.ID: rhel
      feature.node.kubernetes.io/system-os_release.OPENSHIFT_VERSION: "4.21"
      feature.node.kubernetes.io/system-os_release.OSTREE_VERSION: 9.6.20260318-0
      feature.node.kubernetes.io/system-os_release.VERSION_ID: "9.6"
      feature.node.kubernetes.io/system-os_release.VERSION_ID.major: "9"
      feature.node.kubernetes.io/system-os_release.VERSION_ID.minor: "6"
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: <REDACTED_HOSTNAME>
      kubernetes.io/os: linux
      network.nvidia.com/operator.mofed.wait: "false"
      network.nvidia.com/operator.nic-configuration.wait: "false"
      node-role.kubernetes.io/control-plane: ""
      node-role.kubernetes.io/master: ""
      node-role.kubernetes.io/worker: ""
      node.openshift.io/os_id: rhel
      nvidia.com/cuda.driver-version.full: 580.126.20
      nvidia.com/cuda.driver-version.major: "580"
      nvidia.com/cuda.driver-version.minor: "126"
      nvidia.com/cuda.driver-version.revision: "20"
      nvidia.com/cuda.driver.major: "580"
      nvidia.com/cuda.driver.minor: "126"
      nvidia.com/cuda.driver.rev: "20"
      nvidia.com/cuda.runtime-version.full: "13.0"
      nvidia.com/cuda.runtime-version.major: "13"
      nvidia.com/cuda.runtime-version.minor: "0"
      nvidia.com/cuda.runtime.major: "13"
      nvidia.com/cuda.runtime.minor: "0"
      nvidia.com/dra-kubelet-plugin: "true"
      nvidia.com/gfd.timestamp: "1776666680"
      nvidia.com/gpu-driver-upgrade-state: upgrade-done
      nvidia.com/gpu.compute.major: "10"
      nvidia.com/gpu.compute.minor: "0"
      nvidia.com/gpu.count: "8"
      nvidia.com/gpu.deploy.container-toolkit: "true"
      nvidia.com/gpu.deploy.dcgm: "true"
      nvidia.com/gpu.deploy.dcgm-exporter: "true"
      nvidia.com/gpu.deploy.device-plugin: "true"
      nvidia.com/gpu.deploy.driver: "true"
      nvidia.com/gpu.deploy.gpu-feature-discovery: "true"
      nvidia.com/gpu.deploy.mig-manager: "true"
      nvidia.com/gpu.deploy.node-status-exporter: "true"
      nvidia.com/gpu.deploy.nvsm: "true"
      nvidia.com/gpu.deploy.operator-validator: "true"
      nvidia.com/gpu.family: blackwell
      nvidia.com/gpu.machine: PowerEdge-XE9680L
      nvidia.com/gpu.memory: "183359"
      nvidia.com/gpu.mode: compute
      nvidia.com/gpu.present: "true"
      nvidia.com/gpu.product: NVIDIA-B200
      nvidia.com/gpu.replicas: "1"
      nvidia.com/gpu.sharing-strategy: none
      nvidia.com/mig.capable: "true"
      nvidia.com/mig.config: all-disabled
      nvidia.com/mig.config.state: success
      nvidia.com/mps.capable: "false"
      nvidia.com/ofed-driver-upgrade-state: upgrade-done
      nvidia.com/vgpu.present: "false"
      sriovnetwork.openshift.io/device-plugin: Disabled
      topology.topolvm.io/node: <REDACTED_HOSTNAME>
    name: <REDACTED_HOSTNAME>
    resourceVersion: "232218950"
    uid: <REDACTED_UID>
  spec: {}
  status:
    addresses:
    - address: <REDACTED_IP>
      type: InternalIP
    - address: <REDACTED_HOSTNAME>
      type: Hostname
    allocatable:
      cpu: 223500m
      ephemeral-storage: "6909869816654"
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 2112081812Ki
      nvidia.com/gpu: "8"
      nvidia.com/roce: "0"
      pods: "250"
      rdma/ens31f0np0_roce: "0"
      rdma/ens32f0np0_roce: "0"
      rdma/ens34f0np0_roce: "160"
      rdma/ens35f0np0_roce: "0"
      rdma/ens36f0np0_roce: "0"
      rdma/ens37f0np0_roce: "0"
      rdma/ens38f0np0_roce: "0"
      rdma/ens40f0np0_roce: "0"
      rdma/ens41f0np0_roce: "0"
      rdma/ens42f0np0_roce: "160"
      rdma/roce_shared: "0"
      rdma/shared_roce: "100"
    capacity:
      cpu: "224"
      ephemeral-storage: 7498853700Ki
      hugepages-1Gi: "0"
      hugepages-2Mi: "0"
      memory: 2113232788Ki
      nvidia.com/gpu: "8"
      nvidia.com/roce: "0"
      pods: "250"
      rdma/ens31f0np0_roce: "0"
      rdma/ens32f0np0_roce: "0"
      rdma/ens34f0np0_roce: "160"
      rdma/ens35f0np0_roce: "0"
      rdma/ens36f0np0_roce: "0"
      rdma/ens37f0np0_roce: "0"
      rdma/ens38f0np0_roce: "0"
      rdma/ens40f0np0_roce: "0"
      rdma/ens41f0np0_roce: "0"
      rdma/ens42f0np0_roce: "160"
      rdma/roce_shared: "0"
      rdma/shared_roce: "100"
    conditions:
    - lastHeartbeatTime: "2026-04-20T13:52:51Z"
      lastTransitionTime: "2026-02-11T03:32:26Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2026-04-20T13:52:51Z"
      lastTransitionTime: "2026-02-11T03:32:26Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2026-04-20T13:52:51Z"
      lastTransitionTime: "2026-02-11T03:32:26Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2026-04-20T13:52:51Z"
      lastTransitionTime: "2026-04-20T06:15:56Z"
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    features:
      supplementalGroupsPolicy: true
    nodeInfo:
      architecture: amd64
      bootID: <REDACTED>
      containerRuntimeVersion: cri-o://1.34.6-2.rhaos4.21.gitbca534a.el9
      kernelVersion: 5.14.0-570.99.1.el9_6.x86_64
      kubeProxyVersion: ""
      kubeletVersion: v1.34.5
      machineID: <REDACTED>
      operatingSystem: linux
      osImage: Red Hat Enterprise Linux CoreOS 9.6.20260318-0 (Plow)
      systemUUID: <REDACTED>
    runtimeHandlers:
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: nvidia-cdi
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: nvidia-legacy
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: crun
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: ""
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: runc
    - features:
        recursiveReadOnlyMounts: true
        userNamespaces: true
      name: nvidia
kind: List
metadata:
  resourceVersion: ""

Environment:

• Kubernetes version (use kubectl version):

$ oc version 
Client Version: 4.20.0
Kustomize Version: v5.6.0
Server Version: 4.21.9
Kubernetes Version: v1.34.6

• Hardware configuration:
  • PowerEdge XE9680L 8xB200
  • Network adapter model and firmware version: ConnectX-7 NICs + Firmware (To add) (But I don't think it's the NIC firmware that's the issue here)
• OS (e.g: cat /etc/os-release):

NAME="Red Hat Enterprise Linux CoreOS"
VERSION="9.6.20260318-0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 9.6.20260318-0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
OSTREE_VERSION='9.6.20260318-0'
VARIANT=CoreOS
VARIANT_ID=coreos
OPENSHIFT_VERSION="4.21"

• Kernel (e.g. uname -a):

  • Failing : Linux dell-<REDACTED_HOSTNAME> 5.14.0-570.104.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 11 21:37:41 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux
  • Working : Linux dell-<REDACTED_HOSTNAME> 5.14.0-570.99.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 11 21:37:41 EDT 2026 x86_64 x86_64 x86_64 GNU/Linux

• Others:

I had to revert/pin my coreOS Image version to the hash corresponding to OCP 4.21.7 to continue working.

[thameem-abbas]

Digging into the rpmbuild :

  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/cmd.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/context.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/pat_arg.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/buddy.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/pool.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/table.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/action.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/rule.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/matcher.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/send.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/definer.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/vport.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/debug.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/fs_hws_pools.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/bwc_complex.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/fs_hws.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/steering/hws/action_ste_pool.o
  CC [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/dpll.o
  LD [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/infiniband/core/iw_cm.o
  LD [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/infiniband/core/rdma_ucm.o
  LD [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/net/mlxdevm/mlxdevm.o
  LD [M]  /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/infiniband/core/ib_uverbs.o
In file included from /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/dpll.c:4:
/var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/dpll.c: In function 'mlx5_dpll_probe':
./include/linux/dpll.h:270:25: error: too few arguments to function 'dpll_device_get_rh'
  270 | #define dpll_device_get dpll_device_get_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:270:25: note: in definition of macro 'dpll_device_get'
  270 | #define dpll_device_get dpll_device_get_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:270:25: note: declared here
  270 | #define dpll_device_get dpll_device_get_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:270:25: note: in definition of macro 'dpll_device_get'
  270 | #define dpll_device_get dpll_device_get_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:272:25: error: too few arguments to function 'dpll_pin_get_rh'
  272 | #define dpll_pin_get    dpll_pin_get_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:272:25: note: in definition of macro 'dpll_pin_get'
  272 | #define dpll_pin_get    dpll_pin_get_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:272:25: note: declared here
  272 | #define dpll_pin_get    dpll_pin_get_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:272:25: note: in definition of macro 'dpll_pin_get'
  272 | #define dpll_pin_get    dpll_pin_get_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: error: too few arguments to function 'dpll_pin_put_rh'
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: note: in definition of macro 'dpll_pin_put'
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: note: declared here
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: note: in definition of macro 'dpll_pin_put'
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: error: too few arguments to function 'dpll_device_put_rh'
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: note: in definition of macro 'dpll_device_put'
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: note: declared here
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: note: in definition of macro 'dpll_device_put'
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
/var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/dpll.c: In function 'mlx5_dpll_remove':
./include/linux/dpll.h:273:25: error: too few arguments to function 'dpll_pin_put_rh'
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: note: in definition of macro 'dpll_pin_put'
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: note: declared here
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:273:25: note: in definition of macro 'dpll_pin_put'
  273 | #define dpll_pin_put    dpll_pin_put_rh
      |                         ^~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: error: too few arguments to function 'dpll_device_put_rh'
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: note: in definition of macro 'dpll_device_put'
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: note: declared here
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
./include/linux/dpll.h:271:25: note: in definition of macro 'dpll_device_put'
  271 | #define dpll_device_put dpll_device_put_rh
      |                         ^~~~~~~~~~~~~~~~~~
make[3]: *** [scripts/Makefile.build:249: /var/tmp/OFED_topdir/BUILD/mlnx-ofa_kernel-26.01/obj/default/drivers/net/ethernet/mellanox/mlx5/core/dpll.o] Error 1
make[3]: *** Waiting for unfinished jobs....

[rollandf]

Issue in OCP 4.21.9

See:
https://access.redhat.com/solutions/7141156

[schmaustech]

You can work around this by using the doca3.2.2-25.10-2.4.1.0-4 drivers. I also found using the beta from 26.4 also works doca3.4.0-26.04-0.5.3.0-0. Why doesn't it work with doca3.3.0-26.01-1.0.0.0-0 when the kernel for all 3 contain the changes outlined in the KCS 7141156 @rollandf

[michich]

FYI, I am proposing a compatibility workaround on the RHEL kernel side: https://redhat.atlassian.net/browse/RHEL-174822