Worker Node Fails when Network Operator attempts to install DOCA drivers

[pamudithaA]

What happened:
DGX worker node becomes notReady and inaccessible during the network operator install. Node has to be rebooted to recover and then again Network operator will try to install DOCA drivers triggering a loop of node failure.

What you expected to happen:
Network operator install the doca drivers successfully and k8s cluster remains healthy.

How to reproduce it (as minimally and precisely as possible):
In the current setup we are using DGX servers with Kubernetes.
Nvidia network operator 25.10.0 is installed with below settings.

It seems if any mlnx interface is in use for OAM, it could fail to during openibd restart

Anything else we need to know?:
I have only tested in environments with Connect-X card being used for OAM.
Issue is consistent and happens with both bonded OAM interface and physical NIC as the OAM interface.

Logs:

• Network Operator version: 25.10.0
• Helm Configuration (if applicable):

helm install network-operator nvidia/network-operator \
   -n network-operator \
   --create-namespace \
   --version 25.10.0 \
   --wait

This is the snippet of last log lines from the kubectl logs -f mofed-ubuntu24.04-84cb6b9485-ds-mpj9b -n nvidia-network-operator . Upon this point the node will become notReady and SSH/ping connectivity is failing.

[13-Nov-25_12:19:23] Apply blacklisted mofed modules file to host (/host/etc/modprobe.d/blacklist-ofed-modules.conf)
Function: generate_ofed_modules_blacklist

Environment:

• Kubernetes version (use kubectl version): 1.33
• Hardware configuration: Nvidia DGX H200
  • Network adapter model and firmware version: ConnectX-7
• OS (e.g: cat /etc/os-release): Ubuntu 24.04
• Kernel (e.g. uname -a): 6.8.0-90-generic
• Others:

[pamudithaA]

Based on the entryPoint.sh script used inside the doca-driver-build, the failing step is /etc/init.d/openibd restart.
If this is caused by Connect-X card being used for OAM interface, I guess it would be worth mentioning it in the reference docs or as a known issue.

[NurullahMorshed]

I'm seeing the same issue with the same final logs. before the machine goes down.

Environment:

Kubernetes version (use kubectl version): 1.33.1
Hardware configuration: Nvidia B200
Network adapter model and firmware version: ConnectX-7
OS (e.g: cat /etc/os-release): Ubuntu 24.04
Kernel (e.g. uname -a): 6.14.0-1016-oracle
Others:

@pamudithaA were you able to find a workaround for the issue?

[NurullahMorshed]

@rollandf Sorry to annoy on this, but is there a known workaround for this issue?

[pamudithaA]

@NurullahMorshed

• I am manually installing doca drivers using DOCA Online Repository doca-ofed while setting up the node.
• Then restart the node. (Even if I install the drivers manually, still running openibd restart will end up in same state.)
• Then set up Network and GPU operators ( GPU operator with driver.rdma.useHostMofed=true)
• Do not include ofedDriver in the nicclusterpolicy.

[NurullahMorshed]

@pamudithaA I got it working by setting the RESTORE_DRIVER_ON_POD_TERMINATION environment variable to true on the ofed driver container.

As long as you don't force terminate the node, it will gracefully shut down the pod when restarting, and the variable will unload the driver when the pod shuts down. The only catch is I can't get it working on a node that already had drivers installed (host or nicclusterpolicy managed). In both cases, I had to recycle the node or the boot image and start from scratch with the node.

[pamudithaA]

@NurullahMorshed You meant to say you are not hitting this bug when RESTORE_DRIVER_ON_POD_TERMINATION is set to true? That's interesting. I thought that variable is only come into rescue when the pod is gracefully terminated while the network operator is already working.
But the bug I reported here means, I can't get the network operator operational in the first place as it couldn't finish the driver installation process.

[NurullahMorshed]

@pamudithaA My setup is different. In my case, i do not install doca drivers manually during setup. I let network operator take care of installing the driver, and set that variable to handle pod or node restarts.

[almaslennikov]

Hi @pamudithaA could you collect the network operator sos report using this script? it could help us troubleshoot the issue: https://github.com/Mellanox/network-operator/tree/master/scripts/sosreport

[viswar4]

Is this issue resolved ? any workaround ? We are running into a similar issue with OpenShift worker nodes with connect-x6 (primary network) and connectx-8 nics for the gpu backend. Network Operator version 26.1.0