meet a error when trying to use nvkind

[amyburden]

umount: /proc/driver/nvidia: not mounted
F1210 02:39:07.328168 3748403 main.go:45] Error: patching /proc/driver/nvidia on node 'explicit-gpus-worker': running script on explicit-gpus-worker: executing command: exit status 1

[jacobtomlinson]

Can you share the output of nvidia-smi?

[elezar]

@amyburden are you using NVIDIA Container Toolkit v1.18.0 or later?

[amyburden]

[wonkyoc]

@elezar it seems the problem occurs on nvidia-ctk v1.18.0 or later than v1.17.1. I had the same problem but it disappeared after downgrading. More a context, a testing docker was able to see GPUs but the cluster failed to apply the script.

# This fails to umount
VERSION:
   1.18.1
commit: efe99418ef87500dbe059cadc9ab418b2815b9d5

# This works
VERSION:
   1.17.1
commit: 1467f3f339a855f20d179e9883e418a09118a93e

$ nvidia-smi

Thu Jan 15 01:04:56 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A6000               Off |   00000000:21:00.0 Off |                  Off |
| 30%   47C    P8             27W /  300W |       4MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A6000               Off |   00000000:49:00.0 Off |                  Off |
| 30%   55C    P0             88W /  300W |       0MiB /  49140MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[jaiakash]

Hi, I also faced this issue, and applying @wonkyoc’s fix resolved it on my Ubuntu 24.04 VM.

The root cause seems to be that nvidia-ctk moved to a CDI-based architecture, replacing the earlier /proc-based mechanism, which causes compatibility issues with the current implementation.

Thanks a lot for the workaround!

Are there any plans to address this in upstream?
If so, I’d be happy to help work on a fix.