Pod workload deployment step is throwing "executable file not found" error for nvidia-smi

[aoyshi]

I am able to follow all the steps in the readme and get the expected responses, until the very last pod workload step of:

cat << EOF | kubectl --context=kind-${KIND_CLUSTER_NAME} apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: OnFailure
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["nvidia-smi", "-L"]
    resources:
      limits:
        nvidia.com/gpu: 2
EOF

I get an error in the container (which never starts, so I cannot exec -it into into to investigate):

failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.

I have the nvidia-device-plugin pods working correctly. Every step of the setup has worked so far until this very last one. Am I missing something?

More info:

I used this command to spin up the nvidia device plugin pods:

helm upgrade -i \
  --kube-context=kind-${KIND_CLUSTER_NAME} \
  --namespace nvidia \
  --set gfd.enabled=true \
  --set runtimeClassName=nvidia \
  --set deviceListStrategy=volume-mounts \
  --set deviceDiscoveryStrategy=nvml \
  --create-namespace \
  nvidia-device-plugin nvdp/nvidia-device-plugin

Output:

$ kubectl --context=kind-${KIND_CLUSTER_NAME} get pod -n nvidia

NAME                                                                                                      READY   STATUS    RESTARTS        AGE
nvidia-device-plugin-gpu-feature-discovery-r6t7s                                 1/1     Running   0               5m44s
nvidia-device-plugin-lsqvc                                                                      1/1     Running   0               5m44s
nvidia-device-plugin-node-feature-discovery-master-77b96ddqcxlw   1/1     Running   0               5m44s
nvidia-device-plugin-node-feature-discovery-worker-5n8nc                 1/1     Running   1 (5m23s ago)   5m44s

$ kubectl --context=kind-${KIND_CLUSTER_NAME} get nodes -o json | jq -r '.items[] | select(.metadata.name | test("-worker[0-9]*$")) | {name: .metadata.name, "nvidia.com/gpu": .status.allocatable["nvidia.com/gpu"]}'

{
  "name": "nvkind-vknmz-worker",
  "nvidia.com/gpu": "1"
}

Cluster GPUs: cluster was created using ./nvkind cluster create

$ ./nvkind cluster print-gpus

[
    {
        "node": "nvkind-vknmz-worker",
        "gpus": [
            {
                "Index": "0",
                "Name": "NVIDIA A100 80GB PCIe",
                "UUID": "GPU-a80141e3-fabe-57be-6d71-7c1b39e79553"
            }
        ]
    }
]

Error:

$ kubectl describe pod gpu-test

Name:             gpu-test
Namespace:        default
Priority:         0
Service Account:  default
Node:             nvkind-vknmz-worker
Start Time:       Fri, 28 Feb 2025 14:26:13 +0000
Labels:           <none>
Annotations:      <none>
Status:           Running
Containers:
  ctr:
    Container ID:  containerd://1f6e8d3be9c54fdcb182f57320e925bcb1d64e408ea04b17740022cac04d87c0
    Image:         ubuntu:22.04
    Image ID:      docker.io/library/ubuntu@sha256:ed1544e454989078f5dec1bfdabd8c5cc9c48e0705d07b678ab6ae3fb61952d2
    Port:          <none>
    Host Port:     <none>
    Command:
      nvidia-smi
      -L
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "nvidia-smi": executable file not found in $PATH
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 00:00:00 +0000
      Finished:     Fri, 28 Feb 2025 14:32:02 +0000
    Ready:          False
    Restart Count:  6
    Limits:
      nvidia.com/gpu:  1
    Requests:
      nvidia.com/gpu:  1
    Environment:       <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-stvml (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  kube-api-access-stvml:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  8m6s                   default-scheduler  Successfully assigned default/gpu-test to nvkind-vknmz-worker
  Normal   Pulling    8m5s                   kubelet            Pulling image "ubuntu:22.04"
  Normal   Pulled     8m3s                   kubelet            Successfully pulled image "ubuntu:22.04" in 1.715s (1.715s including waiting). Image size: 29545350 bytes.
  Warning  Failed     5m7s (x6 over 8m3s)    kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "nvidia-smi": executable file not found in $PATH
  Warning  BackOff    2m54s (x25 over 8m1s)  kubelet            Back-off restarting failed container ctr in pod gpu-test_default(94b2cc75-8ef2-4df6-98d9-2f4015a0dd3e)
  Normal   Created    2m17s (x7 over 8m3s)   kubelet            Created container: ctr
  Normal   Pulled     2m17s (x6 over 8m2s)   kubelet            Container image "ubuntu:22.04" already present on machine

[geraldstanje]

@aoyshi @ArangoGutierrez @jacobtomlinson @klueska any idea whats wrong or how to debug it? it looks like nvidia driver is not visible in the pod...

i get the same issue as above - error: (exec: "nvidia-smi": executable file not found in $PATH) ....

here some more infos:

• i created a kind cluster using these commands (both cluster fail with error above when creating the gpu-test pod):

./nvkind cluster create --config-template=examples/one-worker-per-gpu.yaml --name test

and

./nvkind cluster create --name test

• cluster infos:

kubectl get po -A
NAMESPACE            NAME                                                              READY   STATUS             RESTARTS        AGE
default              gpu-test                                                          0/1     CrashLoopBackOff   4 (46s ago)     2m21s
kube-system          coredns-668d6bf9bc-p7stg                                          1/1     Running            0               7m36s
kube-system          coredns-668d6bf9bc-tjhp5                                          1/1     Running            0               7m36s
kube-system          etcd-test-control-plane                                           1/1     Running            0               7m45s
kube-system          kindnet-2r8pd                                                     1/1     Running            0               7m36s
kube-system          kindnet-k6t68                                                     1/1     Running            0               7m31s
kube-system          kube-apiserver-test-control-plane                                 1/1     Running            0               7m43s
kube-system          kube-controller-manager-test-control-plane                        1/1     Running            0               7m43s
kube-system          kube-proxy-4f7fz                                                  1/1     Running            0               7m36s
kube-system          kube-proxy-mqzxz                                                  1/1     Running            0               7m31s
kube-system          kube-scheduler-test-control-plane                                 1/1     Running            0               7m43s
local-path-storage   local-path-provisioner-7dc846544d-mjq9h                           1/1     Running            0               7m36s
nvidia               nvidia-device-plugin-5xtnw                                        1/1     Running            0               3m38s
nvidia               nvidia-device-plugin-gpu-feature-discovery-kwbtm                  1/1     Running            0               3m38s
nvidia               nvidia-device-plugin-node-feature-discovery-master-77b96ddzsl2k   1/1     Running            0               3m59s
nvidia               nvidia-device-plugin-node-feature-discovery-worker-n7p4l          1/1     Running            1 (3m38s ago)  

• gpu infos:

nvidia-smi
Fri Mar 21 03:27:17 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.230.02             Driver Version: 535.230.02   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:1E.0 Off |                    0 |
| N/A   18C    P8               9W /  70W |      0MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

• docker run with gpu

docker run -v /dev/null:/var/run/nvidia-container-devices/all ubuntu:20.04 nvidia-smi -L
GPU 0: Tesla T4 (UUID: GPU-3f6a9bbc-6813-18ec-04c9-5bcbf59af2be)

• my gpu-test.yaml file (kubectl --context=kind-${KIND_CLUSTER_NAME} apply -f gpu-test.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: OnFailure
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["nvidia-smi", "-L"]
    resources:
      limits:
        nvidia.com/gpu: 1

• nvidia-device-plugin version:

helm list -n nvidia
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                        APP VERSION
nvidia-device-plugin    nvidia          1               2025-03-21 03:48:01.996627742 +0000 UTC deployed        nvidia-device-plugin-0.17.1  0.17.1    

• nvidia-device-plugin deployed with command (same as above):

helm upgrade -i \
  --kube-context=kind-${KIND_CLUSTER_NAME} \
  --namespace nvidia \
  --set gfd.enabled=true \
  --set runtimeClassName=nvidia \
  --set deviceListStrategy=volume-mounts \
  --set deviceDiscoveryStrategy=nvml \
  --create-namespace \
  nvidia-device-plugin nvdp/nvidia-device-plugin

@aoyshi you could try the following to bash into the container:

cat << EOF | kubectl --context=kind-${KIND_CLUSTER_NAME} apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test
spec:
  restartPolicy: OnFailure
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["sleep", "1000"]
    resources:
      limits:
        nvidia.com/gpu: 2
EOF

and than:

kubect exec -it container_id -- bash

[aoyshi]

Yes I was able to resolve it. Here are the steps:

1. Configure /etc/nvidia-container-runtime/config.toml to have these two lines

accept-nvidia-visible-devices-envvar-when-unprivileged = false
accept-nvidia-visible-devices-as-volume-mounts = true

2. Restart your container service for the above changes to take effect:

sudo systemctl restart containerd
OR
sudo systemctl restart docker

3. Create cluster with 1 worker node that has access to all GPUs (in my case, I had 4) and any volume-mounted files:

export KIND_CLUSTER_NAME=example-cluster

./nvkind cluster create \
--name=${KIND_CLUSTER_NAME} \
--config-template=examples/explicit-gpus-per-worker.yaml \
--config-values=- \
<<EOF
workers:
- devices: [0,1,2,3]
EOF

kubectl label node "${KIND_CLUSTER_NAME}-worker"
  
./nvkind cluster print-gpus

4. Spin up k8s-device-plugin pods inside worker node:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update

helm upgrade -i \
  --kube-context=kind-${KIND_CLUSTER_NAME} \
  --namespace nvidia \
  --set gfd.enabled=true \
  --set runtimeClassName=nvidia \
  --set deviceListStrategy=volume-mounts \
  --set deviceDiscoveryStrategy=nvml \
  --create-namespace \
  nvidia-device-plugin nvdp/nvidia-device-plugin

kubectl --context=kind-${KIND_CLUSTER_NAME} get pod -n nvidia
kubectl --context=kind-${KIND_CLUSTER_NAME} get nodes -o json | jq -r '.items[] | select(.metadata.name | test("-worker[0-9]*$")) | {name: .metadata.name, "nvidia.com/gpu": .status.allocatable["nvidia.com/gpu"]}'

5. If you run kubectl describe node example-cluster-worker before and after spinning up k8-device-plugin pods, you will see that nvidia.com/gpu appears under 'Allocatable' after the device plugin pods are live.

6. Deploy a pod workload to the node. This will start up your custom docker container inside a pod, inside the node.

cat << EOF | kubectl --context=kind-${KIND_CLUSTER_NAME} apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  runtimeClassName: nvidia
  restartPolicy: OnFailure
  containers:
  - name: example-container
    image: your.docker.hub.repo.subdomain/your.docker.image.name:latest
    volumeMounts:
    - mountPath: /path/inside/container
      name: my-host-volume
    ports:
    - containerPort: 8000
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: "all"
    - name: ANY_ENV_ARGS_U_WANT
      value: "x=1"
    resources:
      limits:
        nvidia.com/gpu: 4
  volumes:
  - name: my-host-volume
    hostPath:
      path: /path/inside/node
      type: Directory
EOF

kubectl describe pod example-pod
kubectl logs example-pod -n default -c example-container

7. Forward the port so you can access the endpoint
   kubectl port-forward <pod-name> 8000:8000 and can reach at curl -X GET http://localhost:8000/.....

PS: This was the content of my examples/explicit-gpus-per-worker.yaml file that we get from the nvkind repo:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
{{- if hasKey $ "name" }}
name: {{ $.name }}
{{- end }}
nodes:
- role: control-plane
  extraLabels:
    nvidia.com/gpu.present: "true"
  {{- if hasKey $ "image" }}
  image: {{ $.image }}
  {{- end }}
{{- range $.workers }}
- role: worker
  extraLabels:
    nvidia.com/gpu.present: "true"
  {{- if hasKey $ "image" }}
  image: {{ $.image }}
  {{- end }}

  {{- if hasKey . "devices" }}
  {{- $devices := .devices }}
  {{- if not (kindIs "slice" $devices) }}
    {{- $devices = list .devices }}
  {{- end }}
  extraMounts:
    # We inject all NVIDIA GPUs using the nvidia-container-runtime.
    # This requires `accept-nvidia-visible-devices-as-volume-mounts = true` be set
    # in `/etc/nvidia-container-runtime/config.toml`
    {{- range $d := $devices }}
    - hostPath: /dev/null
      containerPath: /var/run/nvidia-container-devices/{{ $d }}
    {{- end }}
    - hostPath: /path/inside/host
      containerPath: /path/inside/node
  {{- end }}
{{- end }}

[jacobtomlinson]

Was accept-nvidia-visible-devices-envvar-when-unprivileged required to get this working? The README already covers enabling accept-nvidia-visible-devices-as-volume-mounts = true but maybe we need to mentioned the other option too.

[geraldstanje]

@aoyshi @jacobtomlinson i get this warning - idea idea?

install device plugin:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
--kube-context=kind-${KIND_CLUSTER_NAME} \
  --namespace nvidia \
  --set gfd.enabled=true \
  --set runtimeClassName=nvidia \
  --set deviceListStrategy=volume-mounts \
  --set deviceDiscoveryStrategy=nvml \
  --create-namespace \
  nvidia-device-plugin nvdp/nvidia-device-plugin"nvdp" already exists with the same configuration, skipping
⚡ main ~/nvkind helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "nvdp" chart repository
...Successfully got an update from the "nvidia" chart repository
Update Complete. ⎈Happy Helming!⎈
⚡ main ~/nvkind 
⚡ main ~/nvkind helm upgrade -i \
>   --kube-context=kind-${KIND_CLUSTER_NAME} \
>   --namespace nvidia \
>   --set gfd.enabled=true \
>   --set runtimeClassName=nvidia \
>   --set deviceListStrategy=volume-mounts \
>   --set deviceDiscoveryStrategy=nvml \
>   --create-namespace \
>   nvidia-device-plugin nvdp/nvidia-device-plugin
Release "nvidia-device-plugin" does not exist. Installing it now.
W0321 13:22:49.457414   43297 warnings.go:70] spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[0].preference.matchExpressions[0].key: node-role.kubernetes.io/master is use "node-role.kubernetes.io/control-plane" instead
NAME: nvidia-device-plugin
LAST DEPLOYED: Fri Mar 21 13:22:48 2025
NAMESPACE: nvidia
STATUS: deployed
REVISION: 1
TEST SUITE: None

my worker node after install of the plugin - should there be Allocatable somewhere?

kubectl describe node example-cluster-worker
Name:               example-cluster-worker
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    feature.node.kubernetes.io/cpu-cpuid.ADX=true
                    feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX512BW=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX512CD=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX512DQ=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX512F=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX512VL=true
                    feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI=true
                    feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8=true
                    feature.node.kubernetes.io/cpu-cpuid.FMA3=true
                    feature.node.kubernetes.io/cpu-cpuid.FXSR=true
                    feature.node.kubernetes.io/cpu-cpuid.FXSROPT=true
                    feature.node.kubernetes.io/cpu-cpuid.HYPERVISOR=true
                    feature.node.kubernetes.io/cpu-cpuid.LAHF=true
                    feature.node.kubernetes.io/cpu-cpuid.MOVBE=true
                    feature.node.kubernetes.io/cpu-cpuid.MPX=true
                    feature.node.kubernetes.io/cpu-cpuid.OSXSAVE=true
                    feature.node.kubernetes.io/cpu-cpuid.SYSCALL=true
                    feature.node.kubernetes.io/cpu-cpuid.SYSEE=true
                    feature.node.kubernetes.io/cpu-cpuid.X87=true
                    feature.node.kubernetes.io/cpu-cpuid.XGETBV1=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVE=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVEC=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT=true
                    feature.node.kubernetes.io/cpu-cpuid.XSAVES=true
                    feature.node.kubernetes.io/cpu-hardware_multithreading=true
                    feature.node.kubernetes.io/cpu-model.family=6
                    feature.node.kubernetes.io/cpu-model.id=85
                    feature.node.kubernetes.io/cpu-model.vendor_id=Intel
                    feature.node.kubernetes.io/kernel-version.full=5.15.0-1077-aws
                    feature.node.kubernetes.io/kernel-version.major=5
                    feature.node.kubernetes.io/kernel-version.minor=15
                    feature.node.kubernetes.io/kernel-version.revision=0
                    feature.node.kubernetes.io/pci-10de.present=true
                    feature.node.kubernetes.io/pci-1d0f.present=true
                    feature.node.kubernetes.io/storage-nonrotationaldisk=true
                    feature.node.kubernetes.io/system-os_release.ID=debian
                    feature.node.kubernetes.io/system-os_release.VERSION_ID=12
                    feature.node.kubernetes.io/system-os_release.VERSION_ID.major=12
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=example-cluster-worker
                    kubernetes.io/os=linux
                    role=worker
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    nfd.node.kubernetes.io/feature-labels:
                      cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.AVX512BW,cpu-cpuid.AVX512CD,cpu-cpuid.AVX512DQ,cpu-cpuid.AVX512F,cpu-...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 21 Mar 2025 13:15:14 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  example-cluster-worker
  AcquireTime:     <unset>
  RenewTime:       Fri, 21 Mar 2025 13:23:35 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 21 Mar 2025 13:23:24 +0000   Fri, 21 Mar 2025 13:15:14 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 21 Mar 2025 13:23:24 +0000   Fri, 21 Mar 2025 13:15:14 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 21 Mar 2025 13:23:24 +0000   Fri, 21 Mar 2025 13:15:14 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 21 Mar 2025 13:23:24 +0000   Fri, 21 Mar 2025 13:15:28 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.18.0.2
  Hostname:    example-cluster-worker
Capacity:
  cpu:                8
  ephemeral-storage:  431423496Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32387576Ki
  nvidia.com/gpu:     1
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  431423496Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32387576Ki
  nvidia.com/gpu:     1
  pods:               110
System Info:
  Machine ID:                 f9a89173300146df8cfed0213ab26cb4
  System UUID:                67403dd5-40e8-45ed-b64d-eacab7fc776a
  Boot ID:                    096f2740-1c3e-45e8-976a-707c46e5063b
  Kernel Version:             5.15.0-1077-aws
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://2.0.2
  Kubelet Version:            v1.32.2
  Kube-Proxy Version:         v1.32.2
PodCIDR:                      10.244.1.0/24
PodCIDRs:                     10.244.1.0/24
ProviderID:                   kind://docker/example-cluster/example-cluster-worker
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
  kube-system                 kindnet-bff4j                                               100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      8m28s
  kube-system                 kube-proxy-rvzts                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         8m28s
  nvidia                      nvidia-device-plugin-fpmlk                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         33s
  nvidia                      nvidia-device-plugin-gpu-feature-discovery-dqn9n            0 (0%)        0 (0%)      0 (0%)           0 (0%)         33s
  nvidia                      nvidia-device-plugin-node-feature-discovery-worker-q56lr    0 (0%)        0 (0%)      0 (0%)           0 (0%)         53s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (1%)  100m (1%)
  memory             50Mi (0%)  50Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
  nvidia.com/gpu     0          0
Events:
  Type     Reason                             Age                    From             Message
  ----     ------                             ----                   ----             -------
  Normal   Starting                           8m25s                  kube-proxy       
  Warning  PossibleMemoryBackedVolumesOnDisk  8m29s                  kubelet          The tmpfs noswap option is not supported. Memory-backed volumes (e.g. secrets, emptyDirs, etc.) might be swapped to disk and should no longer be considered secure.
  Normal   Starting                           8m29s                  kubelet          Starting kubelet.
  Normal   NodeHasSufficientMemory            8m28s (x2 over 8m28s)  kubelet          Node example-cluster-worker status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure              8m28s (x2 over 8m28s)  kubelet          Node example-cluster-worker status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID               8m28s (x2 over 8m28s)  kubelet          Node example-cluster-worker status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced            8m28s                  kubelet          Updated Node Allocatable limit across pods
  Normal   RegisteredNode                     8m24s                  node-controller  Node example-cluster-worker event: Registered Node example-cluster-worker in Controller
  Normal   NodeReady                          8m14s                  kubelet          Node example-cluster-worker status is now: NodeReady

here the control plane node:

kubectl describe node example-cluster-control-plane
Name:               example-cluster-control-plane
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=example-cluster-control-plane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 21 Mar 2025 13:14:59 +0000
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  example-cluster-control-plane
  AcquireTime:     <unset>
  RenewTime:       Fri, 21 Mar 2025 13:28:05 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 21 Mar 2025 13:23:11 +0000   Fri, 21 Mar 2025 13:14:58 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 21 Mar 2025 13:23:11 +0000   Fri, 21 Mar 2025 13:14:58 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 21 Mar 2025 13:23:11 +0000   Fri, 21 Mar 2025 13:14:58 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 21 Mar 2025 13:23:11 +0000   Fri, 21 Mar 2025 13:15:21 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.18.0.3
  Hostname:    example-cluster-control-plane
Capacity:
  cpu:                8
  ephemeral-storage:  431423496Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32387576Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  431423496Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32387576Ki
  pods:               110
System Info:
  Machine ID:                 1dc45295d41b4f18b3b1bc8b4fb00a11
  System UUID:                1a39486c-8e7b-495c-a998-a85e4c7af500
  Boot ID:                    096f2740-1c3e-45e8-976a-707c46e5063b
  Kernel Version:             5.15.0-1077-aws
  OS Image:                   Debian GNU/Linux 12 (bookworm)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://2.0.2
  Kubelet Version:            v1.32.2
  Kube-Proxy Version:         v1.32.2
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
ProviderID:                   kind://docker/example-cluster/example-cluster-control-plane
Non-terminated Pods:          (10 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-668d6bf9bc-pc2wj                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (0%)     13m
  kube-system                 coredns-668d6bf9bc-ztqkd                                           100m (1%)     0 (0%)      70Mi (0%)        170Mi (0%)     13m
  kube-system                 etcd-example-cluster-control-plane                                 100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         13m
  kube-system                 kindnet-mmgd6                                                      100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      13m
  kube-system                 kube-apiserver-example-cluster-control-plane                       250m (3%)     0 (0%)      0 (0%)           0 (0%)         13m
  kube-system                 kube-controller-manager-example-cluster-control-plane              200m (2%)     0 (0%)      0 (0%)           0 (0%)         13m
  kube-system                 kube-proxy-jd65k                                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m
  kube-system                 kube-scheduler-example-cluster-control-plane                       100m (1%)     0 (0%)      0 (0%)           0 (0%)         13m
  local-path-storage          local-path-provisioner-7dc846544d-zkk59                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         13m
  nvidia                      nvidia-device-plugin-node-feature-discovery-master-77b96dd5nzg4    0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m25s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                950m (11%)  100m (1%)
  memory             290Mi (0%)  390Mi (1%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                             Age                From             Message
  ----     ------                             ----               ----             -------
  Normal   Starting                           13m                kube-proxy       
  Normal   NodeHasSufficientMemory            13m (x8 over 13m)  kubelet          Node example-cluster-control-plane status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure              13m (x8 over 13m)  kubelet          Node example-cluster-control-plane status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID               13m (x7 over 13m)  kubelet          Node example-cluster-control-plane status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced            13m                kubelet          Updated Node Allocatable limit across pods
  Normal   Starting                           13m                kubelet          Starting kubelet.
  Normal   NodeAllocatableEnforced            13m                kubelet          Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory            13m                kubelet          Node example-cluster-control-plane status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure              13m                kubelet          Node example-cluster-control-plane status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID               13m                kubelet          Node example-cluster-control-plane status is now: NodeHasSufficientPID
  Warning  PossibleMemoryBackedVolumesOnDisk  13m                kubelet          The tmpfs noswap option is not supported. Memory-backed volumes (e.g. secrets, emptyDirs, etc.) might be swapped to disk and should no longer be considered secure.
  Normal   RegisteredNode                     13m                node-controller  Node example-cluster-control-plane event: Registered Node example-cluster-control-plane in Controller
  Normal   NodeReady                          12m                kubelet          Node example-cluster-control-plane status is now: NodeReady

check pods after install of device plugin:

kubectl --context=kind-${KIND_CLUSTER_NAME} get pod -n nvidia
NAME} get nodes -o json | jq -r '.items[] | select(.metadata.name | test("-worker[0-9]*$")) | {name: .metadata.name, "nvidia.com/gpu": .status.allocatable["nvidia.com/gpu"]}'NAME                                                              READY   STATUS    RESTARTS       AGE
nvidia-device-plugin-fpmlk                                        1/1     Running   0              9m1s
nvidia-device-plugin-gpu-feature-discovery-dqn9n                  1/1     Running   0              9m1s
nvidia-device-plugin-node-feature-discovery-master-77b96dd5nzg4   1/1     Running   0              9m21s
nvidia-device-plugin-node-feature-discovery-worker-q56lr          1/1     Running   1 (9m2s ago)   9m21s

⚡ main ~/nvkind kubectl --context=kind-${KIND_CLUSTER_NAME} get nodes -o json | jq -r '.items[] | select(.metadata.name | test("-worker[0-9]*
$")) | {name: .metadata.name, "nvidia.com/gpu": .status.allocatable["nvidia.com/gpu"]}'
{
  "name": "example-cluster-worker",
  "nvidia.com/gpu": "1"
}

[dveselov]

I'm experiencing the same errors on my RTX 3060, but I managed to get it working correctly by using the deviceListStrategy=envvar.