GPU time on Kubernetes

[sashalarsoo]

Hey, i passed alphafold 2 on kubernetes on the same shape of host i was using to run with docker.
I observed for msa sequence (cpu TIME) was the same duration than running it on host.
But for GPU time i observed it took 3x times the time it took when running in the host. There is no IOPS issues since all is running from the hostpath.

Do you have any idea ?

[kushvinth]

Hi @sashalarsoo, The slower GPU performance on Kubernetes is likely due to GPU resource sharing or container runtime overhead. Please ensure:

• You’re using the NVIDIA container runtime (--runtime=nvidia) and the correct CUDA drivers.
• GPU is exclusively allocated to the pod (nvidia.com/gpu: 1).
• No throttling from k8 resource limits.

Performance should match bare metal once these are verified.

[NishikaGolani]

K8s GPU scheduling for AF inference has consistent overhead — 2–3× slowdown is what we measure too, mostly from CRI/runc cold-starts and the way K8s device plugins serialize CUDA initialization. Slurm's --gres=gpu:N skips that path entirely. We run AlphaFold on Slurm-on-Kubernetes (Slinky) where the Slurm scheduling path owns GPU allocation directly — closes the gap. Happy to share the AF2 Apptainer + sbatch template we ship.

[sashalarsoo]

Hello thanks for the answer, i would be interested by your code snippets.