Persistence Mode not set for GPU

[xpillons]

In CCWS version 2025.12.01 which uses CC 8.8.1, a new healthagent is always on and will monitor GPU and IB. For old GPU SKUs the following error message can be raised: Persistence Mode not set for GPU x
This is because recent GPU SKUs have this mode enabled and older may not.

How to Fix

This can be fixed by enabling the persistent mode in the cloud-init of the target partition using the nvidia-smi CLI.

#!/bin/bash
nvidia-smi -pm 1

[ossonts]

Is there a recommended way to add a cloud-init?

I have my own cloud init repo and added it to clusterInitSpecs with type github_cluster_init_t

[xpillons]

cloud-init can be manually added using the CycleCloud template editor.
Cluster-Init is different and will run after the heathagent. You can add the persistent mode in your cluster-init project but you may still see the error until your script had run. What is unclear for me is if the cluster-init scripts will be run despite the healthagent error. This need to be checked.

[ossonts]

ah yes, we dont use cloud-init, but we do have a cluster-init

is it just a case of adding to "GPUClusterInitSpecs"?

[xpillons]

@ossonts sorry I realized no one answered you. You can add in cluster init, but it will failed at least once because the health agent is running before cluster-init. But it will be retried and your script should be run.