no runtimeclasses in OCP CI pipeline since ~2025-12-20

[ldoktor]

Describe the bug

The https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-kata-containers-kata-containers-main-peer-pods-e2e-tests ocp ci pipeline stopped working between 2025-12-14 (last good; https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-kata-containers-kata-containers-main-peer-pods-e2e-tests/2000098950161371136/artifacts/e2e-tests/kata-containers-e2e-pre/build-log.txt ) and 2025-12-20 (first bad; https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-kata-containers-kata-containers-main-peer-pods-e2e-tests/2002273411329429504/artifacts/e2e-tests/kata-containers-e2e-pre/build-log.txt ).

When I tried reproducing the last-good it failed anyway (likely some "latest" image of the development versions) but reproducing with released versions showed v0.16.0 as working and v0.17.0 as non-working.

To Reproduce
Steps to reproduce the behavior:

1. get a temporary azure OCP cluster with az credentials configured (beware, this workflow creates new resource groups and other resources so ensure it's temporary and will be cleaned-up afterwards somehow)
2. git clone https://github.com/kata-containers/kata-containers/
3. bash -x ci/openshift-ci/peer-pods-azure.sh

Describe the results you expected
It should report pass

Describe the results you received:
It reports that kata-remote runtimeclass was not configured in 360s

Additional context
I tried using older commits directly by setting OPERATOR_SHA/CAA_GIT_SHA, but the development version often points to "latest" images which resulted in failure anyway. Using released versions helped pin-pointing the breakage likely occurred between v0.16.0 and v0.17.0 as:

CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor" CAA_TAG=v0.16.0 PP_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/2025.12.14" CAA_GIT_SHA="v0.16.0" OPERATOR_SHA=v0.16.0 bash -x ci/openshift-ci/peer-pods-azure.sh

worked (oc get runtimeclasses reports kata-remote), but:

CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor" CAA_TAG=v0.17.0 PP_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/2025.12.14" CAA_GIT_SHA="v0.17.0" OPERATOR_SHA=v0.17.0 bash -x ci/openshift-ci/peer-pods-azure.sh

doesn't.

[stevenhorsman]

So the coco operator was broken (deliberately) around that date, so I'm not surprised that it fails with latest, but we have been using the pinned 0.17 release in peer pod's CI and it seems to be okay, so that not working for you is more of a concern

[ldoktor]

I see, the latest is definitely broken but I have to double-check the v0.17.0 is bad as well. I used a single cluster which I kept cleaning by:

kubectl delete -k cloud-api-adaptor/src/cloud-api-adaptor/install/overlays/azure/
kubectl delete -k cloud-api-adaptor/src/cloud-api-adaptor/operator/config/samples/ccruntime/peer-pods/
kubectl delete -k cloud-api-adaptor/src/cloud-api-adaptor/operator/config/release/
oc label node --all confidentialcontainers.org/preinstall-

After several iterations it ended-up useless so the question is whether v0.17.0 failed because of that or because of actual failure. I need to leave now but I'll give it another round on fresh cluster when I return (in about 6-9h)

[ldoktor]

ok so the v0.17.0 PASSed on a fresh cluster, but the main version didn't.

You mentioned the operator pinning, are you talking about confidential-containers/cloud-api-adaptor@fb399e7 ? So to replicate that in my CI I need to use the v0.17.0 operator with the latest CAA?

Side question, do you know what is the proper way of uninstalling CAA/operator? My approach:

kubectl delete -k cloud-api-adaptor/src/cloud-api-adaptor/install/overlays/azure/
# WAIT FOR UNINSTALL TASKS
kubectl delete -k cloud-api-adaptor/src/cloud-api-adaptor/operator/config/samples/ccruntime/peer-pods/
# WAIT FOR UNINSTALL TASKS
kubectl delete -k cloud-api-adaptor/src/cloud-api-adaptor/operator/config/release/
# WAIT FOR UNINSTALL TASKS
oc label node --all confidentialcontainers.org/preinstall-

worked for some cases but usually the next deployment following the same scripts result in:

NAME                                    READY   STATUS    RESTARTS   AGE
pod/cloud-api-adaptor-daemonset-46vxn   1/1     Running   0          96s
pod/cloud-api-adaptor-daemonset-6dkcq   1/1     Running   0          96s
pod/cloud-api-adaptor-daemonset-r8vh8   1/1     Running   0          96s

NAME                                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
   AGE
service/cc-operator-controller-manager-metrics-service   ClusterIP   172.30.96.243   <none>	   8443/TCP
   110s

NAME                                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELE
CTOR                AGE
daemonset.apps/cloud-api-adaptor-daemonset   3         3         3	 3            3           node.kube
rnetes.io/worker=   97s

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cc-operator-controller-manager   0/1     0            0           110s

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/cc-operator-controller-manager-7f996bcccd   1         0         0	110s

not sure what I omitted in the cleanup and getting a new cluster to re-try is quite slow and ugly.

[stevenhorsman]

ok so the v0.17.0 PASSed on a fresh cluster, but the main version didn't.

ok good, so you are seeing the expected behaviour

You mentioned the operator pinning, are you talking about confidential-containers/cloud-api-adaptor@fb399e7 ? So to replicate that in my CI I need to use the v0.17.0 operator with the latest CAA?

Yes that's correct

Side question, do you know what is the proper way of uninstalling CAA/operator? My approach:

That seems pretty similar to what we do in our e2e tests but we also clean up peerpod-ctrl

[ldoktor]

Thank you, @stevenhorsman, I'll update the configuration. Is there any ETA for when the operator should be fixed again and is there an issue about that I can subscribe to to revert it to testing latest when it's resolved?

[stevenhorsman]

Thank you, @stevenhorsman, I'll update the configuration. Is there any ETA for when the operator should be fixed again and is there an issue about that I can subscribe to to revert it to testing latest when it's resolved?

The operator is EoL, so it won't be fixed as the community it moving to the charts approach instead

[fitzthum]

Before we archive the operator, it might be a good idea to set the tags back to v0.17.0 to reduce confusion.