metrics: use PodCertificate projection for the serving certificate once the feature gate is on by default

[slauger]

Idea

Kubernetes 1.34 introduced PodCertificateProjection: a pod requests a short-lived key and certificate through a projected volume, and the kubelet handles issuance and rotation. If that were available, the metrics endpoint (#510) could serve a verifiable certificate without cert-manager and without the operator generating its own.

volumes:
  - name: metrics-certs
    projected:
      sources:
        - podCertificate:
            signerName: <a signer that issues serving certificates>
            keyType: ECDSAP256
            credentialBundlePath: bundle.pem

Why not yet

Tested against the e2e cluster, Kubernetes v1.36.4+k3s1:

$ kubectl apply --dry-run=server -f podcert-probe.yaml
Warning: volume "certs" (Projected) has no sources provided
pod/podcert-probe created (server dry run)

The API server silently drops the source with the feature gate off. It does not reject the pod, so building on this today would produce an operator that starts without a certificate and reports nothing useful about why. The type exists in the vendored k8s.io/api, which makes it look available when it is not.

There is also an open question the probe does not answer: which signer. kubernetes.io/kube-apiserver-client issues client certificates, not serving ones, and kubernetes.io/kubelet-serving is restricted to kubelets by the approver. A cluster would need a signer that issues serving certificates for workloads, which is not part of stock Kubernetes.

What to do when picking this up

• Check whether the gate is on by default in the minimum supported Kubernetes version
• Establish which signer is expected to issue workload serving certificates, and whether that is portable across distributions
• Detect availability at runtime rather than assuming it -- given the silent drop, absence has to be noticed by the operator rather than by the user
• Keep cert-manager and the self-signed path; this would be a third option, not a replacement

Related

#510 introduced the secure metrics endpoint with cert-manager as the default and a self-signed certificate as the fallback. This would remove the cert-manager dependency for clusters that support it.