Skip to content

Fix pull secret injection timing for caas flavor - #26

Open
akshaynadkarni wants to merge 1 commit into
osac-project:mainfrom
akshaynadkarni:fix-pull-secret-injection
Open

Fix pull secret injection timing for caas flavor#26
akshaynadkarni wants to merge 1 commit into
osac-project:mainfrom
akshaynadkarni:fix-pull-secret-injection

Conversation

@akshaynadkarni

Copy link
Copy Markdown

Summary

Fixes two issues preventing the caas flavor from booting successfully.

Why

The caas flavor has no baked-in pull secret and includes HyperShift webhook
certificates that recert doesn't handle. These issues don't affect the vmaas
flavor, which is why they weren't caught earlier (caas is not in CI yet).

Changes

Fix 1: Pull secret injection before recert

The --pull-secret flag writes to /var/lib/kubelet/config.json at step 6
(after recert), but the etcd container at step 5 uses --authfile on that
same path. When the flavor has no baked-in pull secret, the etcd image pull
fails with unauthorized.

This moves the pull secret file write to after CRI-O is stopped but before
the etcd container starts.

Fix 2: HyperShift webhook CA key for recert

If a hypershift-webhook-ca.key file exists in the flavor's crypto/
directory, it is passed to recert via --use-key. This is needed for the
caas flavor which includes HyperShift webhook certificates.

Known Limitations

Fix 2 alone is not sufficient. Recert fails with "failed to find matching
SKID method" for the HyperShift webhook CA, even when the correct key is
provided. This appears to be a recert bug with the SKID computation method
used by the HyperShift CA certificate.

A workaround that partially works: delete the HyperShift webhook resources
from etcd before running recert (webhook-serving-ca, manager-serving-cert,
and the webhook configurations). Recert then succeeds, but the kube-apiserver
fails to start afterward (likely due to inconsistent cluster state from the
manual etcd deletions).

The proper fix likely requires either:

  1. A recert update to handle the HyperShift SKID format
  2. The snapshot process extracting the HyperShift CA key in a format recert
    can use
  3. A different approach in the boot process for handling HyperShift certs

Testing

  • Fix 1 verified: etcd image pull succeeds with the pull secret injected early
  • Fix 2 verified: the --use-key flag is correctly added to the recert command
    when the key file exists on the server
  • Full caas boot: blocked by the recert SKID bug (see Known Limitations)
  • vmaas boot: not affected by these changes (no HyperShift, pull secret
    typically baked in)

Signed-off-by: akshaynadkarni 25892229+akshaynadkarni@users.noreply.github.com
Assisted-by: Cursor/Claude

The --pull-secret flag writes to /var/lib/kubelet/config.json at step 6
(after recert), but the etcd container at step 5 uses --authfile on that
same path to pull the etcd image. When the flavor has no baked-in pull
secret (e.g. caas), the pull fails with "unauthorized".

This moves pull secret resolution and file injection to before the etcd
podman run, writing the pull secret after CRI-O is stopped but before
the etcd container starts.

Additionally, if a hypershift-webhook-ca.key file exists in the flavor's
crypto directory, it is passed to recert via --use-key. This is needed
for the caas flavor which includes HyperShift webhook certificates that
recert must re-sign.

Note: the HyperShift --use-key fix alone is not sufficient due to a
recert SKID method bug. The full caas fix requires changes in either
recert or the snapshot process. See the PR description for details.

Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com>
Assisted-by: Cursor/Claude
Signed-off-by: akshaynadkarni <25892229+akshaynadkarni@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant