diff --git a/docs/design/ecr-image-migration.md b/docs/design/ecr-image-migration.md new file mode 100644 index 000000000..26c791608 --- /dev/null +++ b/docs/design/ecr-image-migration.md @@ -0,0 +1,267 @@ +# ECR Image Migration: Moving Platform and HCP Images Away From Quay + +**Last Updated Date**: 2026-06-25 + +## Summary + +Migrate all images consumed by the ROSA Regional Platform — both the Hosted Control Plane (HCP) +dataplane images and the platform operational images (kubeapplier, hyperfleet-operator, etc.) — +from Quay.io to Amazon ECR. OCP images already mirrored to ECR by the OpenShift release process +are reused directly. Platform images built via Konflux are mirrored into the same regional ECR +repositories using the existing OCP mirroring infrastructure, extended to cover platform image +sets. A unified IAM authentication path covers both image sets. + +## Context + +- **Problem Statement**: Zero-egress Hosted Control Plane environments cannot + reach Quay.io at all. OCP already publishes a mirrored copy of its release images to regional + ECR repositories to satisfy this requirement. The platform should reuse those mirrored images + rather than maintaining a separate mirroring pipeline or continuing to depend on public Quay + endpoints. + + As part of reducing and minimizing dependencies on infrastructure outside of AWS, the intention + is to move all images required to run the platform to ECR as opposed to Quay. + +- **Constraints**: + - The HCP dataplane images must be sourced from ECR to support zero-egress control plane + environments — however, the `--zero-egress` flag in HyperShift carries side effects beyond + image sourcing (e.g., restricting network path modifications) that are undesirable for standard + Hosted Clusters. ECR image sourcing must therefore be achieved without setting this flag. + - Platform images (kubeapplier, hyperfleet-operator, and others) must be available in the same + regional ECR repositories as OCP images so that MC/RC workloads can pull them without public + internet egress. + - The authentication mechanism for ECR must cover both image sets (OCP HCP dataplane images and + platform operational images) using a single, unified approach. + - Konflux is the build system for platform images; the mirroring pipeline will use the existing + mirroring infrastructure. + +- **Assumptions**: + - OCP continues to mirror its release images to regional ECR repositories as part of the standard + release process. The exact repository paths and tagging conventions for those mirrors are + treated as stable inputs. + - The existing OCP ECR mirroring infrastructure (tooling, IAM roles, regional replication + configuration) can be extended to include platform image sets without a full rewrite. + - ECR private repositories in the relevant AWS accounts and regions are, or can be, accessible + to the IAM identities associated with MC and RC node pools and control plane pods. + - Authentication to ECR for image pulls — particularly for HCP dataplane pods running inside + Hosted Clusters — is achievable, but the exact mechanism requires investigation (see + [Authentication](#authentication) below). + +## Alternatives Considered + +1. **Mirror images to a self-managed registry (e.g., a registry running on the RC)**: Gives full + control over availability but introduces significant operational burden: registry lifecycle + management, storage provisioning, replication, and a new failure domain. Rejected. + +2. **Use ECR for dataplane images only, retain Quay for infra images**: Introduces two + paths for obtaining images. Rejected. + +## Design + +### Image Set Breakdown + +There are two distinct sets of images that need to move to ECR: + +#### Set 1: HCP Dataplane Images + +These are the OCP component images that run inside the Hosted Control Plane (API server, +etcd, ingress operator, etc.). They are built by the OCP release process on Quay and are already +mirrored by OCP to regional ECR repositories as part of the standard release pipeline. + +The goal is to configure HyperShift to always pull these images from ECR rather than from Quay, +regardless of whether the Hosted Cluster is in a zero-egress environment. + +#### Set 2: Platform Operational Images + +These are images required to operate the Regional Platform itself — running on MCs and RCs. +Examples include: + +- `kubeapplier` +- `hyperfleet-operator` +- Other platform controllers and tooling + +These images are built via Konflux and are currently sourced from Quay at runtime. They need to +be mirrored to ECR and pulled from there instead. + +### HyperShift Configuration: ECR Without `--zero-egress` + +HyperShift's `--zero-egress` flag was designed for environments with no public internet access. +Setting it causes HyperShift to source control plane images from ECR mirrors, but it also applies +network path restrictions and other behavioural changes that are undesirable for standard Hosted +Clusters (e.g., it may restrict outbound network paths that are expected to be available in a +normal hosted cluster). + +Since the goal is to use ECR for image sourcing universally — not just in zero-egress environments +— an alternative mechanism is needed to force HyperShift to always pull from ECR. The specific +approach is still being determined, but candidate mechanisms include: + +- Configuring HyperShift's image override mechanism (e.g., via `ImageContentSources` or + `ImageDigestMirrorSet` on the management cluster) to redirect Quay image references to their + ECR equivalents. +- Providing a custom release image or image lookup override in the `HostedCluster` spec that + points directly to ECR-hosted digests. +- Patching HyperShift's image resolution logic to treat ECR as the primary source unconditionally. + +The chosen approach must ensure that HyperShift resolves all control plane images to ECR without +activating the broader `--zero-egress` mode. This will require investigation and likely +coordination with the HyperShift upstream team. + +### Mirroring: Extending the OCP Pipeline for Platform Images + +OCP already operates a mirroring pipeline that replicates release images from Quay to regional ECR +repositories. Rather than building a separate mirroring mechanism for platform images, the existing +pipeline will be extended to include platform image sets. + +The extension will: + +1. Accept platform image lists (produced by Konflux build outputs) as an additional input to the + mirroring configuration. +2. Mirror platform images to the same regional ECR structure used for OCP images, so that the same + repository conventions and IAM access patterns apply. +3. Replicate to all regions where OCP images are currently mirrored, maintaining geographic + consistency. + +The specifics of how Konflux build outputs are fed into the mirroring pipeline (e.g., via image +manifests, SBOMs, or a dedicated image list artifact) need to be defined as part of implementation. + +### Authentication + +Authentication to ECR for image pulls is required in two contexts: + +1. **MC/RC node pools and system components**: EKS nodes pulling platform operational images from + ECR. This is relatively straightforward — EKS node IAM roles can be granted: + - `ecr:GetAuthorizationToken` with `Resource: "*"` (this action is account-level and cannot be + scoped to specific repositories; attempting to restrict it to a repository ARN results in an + `AccessDeniedException`). + - `ecr:BatchGetImage` and `ecr:GetDownloadUrlForLayer` with `Resource` limited to the specific + repository ARNs for platform and OCP images (least-privilege, repository-scoped). + + The kubelet handles token refresh automatically via the ECR credential provider plugin (enabled + by default on EKS). + +2. **HCP dataplane pods inside Hosted Clusters**: Pods running inside Hosted Clusters need to pull + images from ECR. Static `imagePullSecrets` are not suitable here because ECR authorization + tokens expire after 12 hours, causing image pull failures on any new pod start after token + expiry. + + The preferred approach is a dynamic credential mechanism such as AWS Pod Identity (EKS Pod + Identity Associations) or IRSA (IAM Roles for Service Accounts), which allows the HyperShift + Operator to obtain short-lived ECR tokens on demand without operator intervention. An + investigation should determine which mechanism integrates most cleanly with HyperShift's + existing image pull handling (e.g., whether HyperShift can refresh and inject a rotating ECR + token into the control plane namespace, or whether a sidecar credential rotation controller is + required). The goal is that a single IAM identity or dynamic pull mechanism covers both OCP HCP + dataplane images and platform operational images from ECR, with no manual token rotation. + +```mermaid +flowchart LR + subgraph MC_RC["MC / RC Nodes"] + kubelet["kubelet\n(node IAM role)"] + end + + subgraph HCP["HCP Control Plane Pods"] + hcp_pods["etcd / apiserver\ningress / etc"] + end + + subgraph ECR["ECR (regional)"] + ocp_imgs["OCP images\nPlatform images"] + end + + kubelet -->|"ECR credential provider\n(auto token refresh)"| ocp_imgs + hcp_pods -->|"HyperShift dynamic auth\n(Pod Identity / IRSA)"| ocp_imgs +``` + +## Consequences + +### Positive + +- Eliminates runtime dependency on Quay.io for both HCP dataplane images and platform operational + images, reducing infra dependencies. +- Platform images gain the same regional redundancy and proximity as OCP images, reducing pull + latency and transfer costs. +- A single, unified auth mechanism for ECR simplifies credential management and reduces the number + of distinct IAM patterns to maintain. +- Reusing the existing OCP mirroring pipeline for platform images avoids duplicating mirroring + infrastructure and keeps the operational model consistent. + +### Negative + +- The ECR mirroring pipeline becomes a critical path dependency: if mirroring fails, new image + versions will not be available in ECR. The existing OCP pipeline presumably has reliability + requirements; extending it to cover platform images increases its blast radius. +- ECR repository provisioning (IAM policies, lifecycle rules, replication configuration) must be + managed per region and per image set, adding Terraform/IaC surface area. +- Forcing HyperShift to pull from ECR without `--zero-egress` requires either upstream changes or + a local override mechanism that may need to be maintained across HyperShift upgrades. + +## Cross-Cutting Concerns + +### Reliability + +- **Scalability**: ECR scales automatically. Regional replication ensures images are available + close to the clusters that need them. No capacity planning is required for the registry itself. +- **Observability**: ECR pull failures surface as pod image pull errors (`ImagePullBackOff`). + Alerting should be added for sustained pull failure rates on both MCs/RCs and HCP control plane + namespaces. Mirroring pipeline failures should produce actionable alerts before they affect + running clusters (images already pulled are cached on nodes). +- **Resiliency**: Nodes cache pulled images locally. A transient ECR outage will not affect already + running pods, only new pod starts or node replacements. Mirroring pipeline failures need to be + caught before node rotation forces re-pulls of images that are no longer available in ECR. + +### Security + +- ECR repositories should be private with resource-based policies restricting access to known AWS + account IDs and IAM roles. Cross-account pulls (e.g., Hosted Cluster node roles in customer + accounts) may require explicit repository policy statements. +- Leverage existing vulnerability visibility through existing platform mechanisms. +- The auth mechanism for HCP dataplane pods must not grant overly broad ECR permissions. Least + privilege applies: pull access to specific repositories only, not `ecr:*` on all repositories. +- Short-lived ECR tokens (12-hour expiry) reduce credential exposure compared to long-lived Quay + robot account tokens. + +### Cost + +- ECR charges for storage and data transfer. Platform images are small relative to OCP release + images; the incremental storage cost of adding platform images to the mirroring pipeline is + expected to be low. +- Pulling images from ECR within the same region incurs no data transfer charges. Cross-region + pulls are charged; regional replication ensures pulls are always intra-region. +- Lifecycle policies should be configured on ECR repositories to expire old image tags and control + storage growth. + +### Operability + +- The mirroring pipeline extension must be automated. Manual mirroring of platform images is not + acceptable at the cadence of Konflux builds. +- ECR credential rotation for MCs/RCs is handled automatically by the EKS ECR credential provider. + The HCP dataplane auth mechanism must similarly not require manual intervention for token refresh. +- Repository provisioning (new platform image repositories) should be Terraform-managed and follow + the same conventions as existing OCP ECR repositories. +- Runbooks are needed for: mirroring pipeline failure, ECR auth failures on HCP pods, and adding + new platform images to the mirroring configuration. + +## Open Questions / Areas Requiring Investigation + +1. **HCP dataplane auth**: What is the correct mechanism to provide ECR credentials to pods inside + a Hosted Cluster's control plane namespace on the management cluster? Can HyperShift's pull + secret handling be extended, or is a credential injection controller required? + +2. **HyperShift ECR override without `--zero-egress`**: What is the minimal change required to + make HyperShift resolve all control plane images from ECR by default? Is this achievable via + `ImageDigestMirrorSet` on the management cluster, or does it require changes to HyperShift's + image resolution code? + +3. **OCP mirroring pipeline extension**: What is the interface for adding platform image sets to + the existing OCP mirroring pipeline? Who owns that pipeline, and what is the process for + contributing extensions? + +4. **Repository naming conventions**: Platform images will be mirrored into the same regional ECR + repository structure as OCP images (this is a constraint established in the design). The open + question is how to namespace them within that structure — for example, should platform images + share the same repository namespace as OCP release images, or live under a distinct + platform-specific prefix within the same ECR registry? The answer affects IAM policy scoping + (whether a single repository ARN pattern covers both sets) and lifecycle rule management. + +5. **Cross-account ECR access for Hosted Cluster nodes**: If Hosted Cluster nodes run in customer + AWS accounts and need to pull from platform-account ECR repositories, what cross-account IAM + setup is required? Is this handled by HyperShift, or does the platform need to manage it?