From 6b754d7e9656ce7c245f15ace7c825ca34b1be54 Mon Sep 17 00:00:00 2001 From: Pete Savage Date: Thu, 25 Jun 2026 11:02:27 +0100 Subject: [PATCH 1/2] Added ECR design document Co-authored-by: claude-sonnet-4-6 --- docs/design/ecr-image-migration.md | 254 +++++++++++++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 docs/design/ecr-image-migration.md diff --git a/docs/design/ecr-image-migration.md b/docs/design/ecr-image-migration.md new file mode 100644 index 000000000..8bd00e93b --- /dev/null +++ b/docs/design/ecr-image-migration.md @@ -0,0 +1,254 @@ +# ECR Image Migration: Moving Platform and HCP Images Away From Quay + +**Last Updated Date**: 2026-06-25 + +## Summary + +Migrate all images consumed by the ROSA Regional Platform — both the Hosted Control Plane (HCP) +dataplane images and the platform operational images (kubeapplier, hyperfleet-operator, etc.) — +from Quay.io to Amazon ECR. OCP images already mirrored to ECR by the OpenShift release process +are reused directly. Platform images built via Konflux are mirrored into the same regional ECR +repositories using the existing OCP mirroring infrastructure, extended to cover platform image +sets. A unified IAM authentication path covers both image sets. + +## Context + +- **Problem Statement**: Zero-egress Hosted Control Plane environments cannot + reach Quay.io at all. OCP already publishes a mirrored copy of its release images to regional + ECR repositories to satisfy this requirement. The platform should reuse those mirrored images + rather than maintaining a separate mirroring pipeline or continuing to depend on public Quay + endpoints. + + As part of reducing and minimizing dependencies on infrastructure outside of AWS, the intention + is to move all images required to run the platform to ECR as opposed to Quay. + +- **Constraints**: + - The HCP dataplane images must be sourced from ECR to support zero-egress control plane + environments — however, the `--zero-egress` flag in HyperShift carries side effects beyond + image sourcing (e.g., restricting network path modifications) that are undesirable for standard + Hosted Clusters. ECR image sourcing must therefore be achieved without setting this flag. + - Platform images (kubeapplier, hyperfleet-operator, and others) must be available in the same + regional ECR repositories as OCP images so that MC/RC workloads can pull them without public + internet egress. + - The authentication mechanism for ECR must cover both image sets (OCP HCP dataplane images and + platform operational images) using a single, unified approach. + - Konflux is the build system for platform images; the mirroring pipeline will use the existing + mirroring infrastructure. + +- **Assumptions**: + - OCP continues to mirror its release images to regional ECR repositories as part of the standard + release process. The exact repository paths and tagging conventions for those mirrors are + treated as stable inputs. + - The existing OCP ECR mirroring infrastructure (tooling, IAM roles, regional replication + configuration) can be extended to include platform image sets without a full rewrite. + - ECR private repositories in the relevant AWS accounts and regions are, or can be, accessible + to the IAM identities associated with MC and RC node pools and control plane pods. + - Authentication to ECR for image pulls — particularly for HCP dataplane pods running inside + Hosted Clusters — is achievable, but the exact mechanism requires investigation (see + [Authentication](#authentication) below). + +## Alternatives Considered + +1. **Mirror images to a self-managed registry (e.g., a registry running on the RC)**: Gives full + control over availability but introduces significant operational burden: registry lifecycle + management, storage provisioning, replication, and a new failure domain. Rejected. + +2. **Use ECR for dataplane images only, retain Quay for infra images**: Introduces two + paths for obtaining images. Rejected. + +## Design + +### Image Set Breakdown + +There are two distinct sets of images that need to move to ECR: + +#### Set 1: HCP Dataplane Images + +These are the OCP component images that run inside the Hosted Control Plane (API server, +etcd, ingress operator, etc.). They are built by the OCP release process on Quay and are already +mirrored by OCP to regional ECR repositories as part of the standard release pipeline. + +The goal is to configure HyperShift to always pull these images from ECR rather than from Quay, +regardless of whether the Hosted Cluster is in a zero-egress environment. + +#### Set 2: Platform Operational Images + +These are images required to operate the Regional Platform itself — running on MCs and RCs. +Examples include: + +- `kubeapplier` +- `hyperfleet-operator` +- Other platform controllers and tooling + +These images are built via Konflux and are currently sourced from Quay at runtime. They need to +be mirrored to ECR and pulled from there instead. + +### HyperShift Configuration: ECR Without `--zero-egress` + +HyperShift's `--zero-egress` flag was designed for environments with no public internet access. +Setting it causes HyperShift to source control plane images from ECR mirrors, but it also applies +network path restrictions and other behavioural changes that are undesirable for standard Hosted +Clusters (e.g., it may restrict outbound network paths that are expected to be available in a +normal hosted cluster). + +Since the goal is to use ECR for image sourcing universally — not just in zero-egress environments +— an alternative mechanism is needed to force HyperShift to always pull from ECR. The specific +approach is still being determined, but candidate mechanisms include: + +- Configuring HyperShift's image override mechanism (e.g., via `ImageContentSources` or + `ImageDigestMirrorSet` on the management cluster) to redirect Quay image references to their + ECR equivalents. +- Providing a custom release image or image lookup override in the `HostedCluster` spec that + points directly to ECR-hosted digests. +- Patching HyperShift's image resolution logic to treat ECR as the primary source unconditionally. + +The chosen approach must ensure that HyperShift resolves all control plane images to ECR without +activating the broader `--zero-egress` mode. This will require investigation and likely +coordination with the HyperShift upstream team. + +### Mirroring: Extending the OCP Pipeline for Platform Images + +OCP already operates a mirroring pipeline that replicates release images from Quay to regional ECR +repositories. Rather than building a separate mirroring mechanism for platform images, the existing +pipeline will be extended to include platform image sets. + +The extension will: + +1. Accept platform image lists (produced by Konflux build outputs) as an additional input to the + mirroring configuration. +2. Mirror platform images to the same regional ECR structure used for OCP images, so that the same + repository conventions and IAM access patterns apply. +3. Replicate to all regions where OCP images are currently mirrored, maintaining geographic + consistency. + +The specifics of how Konflux build outputs are fed into the mirroring pipeline (e.g., via image +manifests, SBOMs, or a dedicated image list artifact) need to be defined as part of implementation. + +### Authentication + +Authentication to ECR for image pulls is required in two contexts: + +1. **MC/RC node pools and system components**: EKS nodes pulling platform operational images from + ECR. This is relatively straightforward — EKS node IAM roles can be granted `ecr:GetAuthorizationToken` + and `ecr:BatchGetImage` permissions for the relevant repositories, and the kubelet will handle + token refresh automatically via the ECR credential provider plugin (enabled by default on EKS). + +2. **HCP dataplane pods inside Hosted Clusters**: Pods running inside Hosted Clusters need to pull + images from ECR. The most likely scenario is to leverage HyperShift's existing pull secret + handling to include an ECR token. + + An investigation should determine the specifics of the auth mechanism and will be tightly + integrated with the HyperShift Operator changes. The goal is that a + single IAM identity or pull secret mechanism covers both OCP HCP dataplane images and platform + operational images from ECR. + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ Image Pull Flow │ +│ │ +│ MC / RC Nodes │ +│ ┌─────────────────┐ ECR Credential Provider ┌─────────────┐ │ +│ │ kubelet │ ─────────────────────────────▶│ ECR (regional) │ │ +│ │ (node IAM role)│ │ OCP images │ │ +│ └─────────────────┘ │ Platform │ │ +│ │ images │ │ +│ HCP Control Plane Pods └─────────────┘ │ +│ ┌─────────────────┐ [Existing HyperShift auth] ▲ │ +│ │ etcd / apiserver│ ─────────────────────────────────────┘ │ +│ │ ingress / etc │ │ +│ └─────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + +``` + +## Consequences + +### Positive + +- Eliminates runtime dependency on Quay.io for both HCP dataplane images and platform operational + images, reducing infra dependencies. +- Platform images gain the same regional redundancy and proximity as OCP images, reducing pull + latency and transfer costs. +- A single, unified auth mechanism for ECR simplifies credential management and reduces the number + of distinct IAM patterns to maintain. +- Reusing the existing OCP mirroring pipeline for platform images avoids duplicating mirroring + infrastructure and keeps the operational model consistent. + +### Negative + +- The ECR mirroring pipeline becomes a critical path dependency: if mirroring fails, new image + versions will not be available in ECR. The existing OCP pipeline presumably has reliability + requirements; extending it to cover platform images increases its blast radius. +- ECR repository provisioning (IAM policies, lifecycle rules, replication configuration) must be + managed per region and per image set, adding Terraform/IaC surface area. +- Forcing HyperShift to pull from ECR without `--zero-egress` requires either upstream changes or + a local override mechanism that may need to be maintained across HyperShift upgrades. + +## Cross-Cutting Concerns + +### Reliability + +- **Scalability**: ECR scales automatically. Regional replication ensures images are available + close to the clusters that need them. No capacity planning is required for the registry itself. +- **Observability**: ECR pull failures surface as pod image pull errors (`ImagePullBackOff`). + Alerting should be added for sustained pull failure rates on both MCs/RCs and HCP control plane + namespaces. Mirroring pipeline failures should produce actionable alerts before they affect + running clusters (images already pulled are cached on nodes). +- **Resiliency**: Nodes cache pulled images locally. A transient ECR outage will not affect already + running pods, only new pod starts or node replacements. Mirroring pipeline failures need to be + caught before node rotation forces re-pulls of images that are no longer available in ECR. + +### Security + +- ECR repositories should be private with resource-based policies restricting access to known AWS + account IDs and IAM roles. Cross-account pulls (e.g., Hosted Cluster node roles in customer + accounts) may require explicit repository policy statements. +- Leverage existing vulnerability visibility through existing platform mechanisms. +- The auth mechanism for HCP dataplane pods must not grant overly broad ECR permissions. Least + privilege applies: pull access to specific repositories only, not `ecr:*` on all repositories. +- Short-lived ECR tokens (12-hour expiry) reduce credential exposure compared to long-lived Quay + robot account tokens. + +### Cost + +- ECR charges for storage and data transfer. Platform images are small relative to OCP release + images; the incremental storage cost of adding platform images to the mirroring pipeline is + expected to be low. +- Pulling images from ECR within the same region incurs no data transfer charges. Cross-region + pulls are charged; regional replication ensures pulls are always intra-region. +- Lifecycle policies should be configured on ECR repositories to expire old image tags and control + storage growth. + +### Operability + +- The mirroring pipeline extension must be automated. Manual mirroring of platform images is not + acceptable at the cadence of Konflux builds. +- ECR credential rotation for MCs/RCs is handled automatically by the EKS ECR credential provider. + The HCP dataplane auth mechanism must similarly not require manual intervention for token refresh. +- Repository provisioning (new platform image repositories) should be Terraform-managed and follow + the same conventions as existing OCP ECR repositories. +- Runbooks are needed for: mirroring pipeline failure, ECR auth failures on HCP pods, and adding + new platform images to the mirroring configuration. + +## Open Questions / Areas Requiring Investigation + +1. **HCP dataplane auth**: What is the correct mechanism to provide ECR credentials to pods inside + a Hosted Cluster's control plane namespace on the management cluster? Can HyperShift's pull + secret handling be extended, or is a credential injection controller required? + +2. **HyperShift ECR override without `--zero-egress`**: What is the minimal change required to + make HyperShift resolve all control plane images from ECR by default? Is this achievable via + `ImageDigestMirrorSet` on the management cluster, or does it require changes to HyperShift's + image resolution code? + +3. **OCP mirroring pipeline extension**: What is the interface for adding platform image sets to + the existing OCP mirroring pipeline? Who owns that pipeline, and what is the process for + contributing extensions? + +4. **Repository naming conventions**: Should platform images share ECR repositories with OCP + images (under a common namespace) or live in separate repositories? The answer affects IAM + policy scoping and lifecycle rule management. + +5. **Cross-account ECR access for Hosted Cluster nodes**: If Hosted Cluster nodes run in customer + AWS accounts and need to pull from platform-account ECR repositories, what cross-account IAM + setup is required? Is this handled by HyperShift, or does the platform need to manage it? From 7d28d7dd2230b688d28c8b7a195bf8cdb8f0c2a8 Mon Sep 17 00:00:00 2001 From: rrp-bot Date: Thu, 25 Jun 2026 10:27:53 +0000 Subject: [PATCH 2/2] Address CodeRabbit review comments on ECR design doc - Reconcile repository-placement: clarify that platform images use the same regional ECR repo structure as OCP images (already decided); narrow open question 4 to naming/namespace conventions within that shared structure - Fix IAM policy scope: ecr:GetAuthorizationToken requires Resource '*' (account-level action, cannot be scoped to repository ARNs); ecr:BatchGetImage/GetDownloadUrlForLayer should use repo-scoped ARNs - Define dynamic auth for HCP pods: replace vague 'pull secret handling' with explicit requirement for Pod Identity / IRSA-based dynamic token rotation; explain why static imagePullSecrets are unsuitable (12h expiry) - Convert ASCII art diagram to Mermaid as required by repo conventions --- docs/design/ecr-image-migration.md | 75 ++++++++++++++++++------------ 1 file changed, 44 insertions(+), 31 deletions(-) diff --git a/docs/design/ecr-image-migration.md b/docs/design/ecr-image-migration.md index 8bd00e93b..26c791608 100644 --- a/docs/design/ecr-image-migration.md +++ b/docs/design/ecr-image-migration.md @@ -129,36 +129,46 @@ manifests, SBOMs, or a dedicated image list artifact) need to be defined as part Authentication to ECR for image pulls is required in two contexts: 1. **MC/RC node pools and system components**: EKS nodes pulling platform operational images from - ECR. This is relatively straightforward — EKS node IAM roles can be granted `ecr:GetAuthorizationToken` - and `ecr:BatchGetImage` permissions for the relevant repositories, and the kubelet will handle - token refresh automatically via the ECR credential provider plugin (enabled by default on EKS). + ECR. This is relatively straightforward — EKS node IAM roles can be granted: + - `ecr:GetAuthorizationToken` with `Resource: "*"` (this action is account-level and cannot be + scoped to specific repositories; attempting to restrict it to a repository ARN results in an + `AccessDeniedException`). + - `ecr:BatchGetImage` and `ecr:GetDownloadUrlForLayer` with `Resource` limited to the specific + repository ARNs for platform and OCP images (least-privilege, repository-scoped). -2. **HCP dataplane pods inside Hosted Clusters**: Pods running inside Hosted Clusters need to pull - images from ECR. The most likely scenario is to leverage HyperShift's existing pull secret - handling to include an ECR token. - - An investigation should determine the specifics of the auth mechanism and will be tightly - integrated with the HyperShift Operator changes. The goal is that a - single IAM identity or pull secret mechanism covers both OCP HCP dataplane images and platform - operational images from ECR. - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ Image Pull Flow │ -│ │ -│ MC / RC Nodes │ -│ ┌─────────────────┐ ECR Credential Provider ┌─────────────┐ │ -│ │ kubelet │ ─────────────────────────────▶│ ECR (regional) │ │ -│ │ (node IAM role)│ │ OCP images │ │ -│ └─────────────────┘ │ Platform │ │ -│ │ images │ │ -│ HCP Control Plane Pods └─────────────┘ │ -│ ┌─────────────────┐ [Existing HyperShift auth] ▲ │ -│ │ etcd / apiserver│ ─────────────────────────────────────┘ │ -│ │ ingress / etc │ │ -│ └─────────────────┘ │ -└─────────────────────────────────────────────────────────────────────┘ + The kubelet handles token refresh automatically via the ECR credential provider plugin (enabled + by default on EKS). +2. **HCP dataplane pods inside Hosted Clusters**: Pods running inside Hosted Clusters need to pull + images from ECR. Static `imagePullSecrets` are not suitable here because ECR authorization + tokens expire after 12 hours, causing image pull failures on any new pod start after token + expiry. + + The preferred approach is a dynamic credential mechanism such as AWS Pod Identity (EKS Pod + Identity Associations) or IRSA (IAM Roles for Service Accounts), which allows the HyperShift + Operator to obtain short-lived ECR tokens on demand without operator intervention. An + investigation should determine which mechanism integrates most cleanly with HyperShift's + existing image pull handling (e.g., whether HyperShift can refresh and inject a rotating ECR + token into the control plane namespace, or whether a sidecar credential rotation controller is + required). The goal is that a single IAM identity or dynamic pull mechanism covers both OCP HCP + dataplane images and platform operational images from ECR, with no manual token rotation. + +```mermaid +flowchart LR + subgraph MC_RC["MC / RC Nodes"] + kubelet["kubelet\n(node IAM role)"] + end + + subgraph HCP["HCP Control Plane Pods"] + hcp_pods["etcd / apiserver\ningress / etc"] + end + + subgraph ECR["ECR (regional)"] + ocp_imgs["OCP images\nPlatform images"] + end + + kubelet -->|"ECR credential provider\n(auto token refresh)"| ocp_imgs + hcp_pods -->|"HyperShift dynamic auth\n(Pod Identity / IRSA)"| ocp_imgs ``` ## Consequences @@ -245,9 +255,12 @@ Authentication to ECR for image pulls is required in two contexts: the existing OCP mirroring pipeline? Who owns that pipeline, and what is the process for contributing extensions? -4. **Repository naming conventions**: Should platform images share ECR repositories with OCP - images (under a common namespace) or live in separate repositories? The answer affects IAM - policy scoping and lifecycle rule management. +4. **Repository naming conventions**: Platform images will be mirrored into the same regional ECR + repository structure as OCP images (this is a constraint established in the design). The open + question is how to namespace them within that structure — for example, should platform images + share the same repository namespace as OCP release images, or live under a distinct + platform-specific prefix within the same ECR registry? The answer affects IAM policy scoping + (whether a single repository ARN pattern covers both sets) and lifecycle rule management. 5. **Cross-account ECR access for Hosted Cluster nodes**: If Hosted Cluster nodes run in customer AWS accounts and need to pull from platform-account ECR repositories, what cross-account IAM