feat(catalog): the karpenter module — nodes for the workloads on aws, its role and interruption queue through Crossplane

.github/scripts/check-catalog-requests.sh

@@ -118,7 +118,8 @@ pull() { # <dest> <helm pull args...>: three attempts, helm's error kept
     fi
     rm -rf "${dest}"
     echo "helm pull $* (attempt ${n}/3): ${err}" >&2
-    [ "${n}" = 3 ] || sleep $((n * 5))
+    # 20 s, then 40 s: a throttling registry (ECR Public, 429) needs longer than a blip.
+    [ "${n}" = 3 ] || sleep $((n * 20))
   done
   echo "::error::cannot pull $*: ${err}"
   exit 1

.github/scripts/upstream-monitoring.sh

@@ -27,6 +27,8 @@ argocd|dashboard|dashboard-argocd|https://raw.githubusercontent.com/argoproj/arg
 argocd-image-updater|dashboard|dashboard-argocd-image-updater|https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/v1.3.0/config/example-grafana-dashboard.json
 crossplane|dashboard|dashboard-crossplane|https://grafana.com/api/dashboards/24549/revisions/3/download
 external-dns|dashboard|dashboard-external-dns|https://grafana.com/api/dashboards/23969/revisions/1/download
+karpenter|dashboard|dashboard-karpenter-capacity|https://raw.githubusercontent.com/aws/karpenter-provider-aws/v1.15.0/website/content/en/preview/getting-started/getting-started-with-karpenter/karpenter-capacity-dashboard.json
+karpenter|dashboard|dashboard-karpenter-performance|https://raw.githubusercontent.com/aws/karpenter-provider-aws/v1.15.0/website/content/en/preview/getting-started/getting-started-with-karpenter/karpenter-performance-dashboard.json
 keda|dashboard|dashboard-keda|https://raw.githubusercontent.com/kedacore/keda/v2.21.0/config/grafana/keda-dashboard.json
 velero|dashboard|dashboard-velero|https://grafana.com/api/dashboards/16829/revisions/5/download
 EOF

.github/workflows/e2e.yaml

@@ -232,7 +232,10 @@ jobs:
     name: ${{ matrix.module }} (${{ matrix.cloud }})
     runs-on: ubuntu-24.04
     needs: discover
-    timeout-minutes: 20
+    # velero's job ran 17.8 to 20.3 min once Crossplane installs six AWS
+    # providers (#53), and karpenter's is of the same length: 20 left no
+    # room for the destroy.
+    timeout-minutes: 25
     permissions:
       contents: read
       checks: write

.github/workflows/pr-static.yaml

@@ -383,8 +383,13 @@ jobs:
         with:
           path: /tmp/charts
           # The pinned chart versions live in the modules' ResourceSets and
-          # in the clouds' patches.
+          # in the clouds' patches. A chart is cached under its URL, name and
+          # version, which never change content, so the last cache restores
+          # whatever charts it has when the key moves: a registry that
+          # throttles the runners (ECR Public answers 429 to anonymous pulls,
+          # the karpenter chart's) is then asked only for a new version.
           key: catalog-charts-${{ hashFiles('oci/catalog/*/resourceset.yaml', 'oci/clusters/*/*/patch.yaml') }}
+          restore-keys: catalog-charts-
       - name: Every container the catalog runs states its requests
         if: steps.detect.outputs.found == 'true'
         run: |

README.md

@@ -156,13 +156,14 @@ The stack's design: [Observability](docs/architecture/observability.md).
 
 | Module | What it does | Version | Default | Clouds | Notes |
 | --- | --- | --- | :---: | --- | --- |
-| [`crossplane`](docs/catalog/crossplane.md) | Lets each module declare its own cloud IAM | 2.4.2 | off | All | Needed by `velero`, and on AWS and GCP by `external_dns`, `keda` (`services`), `external_secrets` and `argocd_image_updater` (`repositories`) for their own access; on GCP no Google service account |
+| [`crossplane`](docs/catalog/crossplane.md) | Lets each module declare its own cloud IAM | 2.4.2 | off | All | Needed by `velero` and `karpenter`, and on AWS and GCP by `external_dns`, `keda` (`services`), `external_secrets` and `argocd_image_updater` (`repositories`) for their own access; on GCP no Google service account |
 
 ### Autoscaling
 
 | Module | What it does | Version | Default | Clouds | Notes |
 | --- | --- | --- | :---: | --- | --- |
 | [`keda`](docs/catalog/keda.md) | Event-driven autoscaling, down to zero | 2.21.0 | off | All | |
+| [`karpenter`](docs/catalog/karpenter.md) | Nodes for your workloads, launched for what is Pending, Spot and Graviton first, removed when empty; a private pool, and a public one for traffic-heavy workloads (no NAT Gateway) | 1.15.0 | off | AWS | Needs `crossplane`; GKE Autopilot provisions its own nodes |
 
 ### Backup
 
@@ -191,7 +192,8 @@ the minor.
 - The foundations modules exist for all four clouds. The single-apply root
   exists for AWS and GCP, so far.
 - On GCP every catalog module that runs on AWS runs too, metrics-server
-  aside (GKE serves its own), each cloud-backed one with its own access
+  and karpenter aside (GKE serves its own metrics API, and Autopilot
+  provisions its own nodes), each cloud-backed one with its own access
   through Crossplane and no Google service account. Its proof is a GKE
   Autopilot sandbox, by hand: floci-gcp has no Cloud DNS and no Workload
   Identity, so CI renders and plans GCP but runs no GKE e2e. One socle

docs/architecture/cilium-before-flux.md

@@ -60,10 +60,15 @@ follows `cilium.gateway_api`, the class being the catalog's.
 <details>
 <summary>Under the hood</summary>
 
-**AWS.** `eni.enabled: true`: ENI IPAM, native routing, masquerade off; pods
-carry VPC addresses and leave through the NAT Gateway. `kubeProxyReplacement:
-true`, with `k8sServiceHost`/`Port` from the foundations' `cluster_endpoint`.
-The operator creates ENIs from the bootstrap nodes' role.
+**AWS.** `eni.enabled: true`: ENI IPAM, native routing; pods carry VPC
+addresses. What leaves the VPC is masqueraded to the node's primary address
+(`ipv4NativeRoutingCIDR` the foundations' `vpc_cidr`, through `ens+`): a
+pod's secondary address never crosses an Internet Gateway, which a node of
+Karpenter's public pool leaves through
+([AWS-23](../decisions/aws.md#aws-23-public-nodes-their-own-security-group-and-cilium-masquerading-out-of-the-vpc)).
+`kubeProxyReplacement: true`, with `k8sServiceHost`/`Port` from the
+foundations' `cluster_endpoint`. The operator creates ENIs from the bootstrap
+nodes' role.
 
 **Azure.** `aksbyocni.enabled: true`: cluster-pool IPAM over VXLAN, with
 masquerade. The pool is the foundations' `pod_cidr` (`10.244.0.0/16`), since

docs/architecture/module-iam.md

@@ -6,7 +6,8 @@ sidebar:
 ---
 
 Some modules act on the cloud: external-dns writes DNS records, KEDA reads
-queues, External Secrets reads a secret manager, Velero writes a bucket. Each
+queues, External Secrets reads a secret manager, Velero writes a bucket,
+Karpenter launches nodes. Each
 needs a role on each cloud.
 
 ## The rule
@@ -30,7 +31,14 @@ foundations grant it once, the same for every client, bounded by shape
   default) and always denies `iam`, `sts`, `organizations`, `account`, `sso`
   and `identitystore`;
 - Crossplane cannot attach managed policies, edit the boundary, or touch a
-  role outside its path.
+  role outside its path;
+- with `ec2` in the list, and only then, a role may pass the bootstrap
+  nodes' role to EC2 and read instance profiles: Karpenter's nodes run as
+  that role, through the instance profile the foundations give it, and no
+  other IAM action is allowed
+  ([AWS-21](../decisions/aws.md#aws-21-the-boundarys-one-iam-exception-the-node-role-passed-to-ec2));
+- beside roles, Crossplane manages a module's buckets, queues and
+  EventBridge rules under `<cluster>-` only.
 
 A module on a new service needs one word in `allowed_services`: a reviewed
 change. The root wires the boundary into `kube.crossplane.permissions_boundary`.

docs/catalog/crossplane.md

@@ -8,8 +8,8 @@ requires: []
 Crossplane is the tooling a module that needs a cloud service uses to declare
 its own role, beside its workload. It grants nothing by itself. **Off by
 default**; turn it on when a module you want needs cloud access (external-dns,
-external-secrets, keda with `services`, velero, argocd-image-updater with
-`repositories`). Providers exist on aws and gcp.
+external-secrets, keda with `services`, velero, karpenter,
+argocd-image-updater with `repositories`). Providers exist on aws and gcp.
 
 ## Getting started
 
@@ -29,7 +29,7 @@ kube = {
 ```
 
 Then `kubectl get providers.pkg.crossplane.io` shows the providers `Healthy`:
-four on aws, three on gcp.
+six on aws, three on gcp.
 
 ## Settings
 
@@ -88,7 +88,7 @@ kube = {
   or an `extraEnvVars*` entry named like a password, token or key. Use
   `values_secret`.
 - **Upgrades**: the chart moves with `socle_version`, and its CRDs with it
-  (Crossplane applies them at start). The four providers move together, on
+  (Crossplane applies them at start). The providers move together, on
   one tag on each cloud.
 - **Coming from an alpha before 0.1.0?** Once 0.1.0 is applied, strip the
   owners the old providers left on their ServiceAccount, once, or the garbage
@@ -100,7 +100,8 @@ kube = {
 
 **Installed**: chart `crossplane` 2.4.2 from `https://charts.crossplane.io/stable`,
 in `crossplane-system`. On aws, the providers `provider-family-aws`,
-`provider-aws-iam`, `provider-aws-eks` and `provider-aws-s3` (v2.8.1), then
+`provider-aws-iam`, `provider-aws-eks`, `provider-aws-s3`, `provider-aws-sqs`
+and `provider-aws-cloudwatchevents` (v2.8.1), then
 `ClusterProviderConfig/default` on EKS Pod Identity, once they are `Healthy`.
 On gcp, `provider-family-gcp`, `provider-gcp-cloudplatform` and
 `provider-gcp-storage` (v3.0.0), then `ClusterProviderConfig/default` on
@@ -122,7 +123,11 @@ write a billed line of the EKS audit log
 **Cloud access**: on aws, the foundations' role for
 `crossplane-system/provider-aws` creates roles only under `/socle/<cluster>/`
 and only with the boundary, which allows `allowed_services` and always denies
-`iam`, `sts` and the account services. On gcp there is no Google service
+`iam`, `sts` and the account services — but for one exception with `ec2`
+allowed: passing the bootstrap nodes' role to EC2, what Karpenter launches
+its nodes with ([AWS-21](../decisions/aws.md#aws-21-the-boundarys-one-iam-exception-the-node-role-passed-to-ec2)).
+Beside roles, it manages buckets, SQS queues and EventBridge rules under
+`<cluster>-` only, and never deletes a bucket. On gcp there is no Google service
 account: the providers' own ServiceAccount is the principal,
 `principal://iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<project>.svc.id.goog/subject/ns/crossplane-system/sa/provider-gcp`,
 and `InjectedIdentity` makes the provider take its token from the GKE metadata

docs/catalog/index.mdx

@@ -1,6 +1,6 @@
 ---
 title: Catalog
-description: The sixteen catalog modules, what each gives you, whether it is on by default, the clouds it runs on, and how kube turns them on.
+description: The seventeen catalog modules, what each gives you, whether it is on by default, the clouds it runs on, and how kube turns them on.
 sidebar:
   label: Overview
   order: 0