[Demmonius]

Issue #53. Four commits, rebased on main (78ec831):

1. the foundations (feat(aws)), as Catalog: Karpenter as a socle module (aws) #53 asked when the boundary answer touches a foundations concern;
2. the module (feat(catalog));
3. a CI fix the module needed (ci);
4. the public node pool Hugo asked for on 2026-10-17 (feat(karpenter)).

Proven on the aws sandbox (see Measured). Design: docs/catalog/karpenter.md, docs/decisions/karpenter.md (KARPENTER-01…05), docs/decisions/aws.md (AWS-21, AWS-22, AWS-23; main took AWS-20).

The blocking question: IAM under the boundary

Karpenter's controller needs iam:PassRole on its nodes' role; AWS's setup also lets it create and tag instance profiles. The Crossplane boundary denies iam:*.

• Karpenter's nodes run as the bootstrap nodes' role. That role already holds what a node needs: kubelet, ECR pull, Cilium's ENI calls. EKS also made its EC2_LINUX access entry. The foundations give it an instance profile of the same name. The EC2NodeClasses name it with spec.instanceProfile, so Karpenter creates no profile and only reads it (GetInstanceProfile, ListInstanceProfiles). A profile, not a role: OpenTofu still creates no role for a module.
• The boundary's one IAM exception (AWS-21) exists only when ec2 is in crossplane.allowed_services. It allows iam:PassRole on that one role, with iam:PassedToService = ec2.amazonaws.com, plus the two profile reads; iam:* stays denied everywhere else. Nothing in the foundations names Karpenter.
• Rejected:
  • a second node role made by the foundations: an OpenTofu role for a catalog module;
  • the node role, profile and access entry made by Crossplane: CreateInstanceProfile, PassRole of any role it makes, and eks:CreateAccessEntry for Crossplane.
• The controller's role is a Crossplane Role under /socle/<cluster>/. Its policy is AWS's own controller policy for 1.15.0, statement for statement (getting-started CloudFormation), minus the instance-profile writes and the zonal-shift read. The client allows ec2, ssm, pricing, sqs and eks; the aws root warns when one is missing.

What

The karpenter catalog module, aws only, off by default.

• oci/catalog/karpenter, steps namespace, workload, nodes:

  • the chart 1.15.0 from oci://public.ecr.aws/karpenter, CRDs CreateReplace on install and upgrade;
  • two replicas, never on a node Karpenter manages, preferably one per bootstrap node and zone;
  • AWS_REGION set, because pods cannot reach IMDS on one hop;
  • eksControlPlane: true, because AL2023's nodeadm needs the service range.

• Two NodePools, always (KARPENTER-03), put in by the karpenter-nodes Kustomization's patches:

  • private, the default: private subnets, egress through the NAT Gateway, the cluster security group;
  • public, tainted socle.do-now.io/node-pool=public:NoSchedule: public subnets, a public IP per node, egress straight through the Internet Gateway, so no NAT Gateway bill for traffic-heavy workloads. A pod asks for it with that toleration and a nodeSelector on socle.do-now.io/node-pool.

  Both: Spot and Graviton first, families c/m/r gen 6+, WhenEmptyOrUnderutilized, Cilium's startup taint node.cilium.io/agent-not-ready, AL2023, IMDSv2 on one hop, an encrypted gp3 root volume. kube.karpenter.private_pool and public_pool type each pool's capacity types, architectures, limits, consolidation delay and AMI alias, validated at plan. No new subnet tag.

• oci/clusters/aws/karpenter/patch.yaml, step cloud:

  • the Role, the Pod Identity association, the interruption queue (AWS's queue: 300 s, SSE, HTTPS only, EventBridge only) and the five EventBridge rules with their targets;
  • per main's Pod Identity: a module's pod can be admitted before EKS propagates its association — no AWS credentials, no recovery #97 contract, the ServiceAccount (the step's, the chart's off) and karpenter-access. It waits for the role, the association and the queue, then for a sts get-caller-identity Job under that ServiceAccount to complete.

• Monitoring: Karpenter's capacity and performance dashboards (upstream, v1.15.0) and five socle rules:

  • an EC2NodeClass not Ready;
  • a NodePool not Ready;
  • a NodePool near its limits;
  • EC2 calls failing;
  • pods that no NodePool can place.

The public pool, and what it took (AWS-23)

Each point below was found on the sandbox, not assumed.

1. Egress. A pod's address is an ENI's secondary one, which the Internet Gateway never translates: a pod on a public node reached nothing. Cilium now masquerades what leaves the VPC on aws (enableIPv4Masquerade, ipv4NativeRoutingCIDR the VPC's CIDR, egressMasqueradeInterfaces: ens+). Cilium refuses masquerading in ENI mode without the interfaces: the agents crashed. The aws root wires the VPC's CIDR as cluster_network.pod_cidr, now required on aws. Inside the VPC nothing changes; the NAT Gateway and the flow logs now see nodes, not pods.
2. Exposure. EKS's in-tree load balancer controller opens a Service's NodePort to 0.0.0.0/0 in the security group of every node it targets. On the cluster security group, a public node's NodePort answered from the internet. The fix: the public nodes get their own security group (<cluster>-public-nodes: the VPC's CIDR in, nothing from the internet, not tagged for the cluster; the cluster security group admits it back. floci reports no cluster security group, so that one rule then lands on the group itself) and the label node.kubernetes.io/exclude-from-external-load-balancers.
3. The label's timing. Set by Karpenter, the label arrived seconds after registration. The controller had already targeted the node, opened the NodePort in its group, and never closed it. Karpenter keeps kubernetes.io labels off the kubelet's flags, so the kubelet sets it at registration through an AL2023 NodeConfig in the public EC2NodeClass's userData. With both, a public node was never a target and its NodePort stayed closed.

A Gateway's traffic reaches pods on public nodes through the private nodes.

Uninstall order (KARPENTER-04)

flux-operator 0.60.0 deletes a ResourceSet's objects in two stages (deleteAllStaged): Flux objects first, waiting up to 60 s for them, then the rest. So:

• the NodePools and the EC2NodeClasses are in a Flux Kustomization with deletionPolicy: WaitForTermination;
• the controller's HelmRelease is the one object of a child ResourceSet, karpenter-controller.

Nodes go before the controller. The docs still say to turn the module off and wait for nodeclaims to be empty before tofu destroy, because of the 60 s cap.

GCP

Not offered (KARPENTER-05). GKE Autopilot provisions, sizes and removes a node for every pod. There is no node pool to scale and no NAP to turn on, and Karpenter has no GCP provider. tofu plan refuses the module on gcp; the docs, the README and the gcp status page say so.

Outside the module, each generic

Where	Change
opentofu/aws	the instance profile; the boundary exception with ec2; Crossplane may manage queues and EventBridge rules (and targets) under <cluster>-, never read or send a message; the public nodes' security group (the VPC in, anywhere out) and the cluster security group admitting it; outputs node_instance_profile, vpc_cidr, public_nodes_security_group; tofu test asserts each
opentofu/bootstrap	Cilium masquerades out of the VPC on aws; cluster_network.pod_cidr required on aws
opentofu/clusters/aws	pod_cidr wired from the foundations; a check warning when a needed service is missing; the tfvars example
oci/clusters/aws/crossplane	provider-aws-sqs and provider-aws-cloudwatchevents v2.8.1; crossplane's e2e asserts them Healthy
.github (ci commit)	the charts' cache gets restore-keys, safe because a chart is cached under its URL, name and version: ECR Public 429s anonymous runner pulls; pull retries wait 20/40 s; module e2e jobs get 25 min (velero hit 20 with six providers)
site	the catalog brick's monogram

Checks run locally, on the rebased head (every pr-static.yaml step)

• tofu fmt, validate, tflint on opentofu/aws, opentofu/bootstrap and opentofu/clusters/aws.
• tofu test: 44/44 in opentofu/aws, 324/324 in opentofu/bootstrap.
• terraform-docs 0.20.0 current.
• yamllint --strict, actionlint, check-catalog-clouds.sh, check-catalog-pages.sh (17 modules, no cycle), check-version.sh.
• kubeconform strict: 365 rendered objects, child ResourceSets included; 68 templates.
• The requests guard: 63 containers.
• vmalert -dryRun; upstream-monitoring.sh check.
• Trivy: config HIGH/CRITICAL and secrets, clean.
• The strict site build: links valid.

Measured

On the aws sandbox, eu-west-3, 2026-10-17, cluster socle-sbx-53, applied from this branch's root. Two full cycles, the second on the final code: artifact 0.0.0-issue-53-karpenter.42809ab, whose oci/ is the head's.

1. Fresh apply (61 resources).
   • The module Ready 5 min after the apply: role, association, queue, five rules and targets through Crossplane.
   • The Pod Identity check Job complete.
   • Both EC2NodeClasses ValidationSucceeded (Karpenter's dry-run RunInstances, so the PassRole works under the boundary), public finding its security group by name.
2. karpenter-module-eks passed (4 min 26 s):
   • a private node Ready with Cilium in 49 s;
   • a public node with a public IP, labelled by its kubelet out of every load balancer, with a Job reaching the internet from it, in 57 s;
   • both consolidated 2 min 20 s after their pods went.
3. Exposure, on the final rules. With a public NLB Service and a public node running for 3 min:
   • the public group held only 10.0.0.0/16;
   • the public node was never a target, and its NodePort never answered from the internet;
   • the NLB answered 200 through the private nodes.
4. Module turned off with a public node running:
   • the instance went to terminated within a minute, then the controller went;
   • the queue and the role were deleted with the module.
5. tofu destroy: 61 destroyed. Nothing left: no cluster, VPC, instance, EIP, NLB, security group, instance profile, role, rule or log group.
6. The experiments behind AWS-23 (first cycle, by hand), as described above:
   • no egress from a public node without masquerading;
   • agents crashing without the masquerade interfaces;
   • the NodePort open from the internet on the cluster security group;
   • the label's window when Karpenter sets it.

CI, on the head 9ab26c5: all 57 checks green, the 5 protect-main required checks included, and the 17 floci e2e jobs (karpenter (aws) and root (aws) among them).

Fixed on the way:

• Helm merged the chart's required anti-affinity beside the socle's preferred one (null now).
• A parenthesised key in a Chainsaw assertion is an expression.
• Flux applies any *Class first and waits for it whatever wait (fluxcd/pkg ssa). floci's EC2NodeClass is never Ready, so the e2e sets the pools at enable.
• ECR Public 429.
• velero's 20-minute budget.
• floci's missing cluster security group.
• One unrelated flake in external-secrets.

Decisions to confirm

1. The boundary exception follows ec2, not a karpenter flag. Any module role allowed ec2 could pass the nodes' role to EC2, and nothing more.
2. Karpenter's nodes share the bootstrap nodes' identity, not a second node role.
3. Two Crossplane providers more on every aws cluster with Crossplane on, about 2×320Mi.
4. The pools are typed attributes rather than free values; other NodePools are the client's.
5. The public pool:
   • always present (no node until a pod asks for it), with the same default limits as the private one;
   • tainted, so a pod must opt in;
   • Cilium masquerades out of the VPC on every node, which changes what the NAT Gateway and the flow logs see for private nodes too.

Closes #53.

[github-actions]

📖 Docs preview: https://do-now-io.github.io/socle/pr/116/
Built from 9ab26c5. It follows every push and goes when the PR closes.