Skip to content

gate-web: pin to Karpenter NodePool batch-spot + do-not-disrupt (DO NOT MERGE) - #20

Closed
pvishwa3 wants to merge 1 commit into
mainfrom
pr/karpenter-missing-nodepool
Closed

pvishwa3 wants to merge 1 commit into
mainfrom
pr/karpenter-missing-nodepool

Conversation

@pvishwa3

@pvishwa3 pvishwa3 commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Karpenter demo for the deployment-risk gate.

  • nodeSelector: karpenter.sh/nodepool: batch-spot. rtz-eks has only the NodePools spot and rtzkd-capped, so Karpenter never launches a node and the pods stay Pending. Expected: HIGH wrs-addon-karpenter-pod-pins-missing-nodepool, and the gate blocks.
  • karpenter.sh/do-not-disrupt: "true" on the pod template. Expected: LOW wrs-addon-karpenter-do-not-disrupt-blocks-node-rotation.

The Karpenter rules ship in action image 1.0.16 (runtimez-com/eac#605). Until that release, @v1 runs 1.0.15 and shows no Karpenter findings. Re-run this check after the release.

🤖 Generated with Claude Code

https://claude.ai/code/session_014NPEGf1kUYALnqg7HFRtBH

…OT MERGE)

rtz-eks has NodePools spot and rtzkd-capped only, so these pods would stay Pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014NPEGf1kUYALnqg7HFRtBH
@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Deployment risk: HIGH (score 61) — 5 findings

Severity Rule Resource Title
LOW mrs-diff-nodeselector-narrowed Deployment/rtz-gate-demo/gate-web nodeSelector narrowed
LOW mrs-diff-nodeselector-tightened Deployment/rtz-gate-demo/gate-web nodeSelector added or tightened

Capacity fit (1)

  • Not known Deployment rtz-gate-demo/gate-web at 2 replica(s) (spec.replicas) on the cluster's last sync 0ba1786d-bb57-45db-b84c-5578f975ba56: where its pods go is not known: no new node the pools may add takes any of the 2 pods left without a home (2 no node matches its placement); rtzkd-capped: it has no node that takes pods now, so what it would add is not known.

If a zone is lost (each on its own, this PR's workloads placed on what is left):

  • us-east-1b: not known (rtz-gate-demo/gate-web) · the zone's pods back in ~8 min
  • us-east-1f: not known (rtz-gate-demo/gate-web) · the zone's pods back in ~8 min · 1 not back on their own

Add-on findings · Karpenter (2)

Severity Rule Resource Title
HIGH wrs-addon-karpenter-pod-pins-missing-nodepool Deployment/rtz-gate-demo/gate-web Workload is pinned to a Karpenter NodePool that does not exist
LOW wrs-addon-karpenter-do-not-disrupt-blocks-node-rotation Deployment/rtz-gate-demo/gate-web karpenter.sh/do-not-disrupt on a long-running workload
What changed and how to fix it
  • mrs-diff-nodeselector-narrowed · Deployment/rtz-gate-demo/gate-web — Deployment rtz-gate-demo/gate-web nodeSelector was narrowed ({} → {karpenter.sh/nodepool=batch-spot,}). A tighter nodeSelector reduces the pool of nodes replicas can land on, increasing single-node concentration risk. Confirm enough distinct nodes still match.
    Fix: A Deployment/StatefulSet's nodeSelector was narrowed (a key was added or changed). A tighter nodeSelector reduces the pool of nodes replicas can land on, increasing single-node concentration risk. Confirm enough distinct nodes still match.
  • mrs-diff-nodeselector-tightened · Deployment/rtz-gate-demo/gate-web — Deployment rtz-gate-demo/gate-web nodeSelector was added/tightened ({} → {karpenter.sh/nodepool=batch-spot,}). Fewer nodes may satisfy the new constraint; verify at least one live node carries the required label(s).
    Fix: A workload's nodeSelector had a key added or changed. Fewer nodes may satisfy the new constraint. Confirm at least one live node carries the required label(s).
  • wrs-addon-karpenter-pod-pins-missing-nodepool · Deployment/rtz-gate-demo/gate-web — Deployment rtz-gate-demo/gate-web: Its placement requires karpenter.sh/nodepool in [batch-spot], and the cluster (with this change applied) has no such NodePool — it has rtzkd-capped, spot. Karpenter launches nodes only for an existing NodePool, so these pods stay Pending.
    Fix: Point karpenter.sh/nodepool at an existing NodePool, or add the NodePool in the same change. Karpenter launches nodes only for an existing NodePool, so these pods stay Pending.
  • wrs-addon-karpenter-do-not-disrupt-blocks-node-rotation · Deployment/rtz-gate-demo/gate-web — Deployment rtz-gate-demo/gate-web: The change sets karpenter.sh/do-not-disrupt: "true" on the pod template of a long-running workload. While these pods run, Karpenter never consolidates the nodes they are on, and drift replacement and expiry wait for them — indefinitely unless the NodePool sets terminationGracePeriod, which bounds the wait (with it set, drift and expiry go ahead regardless of the annotation). A NodePool or AMI change may therefore never reach those nodes. Use it for jobs that must finish, not for services — a PodDisruptionBudget limits disruption without blocking it.
    Fix: Pods with do-not-disrupt keep Karpenter from consolidating or drift-replacing their nodes. Prefer a PodDisruptionBudget for services, and set terminationGracePeriod on NodePools that host such pods.

Cluster-aware checks: on — 55 runtimez lookup(s) answered.

@pvishwa3

pvishwa3 commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Demo finished (Karpenter gate verified live) — closing, never meant to merge.

@pvishwa3 pvishwa3 closed this Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant