Right-size runners onto per-size nodepool tiers and move the c7i-runner pool to Graviton#871
Merged
Merged
Conversation
…ollision-safe **Impact:** ARC runner scheduling on clusters using the c7i-runner fleet; nodepool generator + smoke tests (CI tooling) **Risk:** medium ## What Repoints the `c7i-runner` fleet from amd64 `c7i.*` instances to arm64 Graviton (`m7g.*` / `m8g.*`), keeping the fleet name unchanged, and teaches the nodepool generator to family-qualify NodePool names when two instance families in the same fleet share a size suffix. ## Why The fleet name `c7i-runner` is load-bearing — the `node-fleet=c7i-runner` taint/nodeSelector is referenced by `runner.yaml.tpl` and the hooks-warmer, so it can't be renamed to match the new instance family. But mixing `m7g` and `m8g` at the same size (e.g. `m7g.12xlarge` + `m8g.12xlarge`) made both map to `c7i-runner-12xlarge`, so the second def would silently overwrite the first (same `metadata.name` and output filename). `_fleet_size_collisions` detects cross-family same-size clashes and inserts the family (`c7i-runner-m7g-12xlarge`) only for the colliding entries, leaving single-family fleets untouched. Runner and hooks images are multi-arch, so the arm64 switch is safe. ## Notes - Single-family override fleets keep their existing `<fleet>-<size>` names — no rename churn for `c7i-large`, `m7g-metal`, etc. (guarded by a test). - Smoke test now delegates naming to `_build_fleet_nodepool_def` so its expected-name set always tracks the generator, instead of duplicating the naming logic. - `instance_specs.py` gains `m7g.12xlarge` (specs + ENI max-pods) since it's now referenced by the fleet. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
**Impact:** CI runner scheduling and node capacity across all OSDC clusters (ARC self-hosted runners only; no user-facing services) **Risk:** medium ## What Splits the single per-family Karpenter fleets (c7a, c7i, m7i, r7i, r7a, g4dn, g5, g6, m7g, m8g) into per-size "sub-nodepool" tiers (e.g. `c7a-2xl`, `c7a-8xl`, `c7a-12xl`, `c7a-16xl`), each with its ideal instance at weight 100 and fallback *up* to larger sizes at descending weight. Every affected ARC runner def now pins an explicit `node_fleet` at the tier that packs it best, and its `vcpu`/`memory` are bumped to consume that ideal node instead of stranding capacity on an oversized one. ## Why Runners previously shared one big fleet per family weighted toward the 48xlarge, so a small runner (e.g. 8c/32Gi) could land on a 48xlarge node — terrible packing and wasted spend. The `node-size-sweep` optimization (report `20260705T151451Z-4b52af9`) computed the ideal node size per runner. Pinning each runner to a size-matched tier improves utilization while the descending-weight fallback preserves capacity headroom under scarcity. Notable design points for reviewers: - **CPU fleets** fall back to *larger* sizes (never smaller). **GPU fleets** (g4dn/g5/g6) fall back only within the same GPU count to avoid stranding GPUs. - `m8g` and `r7a` become release-only fleets — their packing instances moved to the new `*-16xl`/`*-8xl`/etc. tiers; the `release:` blocks that back `rel-` runners are untouched. - `instance_specs.py` gains `c7a.2xlarge` and `m7g.16xlarge` (specs + ENI max-pods) for the new tiers. # Notes - Deploy the `nodepools` and `arc-runners` modules together — the fleet renames remove the old NodePools, and a runner referencing a fleet that no longer exists will sit Pending on a toleration mismatch. - `us-west-1` exclusions and the c7a "not available in us-west-1" notes carry over unchanged to the split fleets. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
Capacity reportcommit ✅ simulate-cluster✅ analyze-utilization |
tofu plan — meta-prod-aws-ue2✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-uw1✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue2✅ Plan succeeded · commit Plan output |
- Collapse per-size nodepool tiers (c7a/c7i/g4dn/g5/g6/m7g/m7i/m8g/r7a/r7i) into one per-family fleet each, largest instance weighted highest for bin-packing; add baremetal rungs (c7i.metal-24xl, g4dn.metal, m7g.metal) - Drop node_fleet pinning from arc-runner defs and right-size vcpu/memory to exact job budgets so runners pack onto the shared per-family fleets - Revert c7i-runner to Intel amd64 (name-only clone of c7i; separation kept by the node-fleet=c7i-runner taint) - Add normalize_modules() so -opt shim variants satisfy their base-module gates (arc-runners-opt→arc-runners, nodepools-opt→nodepools) in run/phases; -opt clusters shard alongside the base module in generate_runners - Switch meta-prod-aws-ue1 to arc-runners-opt / nodepools-opt modules GPU fleets (g4dn/g5/g6) move to a single fleet each with allocation driven by nvidia.com/gpu resource requests rather than per-GPU-count sub-pools. -opt is a shim variant for piloting the new placement on ue1 without disturbing other clusters; GPU capability variants (-h100/-b200) are intentionally NOT aliased to their base module — they keep independent gating and sharding. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- New nodepools-opt module: per-size Karpenter fleet defs with descending-weight fallback rungs, plus dedicated name-only pools isolated by node-fleet taints - New arc-runners-opt module: runner defs right-sized (vcpu/memory after overhead) and pinned to matching node_fleet tiers for tight bin-packing - Both modules ship a thin deploy.sh that delegates to the upstream arc-runners/nodepools deploy via *_DEFS_DIR / *_OUTPUT_DIR / *_MODULE_NAME env vars - GPU tiers are GPU-count-locked to avoid stranding GPUs; p4d packing pool keeps single-numa-node while the 8-GPU p4d-large pool stays best-effort to avoid TopologyAffinityError hangs - Release runners (rel-*) get dedicated runner_group/runner_class backed by release-only fleets Notes: Establishes optimized copies of the runner and nodepool definitions so placement changes can be piloted independently while reusing the upstream generators. Fleet/runner names are decoupled from instance families where needed (e.g. c7i-runner runs Graviton arm64; taint name is load-bearing). c7a fleets exclude us-west-1 where those sizes are unavailable. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Impact: ARC runners and Karpenter node provisioning on
meta-prod-aws-ue1only (pilot cluster); all other OSDC clusters unchangedRisk: medium (single-cluster canary; other clusters stay on the current baseline behavior)
Simulation/Optimization scripts: #870
What
Rolls out the fleet-packing optimization to a single cluster (
meta-prod-aws-ue1) instead of fleet-wide, using new-optshim modules so the rest of the fleet is untouched:modules/nodepools/defsandmodules/arc-runners/defsare reverted to their pre-optimization state, so every other cluster provisions exactly as before.nodepools-opt+arc-runners-optmodules hold the optimized definitions. They are thin shims that delegate to the basedeploy.shvia env vars (same pattern as the existingnodepools-h100/arc-runners-h100GPU shims), so base and-optshare one generator — only the def files differ.arc-runners-optadditionally pointsNODEPOOLS_DEFS_DIRatnodepools-opt/defsso region-exclusion resolves against the optimized fleets.meta-prod-aws-ue1is switched to the-optmodules inclusters.yaml(module list swap + a smallarc-runners-opt:config block carrying the0.8fresh-multiplier).meta-prod-aws-ue1andmeta-prod-aws-ue2remain in the same capacity-aware HUD shard group, and the integration/workload test harness recognizes the-optmodules.The optimization itself (embodied in the
-optdefs): splits the monolithic per-family Karpenter fleets (c7a,c7i,m7i,m8g,r7a,r7i,g4dn,g5,g6) into per-size tier nodepools where each tier's ideal instance sits at weight 100 and falls back up to larger sizes, then re-pins each runner def to its matching tier vianode_fleetand grows the runner's vcpu/memory to fill the node. Also switches thec7i-runnerpool from Intel c7i to Graviton (m7g/m8g, arm64) while keeping the load-bearingnode-fleet=c7i-runnertaint name.Why
The old single-fleet-per-family design let Karpenter satisfy a small runner by spinning up (or bin-packing onto) an oversized node — e.g. an 8-vcpu job landing on a
.48xlarge— which stranded capacity and hurt packing efficiency. Anchoring each runner to a size-matched tier with larger-only fallback gives Karpenter a tight, predictable node choice and lets the runner request nearly the whole node. The Graviton switch on the runner pool trades Intel for cheaper arm64 capacity; the runner image and ARC hooks are already multi-arch.Rolling out to one cluster first (rather than fleet-wide) limits blast radius:
meta-prod-aws-ue1acts as a canary while every other cluster stays on the proven baseline. If the pilot regresses, only ue1 is affected and the change is reverted by pointing ue1 back at the base modules.Note
Cluster-wide priced spend: baseline ~$1,992,012 → rec ~$1,877,065 (approx on-demand list price, excludes reserved GPU). Priced-$ Δ negative = cheaper.
Changes per fleet
Do we have room to reduce pod requests and get better packing?
I asked claude to have a quick look, and seems that before getting substantial results, we should reduce the size subtantialy, what makes this as a probably no-go:
Biggest current cost usage