Adds a simulation + optimization script to find ideal node/runner sizes based on historical data#870
Draft
jeanschmidt wants to merge 12 commits into
Draft
Adds a simulation + optimization script to find ideal node/runner sizes based on historical data#870jeanschmidt wants to merge 12 commits into
jeanschmidt wants to merge 12 commits into
Conversation
Real 2D CPU+memory+GPU bin-packing simulator that replays HUD workflow_job data against a Karpenter-style cluster. Warming-node model, ARC placeholder emulation, DaemonSet + phantom-pod prod-parity toggles, and per-fleet / cluster-wide utilization reports.
- Add optimize.md: design for a per-fleet-family node/pod sizing optimizer targeting max(CPU, mem) allocatable-weighted utilization on HUD data - Define objective split: opt metric (workload-only over post-kubelet capacity) for ranking, cal metric matching prod PromQL for calibration - Specify shape catalog + eligibility (data-driven N, not power-of-2) and a checkpointed, resumable multi-restart hill-climb search - Document 8 design decisions (family-locked, virtual sub-fleets, GPU/ baremetal/reserved scope-out) and 11 risks with mitigations - Lay out phased plan (Phase 0 calibration -> catalog -> search -> sensitivity -> git-apply-able patch deliverables) and file layout Notes: design only, no executable code yet. Scope excludes p4d/p5/p6, -large/-metal variants, reserved-capacity fleets, and c7i-runner (fixed ARC pod shape). Coupling across families is treated as zero on the workload side, so each family is searched independently. Absolute util numbers stay ungrounded until the Phase 0 sim-vs-prod delta is measured; if it exceeds 5pp, recs are reframed as deltas vs baseline. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- add benchmark.py: phase 0 harness (single-run wall-clock, noise floor across seeds, sim-vs-prod calibration incl. per-fleet cal) - add optimize_catalog.py: phase 1 analytical shape catalog + per-def eligibility + theoretical util ceiling, with JSON dump - simulate.py: expose per-pool workload/ds/raw-alloc breakdowns so opt/cal metrics can be derived; snapshot per-bucket arrivals to keep simulate() safe to re-run in-process - instance_specs.py: fill in missing instance sizes (2xl/4xl and metal variants) across CPU/GPU families for catalog enumeration - lint/style cleanups across build_csv, pull_hud, sim_load, sim_nodes, sim_report (imports, S311 noqa, formatting) Notes: The optimizer separates two utilization numbers: opt (workload over allocatable+ds, the ranking metric) and cal (used/alloc from per_pool, matching the prod PromQL/Grafana dashboard). The benchmark harness measures both plus the seed noise floor so later phases know what signal-vs-noise threshold their changes must clear. Phase 1 is purely analytical (no sim runs) — it enumerates every (instance, N) split per fleet family, filters to shapes that fit each runner def, and reports the best fit and a uniform-weighted ceiling. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add optimize_search.py: per-family multi-restart hill-climb over (instance_type, N) shapes, ranked on max(opt_cpu, opt_mem) with node-hours tie-break; SQLite sim cache + resumable checkpoint state - Extract optimize_config.py as single source for scope constants, def_totals/load_defs_by_family, and PROD_PARITY_SIM_FLAGS; rewire optimize_catalog.py to it (dropping the duplicated definitions) - Add last_days job filtering to sim_load.load_jobs, wired through benchmark.py and simulate.py CLIs - Let ClusterModel take a fleets_override so the search can swap in virtual per-(family, instance) sub-fleets - Regenerate optimize_catalog.json Notes: The search runs families as independent spawn subprocesses (D3 independence), keeping c7i-runner injected so runner pods still schedule. The sim cache key pins hashes of every sim/loader/analyzer source file plus the discovered DaemonSet set, so any change that affects results invalidates stale entries. Empty-window families are skipped but a prior best under a wider window is preserved rather than overwritten with a "skipped" verdict. Phase 4 (real patch generation) is still stubbed. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add ProgressDisplay class rendering single-family live terminal progress via rich.live.Live, with a plain carriage-return fallback - Swap loggers' stderr handlers for RichHandler (or bump them to WARNING) so log records don't shred the progress line - Thread progress hooks through _search_family (baseline, restart, step, neighbor advance, best updates, end-of-family) - Add --no-progress opt-out; auto-disable for non-TTY stderr or multi-worker runs; add rich>=13 dependency Notes: Display only runs in the single-process path (workers == 1) because multiprocessing workers can't share a terminal cleanly and ANSI/CR output would corrupt non-TTY logs. All state mutations are lock-guarded so the rich background-refresh thread never races the search thread. When disabled, ProgressDisplay is a null object whose methods are safe no-ops. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Add QueueProgressDisplay: worker-side ProgressDisplay stand-in that forwards state changes to the parent over a multiprocessing.Queue - Add MultiFamilyProgressDisplay: parent aggregator rendering one row per family in a rich.Live panel, with stalled-worker detection - Wire the queue into _family_worker and main() so parallel searches report progress instead of falling back to plain heartbeat logs - Bump worker stderr StreamHandlers to WARNING+ when queued so INFO chatter doesn't interleave into the parent panel; file logs stay full Previously live progress only worked with a single worker; multi-worker runs disabled the display entirely. The queue-fed aggregator lets each family stream restart/step/candidate/best-score updates to a shared panel. Non-TTY or missing-rich falls back to a no-op display while still suppressing INFO noise. Stalled rows surface worker crashes (segfault/OOM) instead of letting them vanish silently. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Reformulate the search from per-def (instance, N) shapes into a partition of a family's defs across sub-nodepools plus a per-subpool instance choice; pod cpu/mem are now derived deterministically via the D4 tight-fit rule instead of being a free search variable - Rewrite optimize_catalog.py as a (def, instance) eligibility oracle and drop the pre-generated optimize_catalog.json - Split optimize_search.py into optimize_engine (search + sim wrapper), optimize_progress (live display), optimize_report (reports + patches), and optimize_storage (SimCache + StateStore) - Add exhaustive/hillclimb/auto modes and a full-dataset validation phase; replace the node-hours tie-breaker with size-invariant vCPU-hours - Add runner_hooks.py as the single source of hooks/runner overhead shared by sim_load and optimize_config so catalog feasibility matches what the sim schedules The prior formulation let the optimizer arbitrarily upsize pod requests to whatever slot maximized utilization, which is not a deployable change — labels encode shape and requests are bounded by what a def needs. The new formulation makes pod adjustment a mechanical byproduct of the (def, instance) pair, bounded by D4 tolerances; any pair needing a larger adjustment is infeasible and pruned before any sim runs. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Replace per-family full-dataset validation with two full-cluster sims (baseline as-is + all improved families' best_configs merged in) - Add cluster-sim helpers to optimize_engine: apply_recommendations_to_jobs, build_cluster_fleets_extra, run_cluster_sim, and cluster/per-family metric extraction - Add ClusterModel.fleets_extra to steer named fleets onto specific instances while unchanged fleets keep their YAML shape - Add ClusterValidationResult and rework reports: cluster-wide validation table plus a per-family "cluster contribution" section - Extract cluster + per-family metrics inside each sim worker so the large sim_out never crosses the process boundary A single full-cluster sim captures cross-family packing and interaction that isolated per-family sims miss. Each family's share is derived by pool-filtering the same two sim outputs (baseline uses original nodepool names, rec uses best_config sub_nodepool_ids), so per-family contribution numbers come for free without running extra sims. The validation phase now runs exactly two sims (2-worker spawn pool) instead of two per family. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- D4 now adjusts only the main pod's integer vcpu/memory, treating the runner-container-hooks sidecar (320 mcpu / 522 MiB) as a fixed tax that rides on top; subtract sidecar before flooring to whole vCPU/GiB - Replace mcpu-based CPU tolerance with integer main-vcpu bounds [min(orig-1, ceil(orig*0.95)), max(orig+1, ceil(orig*1.35))] - def_totals returns main_vcpu/main_memory_gib; catalog, report, and patch emit adjustments in operator YAML units (vcpu:/memory:) with new_main_* - Baseline now uses prod-reality (unadjusted) pod shapes routed to real nodepool names; add is_baseline_feasible physical-fit gate and thread baseline_defs through rebuild_jobs_for_config/run_sim_for_config/cached_sim - Add generated optimize_catalog.json eligibility output Notes: The old baseline was gated against the recommendation catalog, which enforces D4 bounds — so a perfectly-fine prod config could be rejected when its tight-fit on the largest instance overshot the upper bound. The baseline must reflect what prod actually runs (original shapes, real weighted fleet from YAML), so it is now checked only for physical fit and cached separately from recommendation sims to avoid shape collisions. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- Check baseline fit against every instance in the real fleet's weighted list, not just the single reference instance - Load real fleets via ClusterModel and thread them into is_baseline_feasible - Drop GPU sign-parity gate; physical fit on any fleet member is the only invariant - Pick the reference instance as the largest prod choice by vCPU from INSTANCE_SPECS, keeping spec["instance"] display-only Baseline routing mirrors prod: pods go to the real nodepool and Karpenter picks a fitting instance per pod at scheduling time. Gating feasibility on a single family-largest instance wrongly skipped families whose pods fit some (but not the biggest) fleet member, and enforcing alloc_gpu == 0 for rare 0-GPU defs in a GPU family's GPU-only fleet was never satisfiable. The correct check is "does the original pod shape fit on at least one fleet member?". Signed-off-by: Jean Schmidt <contato@jschmidt.me>
- add optimize_pricing: static on-demand price table (per-region + blended, reserved GPU families surface as unpriced) joined to instance specs - add optimize_cost: node-hours and USD post-processing of sim node-count buckets, with relative savings and family/cluster cost report blocks - add Phase 2.5 optimize_runner_fleet + report: closed-form cheapest instance to host the fixed 750m/1Gi ARC runner pods, dual amd64/arm64 winners - thread cost through the engine (cost_for_config), search (--skip-runner-fleet, --runner-fleet-arch), cluster validation, and family/global reports - emit per-type node_counts_by_type from simulate; label vcpu_hours a compute proxy vs authoritative node-hours x price - add unit tests for cost, pricing, engine, report, runner_fleet, search, storage, and simulate; scope coverage omit to untested sweep scripts Notes: Absolute dollars are approximate (on-demand list price, sim node-hours are a lower bound); relative percentage/ratio deltas are the trustworthy figures. arm64 runner-fleet winners are gated behind a multi-arch image + warmer DaemonSet, so amd64 is the actionable pick today. Reserved Capacity-Block families (p5, p6-b200) are intentionally absent from the price table. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
Capacity reportcommit ✅ simulate-cluster✅ analyze-utilization |
tofu plan — meta-prod-aws-ue2✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-uw1✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue2✅ Plan succeeded · commit Plan output |
- Reword per-family rec cost caveat: priced-$ tracks vcpu_hours x $/vcpu; node_hours is a size-blind count, not the cost signal - Update test_optimize_cost assertion to match new caveat text - Update test_optimize_report assertion to match new caveat text The old "node-hours x price = authoritative cost" phrasing was misleading — node_hours is size-blind and does not reflect the priced dollar figure, which is driven by vcpu_hours x $/vcpu. The new wording steers readers to the actual cost signal. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
huydhn
pushed a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jul 7, 2026
…er pool to Graviton (pytorch#871) **Impact:** ARC runners and Karpenter node provisioning on `meta-prod-aws-ue1` only (pilot cluster); all other OSDC clusters unchanged **Risk:** medium (single-cluster canary; other clusters stay on the current baseline behavior) Simulation/Optimization scripts: pytorch#870 ## What Rolls out the fleet-packing optimization to a **single cluster** (`meta-prod-aws-ue1`) instead of fleet-wide, using new `-opt` shim modules so the rest of the fleet is untouched: - **Base modules restored to baseline.** `modules/nodepools/defs` and `modules/arc-runners/defs` are reverted to their pre-optimization state, so every other cluster provisions exactly as before. - **New `nodepools-opt` + `arc-runners-opt` modules** hold the optimized definitions. They are thin shims that delegate to the base `deploy.sh` via env vars (same pattern as the existing `nodepools-h100` / `arc-runners-h100` GPU shims), so base and `-opt` share one generator — only the def files differ. `arc-runners-opt` additionally points `NODEPOOLS_DEFS_DIR` at `nodepools-opt/defs` so region-exclusion resolves against the optimized fleets. - **Only `meta-prod-aws-ue1`** is switched to the `-opt` modules in `clusters.yaml` (module list swap + a small `arc-runners-opt:` config block carrying the `0.8` fresh-multiplier). `meta-prod-aws-ue1` and `meta-prod-aws-ue2` remain in the same capacity-aware HUD shard group, and the integration/workload test harness recognizes the `-opt` modules. The optimization itself (embodied in the `-opt` defs): splits the monolithic per-family Karpenter fleets (`c7a`, `c7i`, `m7i`, `m8g`, `r7a`, `r7i`, `g4dn`, `g5`, `g6`) into per-size tier nodepools where each tier's ideal instance sits at weight 100 and falls back *up* to larger sizes, then re-pins each runner def to its matching tier via `node_fleet` and grows the runner's vcpu/memory to fill the node. Also switches the `c7i-runner` pool from Intel c7i to Graviton (m7g/m8g, arm64) while keeping the load-bearing `node-fleet=c7i-runner` taint name. ## Why The old single-fleet-per-family design let Karpenter satisfy a small runner by spinning up (or bin-packing onto) an oversized node — e.g. an 8-vcpu job landing on a `.48xlarge` — which stranded capacity and hurt packing efficiency. Anchoring each runner to a size-matched tier with larger-only fallback gives Karpenter a tight, predictable node choice and lets the runner request nearly the whole node. The Graviton switch on the runner pool trades Intel for cheaper arm64 capacity; the runner image and ARC hooks are already multi-arch. Rolling out to one cluster first (rather than fleet-wide) limits blast radius: `meta-prod-aws-ue1` acts as a canary while every other cluster stays on the proven baseline. If the pilot regresses, only ue1 is affected and the change is reverted by pointing ue1 back at the base modules. ## Note Fleet | Baseline util (opt_max) | Optimized util (opt_max) | Δ (pp) | Priced-$ Δ | Notes | |:------|------------------------:|-------------------------:|-------:|-----------:|:------| | c7a | 12.4% | 56.9% | +44.52 | -75.3% | biggest win; 48xl → 4 tiers | | r7i | 62.0% | 81.2% | +19.22 | -10.0% | | | g5 | 72.5% | 88.0% | +15.45 | -20.1% | GPU family | | g6 | 72.6% | 87.6% | +15.01 | -25.4% | GPU family | | m7g | 66.4% | 79.4% | +12.99 | +1.1% | | | m8g | 68.0% | 80.2% | +12.28 | +1.5% | old fleet → release-only stub | | c7i | 40.9% | 51.0% | +10.14 | -25.6% | | | r7a | 75.3% | 85.4% | +10.09 | +5.4% | old fleet → release-only stub | | g4dn | 84.2% | 90.0% | +5.79 | -7.3% | GPU family | | m7i | 42.6% | 48.4% | +5.75 | -5.0% | | | m6i | 84.5% | 84.7% | +0.18 | +0.0% | **skipped (noise)** — NOT changed | | **Cluster-wide** | **70.5%** | **81.3%** | **+10.9** | **-5.8%** | 61-day full-dataset validation | Cluster-wide priced spend: baseline ~$1,992,012 → rec ~$1,877,065 (approx on-demand list price, excludes reserved GPU). Priced-$ Δ negative = cheaper. ## Changes per fleet | Fleet | Sub-nodepools created | Runner defs reshaped | Old fleet | Key change | |:------|:----------------------|---------------------:|:----------|:-----------| | c7a | c7a-2xl, c7a-8xl, c7a-12xl, c7a-16xl | 5 | deleted | split 48xl into 4 size tiers | | r7i | r7i-8xl, r7i-16xl | 2 | deleted | downsized 48xl → 8xl/16xl | | g5 | g5-8xl, g5-12xl | 2 | deleted | GPU-count split 1-GPU/4-GPU | | g6 | g6-8xl, g6-12xl | 2 | deleted | GPU-count split 1-GPU/4-GPU | | m7g | m7g-16xl | 1 | deleted | 8xl → 16xl, 3 runners/node | | m8g | m8g-16xl | 1 | release-stub | 48xl → 16xl; old fleet stubbed | | c7i | c7i-8xl, c7i-12xl | 4 | deleted | split 12xl into 8xl/12xl | | r7a | r7a-8xl, r7a-16xl, r7a-48xl | 4 | release-stub | split 48xl into 3 tiers | | g4dn | g4dn-8xl, g4dn-12xl | 2 | deleted | GPU-count split 1-GPU/4-GPU | | m7i | m7i-8xl, m7i-12xl, m7i-48xl | 4 | deleted | split 48xl into 3 tiers | ``` $ rm -rf scripts/node-size-sweep/output ; uv run scripts/node-size-sweep/optimize_search.py \ --last-days 35 --drop-provider lf --keep-fraction 0.5 \ --num-workers $(sysctl -n hw.ncpu) --num-restarts 20 \ --search-mode auto 2026-07-05 08:14:51,222 [global] INFO output_dir=/Users/jschmidt/meta/ci-infra-paralle-task-2/osdc/scripts/node-size-sweep/output/20260705T151451Z-4b52af9 2026-07-05 08:14:51,316 [global] INFO hashing sim source files 2026-07-05 08:14:53,413 [global] INFO hashing CSV 2026-07-05 08:14:53,505 [global] INFO loading CSV /Users/jschmidt/meta/ci-infra-paralle-task-2/osdc/scripts/node-size-sweep/pytorch_60d.csv (last_days=35) dropped by downsample: 1,316,846 filtered to last 35 days: 1,795,132 jobs kept, 833,236 dropped 2026-07-05 08:14:57,855 [global] INFO loaded 1795132 jobs 2026-07-05 08:14:59,498 [global] INFO sim_flags prod-parity: daemonsets_in_metric=True phantom_pods_enabled=True empty_ttl_buckets=1 2026-07-05 08:14:59,510 [global] INFO dispatching 11 families across 16 workers [07/05/26 08:15:20] INFO global heartbeat: 1/11 families done, elapsed=21s, eta=210s INFO global heartbeat: 2/11 families done, elapsed=21s, eta=96s [07/05/26 08:15:28] INFO global heartbeat: 3/11 families done, elapsed=29s, eta=78s [07/05/26 08:15:30] INFO global heartbeat: 4/11 families done, elapsed=31s, eta=54s [07/05/26 08:15:31] INFO global heartbeat: 5/11 families done, elapsed=32s, eta=38s [07/05/26 08:15:34] INFO global heartbeat: 6/11 families done, elapsed=35s, eta=29s [07/05/26 08:15:43] INFO global heartbeat: 7/11 families done, elapsed=44s, eta=25s [07/05/26 08:16:25] INFO global heartbeat: 8/11 families done, elapsed=86s, eta=32s [07/05/26 08:19:10] INFO global heartbeat: 9/11 families done, elapsed=251s, eta=56s [07/05/26 08:24:36] INFO global heartbeat: 10/11 families done, elapsed=577s, eta=58s [07/05/26 09:01:12] INFO global heartbeat: 11/11 families done, elapsed=2773s, eta=0s c7a DONE opt_max 0.5693 (baseline 0.1241, +44.5pp) [improved] c7i DONE opt_max 0.5105 (baseline 0.4090, +10.1pp) [improved] g4dn DONE opt_max 0.8998 (baseline 0.8419, +5.8pp) [improved] g5 DONE opt_max 0.8799 (baseline 0.7253, +15.5pp) [improved] g6 DONE opt_max 0.8759 (baseline 0.7259, +15.0pp) [improved] m6i DONE opt_max 0.8471 (baseline 0.8453, +0.2pp) [improved] m7g DONE opt_max 0.7935 (baseline 0.6637, +13.0pp) [improved] m7i DONE opt_max 0.4837 (baseline 0.4261, +5.8pp) [improved] c7a DONE opt_max 0.5693 (baseline 0.1241, +44.5pp) [improved] c7i DONE opt_max 0.5105 (baseline 0.4090, +10.1pp) [improved] g4dn DONE opt_max 0.8998 (baseline 0.8419, +5.8pp) [improved] g5 DONE opt_max 0.8799 (baseline 0.7253, +15.5pp) [improved] g6 DONE opt_max 0.8759 (baseline 0.7259, +15.0pp) [improved] m6i DONE opt_max 0.8471 (baseline 0.8453, +0.2pp) [improved] m7g DONE opt_max 0.7935 (baseline 0.6637, +13.0pp) [improved] m7i DONE opt_max 0.4837 (baseline 0.4261, +5.8pp) [improved] m8g DONE opt_max 0.8024 (baseline 0.6796, +12.3pp) [improved] r7a DONE opt_max 0.8543 (baseline 0.7534, +10.1pp) [improved] r7i DONE opt_max 0.8121 (baseline 0.6199, +19.2pp) [improved] 2026-07-05 09:01:12,841 [global] INFO cluster validation: loading full dataset (no --last-days filter) from /Users/jschmidt/meta/ci-infra-paralle-task-2/osdc/scripts/node-size-sweep/pytorch_60d.csv dropped by downsample: 1,316,846 2026-07-05 09:01:17,054 [global] INFO cluster validation: loaded 2628368 jobs (vs 35 in search window) 2026-07-05 09:01:17,944 [global] INFO cluster validation: dispatching 2 sims (baseline + recommendation) across 2 workers 2026-07-05 09:01:22,158 [global] INFO cluster sim baseline: starting (2628368 jobs, 0 extra fleets) 2026-07-05 09:01:25,193 [global] INFO cluster sim recommendation: starting (2628368 jobs, 23 extra fleets) 2026-07-05 09:01:52,095 [global] INFO cluster sim baseline: done opt_max=0.7050 vcpu_hours=27323429 node_hours=164059 elapsed=29.9s 2026-07-05 09:02:14,323 [global] INFO cluster sim recommendation: done opt_max=0.8135 vcpu_hours=26139938 node_hours=544071 elapsed=49.1s 2026-07-05 09:02:14,536 [global] INFO cluster validation: baseline opt_max=70.5% rec opt_max=81.3% delta=+10.85pp elapsed=56.6s 2026-07-05 09:02:14,657 [global] INFO Phase 2.5: runner-fleet host search (arch=('amd64', 'arm64')) 2026-07-05 09:02:15,817 [global] INFO Phase 2.5: best amd64=c7i.12xlarge (-8.3% vs baseline); best arm64=m7g.12xlarge 2026-07-05 09:02:15,977 [global] INFO all done. Reports in /Users/jschmidt/meta/ci-infra-paralle-task-2/osdc/scripts/node-size-sweep/output/20260705T151451Z-4b52af9/reports ``` ### Do we have room to reduce pod requests and get better packing? I asked claude to have a quick look, and seems that before getting substantial results, we should reduce the size subtantialy, what makes this as a probably no-go: ``` The prize (r7a + g5, ~59% of cluster) g5 (GPU): $0. No lever at all. Confirmed empirically — both g5 defs are GPU-count-bound at N=1, so shrinking CPU/mem changes nothing across the entire 1.0→0.5× sweep. GPU families need a GPU-packing strategy, not request reduction. (My prediction held.) r7a (CPU): real, and it's bimodal — this is the important finding: ┌─────────────────────┬──────────────┬─────────────────┐ │ pod request │ r7a+g5 saved │ $/yr unlocked │ ├─────────────────────┼──────────────┼─────────────────┤ │ 0.9× │ ~1% │ trivial │ ├─────────────────────┼──────────────┼─────────────────┤ │ 0.8× ("moderate") │ 1.2% │ ~$101k — a trap │ ├─────────────────────┼──────────────┼─────────────────┤ │ 0.7× (the knee) │ 17.4% │ ~$1.5M │ ├─────────────────────┼──────────────┼─────────────────┤ │ 0.6× ("aggressive") │ 24.1% │ ~$2.1M │ └─────────────────────┴──────────────┴─────────────────┘ ``` ### Biggest current cost usage ``` ┌─────────────────────────────────┬─────────────────┐ │ Bucket │ % of node-hours │ ├─────────────────────────────────┼─────────────────┤ │ Actively useful (≥40% packed) │ 61.5% │ ├─────────────────────────────────┼─────────────────┤ │ Straggler/fragmentation (2-40%) │ ~21% │ ├─────────────────────────────────┼─────────────────┤ │ Empty-lingering │ ~10.7% │ ├─────────────────────────────────┼─────────────────┤ │ Warming/startup │ ≤6.8% │ └─────────────────────────────────┴─────────────────┘ ``` --------- Signed-off-by: Jean Schmidt <contato@jschmidt.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full run and optimal values: