feat(ci): dispatch a CI run to a GPU generation - #2703
Open
yueming-yuan wants to merge 1 commit into
Open
Conversation
Adds the second axis. PR 2 gave every CUDA test a `hardware=[...]` capability; this lets a PR say which of those generations the run should execute on, and routes tests accordingly. (none) each test runs once, at its home stage run-on-hopper Hopper stages only; Blackwell-only tests do not run run-on-blackwell everything that supports Blackwell moves there both run-on-* a portable test runs twice, once per generation run-ci-blackwell-only the Blackwell-exclusive tests, across all domains `absorb` is the safety valve: only an explicit `run-on-*` lets a test leave its home stage. Plain PRs, nightly, weekly, release and called workflows all resolve to absorb=False, where the home-stage invariant makes routing unreachable -- so their selection is byte-identical to before this commit. Routing is derived, not tabulated: the destination is the smallest stage on the target generation with enough GPUs. A hand-written home -> destination map would encode a fleet shape that does not exist yet. The rule gives today's degenerate answer (one Blackwell stage takes everything) and a 1:1 mirror once a second Blackwell host is partitioned, from the same three lines. No new flag on run_suite.py and no workflow edit: a stage belongs to exactly one generation, so `--suite` already names it. An `--arch` flag would be a second copy of that fact, free to drift. hardware.py grows num_gpus and runs_on and becomes the single source of truth -- CI_SUITES and CUDA_SUITE_RUNS_ON now derive from it. stage_selection shares run_suite's predicate, without which a run-on-blackwell PR would still allocate every Hopper runner and then find nothing to run on them. The comment gateway's label pattern accepts run-on-* so `/run-on-blackwell` works. That does not widen what a comment may add: allowed_labels stays an exact default-deny allowlist and the pattern only rejects malformed entries in it. Also corrects the B200 host's README entry -- it runs the docker-compose flow, not the gh-runner image.
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chain 4/5 — stacked on
b200-ci-baseline-identity.The second axis. 2/5 gave every CUDA test a
hardware=[...]capability; this lets a PR say which of those generations the run should execute on, and routes tests accordingly.run-on-hopperrun-on-blackwellrun-ci-blackwell-onlySo
run-ci-megatron+run-on-blackwell= every megatron test that can run on Blackwell, on Blackwell, and nowhere else.Safety
absorbis the valve: only an explicitrun-on-*lets a test leave its home stage. Plain PRs, nightly, weekly, release and called workflows all resolve toabsorb=False, where the home-stage invariant makes routing unreachable — so their selection is byte-identical to before this commit. A test pins that property across every stage, arch and dispatch combination.Design notes
Routing is derived, not tabulated. The destination is the smallest stage on the target generation with enough GPUs. A hand-written home → destination map would encode a fleet shape that does not exist yet; this rule gives today's degenerate answer (one Blackwell stage takes everything) and a 1:1 mirror once a second Blackwell host is partitioned, from the same three lines.
No new flag, no workflow edit. A stage belongs to exactly one generation, so
--suitealready names it. An--archflag would be a second copy of that fact, free to drift out of step.One predicate.
stage_selectionsharesrun_suite'sruns_in_stage; without that arun-on-blackwellPR would still allocate every Hopper runner and then find nothing to run on them.hardware.pygrowsnum_gpus/runs_onand becomes the single source of truth —CI_SUITESandCUDA_SUITE_RUNS_ONnow derive from it.Comment gateway
LABEL_PATTERNacceptsrun-on-*so/run-on-blackwellworks. This does not widen what a comment may add:allowed_labelsstays an exact default-deny allowlist, and the pattern only rejects malformed entries in it.🤖 Generated with Claude Code