feat(ci): turn on the Blackwell-only tests - #2704
Open
yueming-yuan wants to merge 2 commits into
Open
Conversation
The B200 runner is online and radixark/miles:dev is verified on sm100 (torch 2.11.0+cu130, capability (10, 0), 8 devices, a bf16 matmul), so the three tests parked on missing hardware can run: test_joyai_llm_flash_mxfp8 megatron 3600s test_glm5_2_744b_a40b_5layer_nvfp4 model-scripts 3600s test_nvfp4_quantizer precision 60s test_deepseek_v32_5layer_mxfp8 stays disabled. Its reason is "superseded by test_deepseek_v32_5layer_ci on H100", not missing hardware, so hardware arriving is not an argument for re-enabling it. They keep their real domain labels rather than being gated behind a Blackwell-specific one. Measured against the Hopper critical path per label: run-ci-megatron H worst shard 1.69h B200 1.00h off critical path run-ci-model-scripts H worst shard 0.87h B200 1.00h +8 min run-ci-precision H worst shard 0.05h B200 0.02h off critical path nightly / weekly B200 2.02h single shard, against a 6h job timeout Eight minutes on model-scripts PRs is not worth a gate, and gating would leave the Blackwell paths uncovered by default -- which is how they ended up parked. joy_ai_llm_flash's ScriptArgs already defaults to B200; passing it explicitly states the intent rather than inheriting it.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
yueming-yuan
added
the
run-ci-blackwell-only
Its first CI run ever -- it had been disabled since it was added -- hung in tools/convert_hf_to_torch_dist.py, not in training. Four ranks reached the barrier in Megatron's dist_checkpointing execute_sync and rank 0's NCCL watchdog killed the job after the 600s default: WorkNCCL(SeqNum=9, OpType=ALLREDUCE, NumelIn=1) timed out #0 barrier torch/distributed/distributed_c10d.py:5030 #2 execute_sync Megatron-LM/.../strategies/async_utils.py:99 #7 main tools/convert_hf_to_torch_dist.py:131 Not a Blackwell problem: test_glm5_2_744b_a40b_5layer_nvfp4 passed in the same job, and nvfp4 kernels ran fine. It converts with --nproc-per-node 1, so it has no cross-rank barrier to time out on; this one uses 4. Not a slow disk either. The save wrote 22G of a ~50B-parameter MoE in 13 minutes, about 28 MB/s, while the host's NVMe measures 2.8 GB/s write and 3.2 GB/s read. Whatever stalls that path, raising the NCCL timeout only makes it hang longer. scripts/run_joy_ai_llm_flash.py stays -- it is a documented launch script and this only removes the CI wrapper that never worked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chain 5/5 — stacked on
b200-ci-dispatch. The first PR in the chain that changes what CI actually runs.The B200 runner is online and
radixark/miles:devis verified on sm100 (torch 2.11.0+cu130, capability(10, 0), 8 devices, a bf16 matmul), so the three tests parked on missing hardware can run:est_timetest_joyai_llm_flash_mxfp8megatrontest_glm5_2_744b_a40b_5layer_nvfp4model-scriptstest_nvfp4_quantizerprecisiontest_deepseek_v32_5layer_mxfp8stays disabled. Its reason is "superseded by test_deepseek_v32_5layer_ci on H100", not missing hardware, so hardware arriving is not an argument for re-enabling it.Why no Blackwell-specific gate
They keep their real domain labels. Measured against the Hopper critical path per label:
run-ci-megatronrun-ci-model-scriptsrun-ci-precisionEight minutes on model-scripts PRs is not worth a gate, and gating would leave the Blackwell paths uncovered by default — which is how they ended up parked in the first place.
joy_ai_llm_flash'sScriptArgsalready defaults toB200; passing it explicitly states the intent rather than inheriting it.🤖 Generated with Claude Code