Repository navigation
[Bugfix][Distributed] Guard NCCL registration with expandable segments - #761
Merged
Merged
Conversation
…cies Fill mode and cudagraph_mode independently only when they are unset in both SM70 Flash-V100 graph policies. Preserve explicit NONE, PIECEWISE, STOCK_TORCH_COMPILE and the other enum values without disabling the default baseline for unconfigured fields. Report the resolved policy modes instead of hard-coded defaults. Add 75 CPU-only source-policy regression cases using the real policy body, enums and environment readers with mocked hardware, and document the precedence rule. No Torch, GPU or model download is required by these isolated tests. Related: #614 (configuration precedence only). This broader alternative to #727 also covers non-NONE overrides and the no-compile decode policy; it does not claim to fix the separate dynamic-shape inference crash. Validation: 58 failures / 17 passes with the unchanged main policy, 75 passes after the fix; repository pre-commit hooks pass, including Ruff, typos, mypy and documentation checks. Signed-off-by: Zhaochengggg <87113558+zhaochengggg@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…d_tokens The SM70 Flash-V100 graph setup picks its own capture sizes and sets max_cudagraph_capture_size to their maximum. The generic sizing afterwards drops capture sizes above max_num_batched_tokens, and since both values are set, a cap above the largest remaining size is treated as contradicting user input: with speculative decoding (sizes up to 64) and max_num_batched_tokens=50 the config fails with "customized max_cudagraph_capture_size(=64) should be consistent with the max value of cudagraph_capture_sizes(=48)". Take the cap from the capture sizes that fit the token budget, in both SM70 setups (compile graph and decode-graph-no-compile). Sizes above the budget were dropped before as well, so the captured graphs do not change. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
… shard _advise_random_file_access requires the PLE disk-tier shards to be file-backed and looks the address up in /proc/self/maps. Shared anonymous memory (for example mmap.mmap(-1, n) with its default flags) is listed there as "/dev/zero (deleted)" and passed the check as a file, so the disk tier kept such a table in memory that only swap can evict while it reported file-backed shards. The check rejects /dev/zero now. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…cies Fill mode and cudagraph_mode independently only when they are unset in both SM70 Flash-V100 graph policies. Preserve explicit NONE, PIECEWISE, STOCK_TORCH_COMPILE and the other enum values without disabling the default baseline for unconfigured fields. Report the resolved policy modes instead of hard-coded defaults. Add 75 CPU-only source-policy regression cases using the real policy body, enums and environment readers with mocked hardware, and document the precedence rule. No Torch, GPU or model download is required by these isolated tests. Related: #614 (configuration precedence only). This broader alternative to #727 also covers non-NONE overrides and the no-compile decode policy; it does not claim to fix the separate dynamic-shape inference crash. Validation: 58 failures / 17 passes with the unchanged main policy, 75 passes after the fix; repository pre-commit hooks pass, including Ruff, typos, mypy and documentation checks. Signed-off-by: Zhaochengggg <87113558+zhaochengggg@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…d_tokens The SM70 Flash-V100 graph setup picks its own capture sizes and sets max_cudagraph_capture_size to their maximum. The generic sizing afterwards drops capture sizes above max_num_batched_tokens, and since both values are set, a cap above the largest remaining size is treated as contradicting user input: with speculative decoding (sizes up to 64) and max_num_batched_tokens=50 the config fails with "customized max_cudagraph_capture_size(=64) should be consistent with the max value of cudagraph_capture_sizes(=48)". Take the cap from the capture sizes that fit the token budget, in both SM70 setups (compile graph and decode-graph-no-compile). Sizes above the budget were dropped before as well, so the captured graphs do not change. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
… shard _advise_random_file_access requires the PLE disk-tier shards to be file-backed and looks the address up in /proc/self/maps. Shared anonymous memory (for example mmap.mmap(-1, n) with its default flags) is listed there as "/dev/zero (deleted)" and passed the check as a file, so the disk tier kept such a table in memory that only swap can evict while it reported file-backed shards. The check rejects /dev/zero now. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…pandable segments NCCL registers the buffers of collectives captured in a CUDA graph and assumes each one lies in a single physical allocation. With PyTorch expandable segments a tensor can span several, and a later graph replay can fault inside the collective (pytorch/pytorch#158029; NVIDIA's CUDA graph guide gives NCCL_GRAPH_REGISTER=0 as the workaround). NCCL caches the setting on first use, so env_override, which runs on `import vllm` before any communicator exists, now defaults it to 0 whenever the allocator configuration PyTorch selects enables expandable segments. An explicit NCCL_GRAPH_REGISTER is kept. On 4x V100 PCIe with TP4 and DCP2 (custom all-reduce is disabled there, so every collective in a decode graph is NCCL), an NCCL AllGather in the decode graph faulted with an out-of-range address in 7 of 10 server starts; with the setting, 3 of 3 starts ran through the same step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leon <tomsel1010.us2@gmail.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
yangzhuxinyzx
marked this pull request as ready for review
October 2, 2026 12:54
Purpose: prevent SM70 graph policies from overwriting explicit settings or producing capture caps above the usable scheduler token budget. Changes: combine #733 and #725, preserve explicit compilation/graph fields, derive caps from actual capture sizes, and cover their interaction. Tests: 91 policy tests and 32 SM70 startup/profile/cache tests passed; scoped pre-commit and repository CI passed. Not verified: CUDA graph capture, model execution, or performance. Existing release acceleration defaults are retained.
… shard _advise_random_file_access requires the PLE disk-tier shards to be file-backed and looks the address up in /proc/self/maps. Shared anonymous memory (for example mmap.mmap(-1, n) with its default flags) is listed there as "/dev/zero (deleted)" and passed the check as a file, so the disk tier kept such a table in memory that only swap can evict while it reported file-backed shards. The check rejects /dev/zero now. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Purpose: prevent anonymous shared memory from being accepted as a reclaimable PLE disk shard. Changes: reject /dev/zero mappings while preserving real file-backed shard handling and the original contributor's fix. Tests: real anonymous/file mmap regression and 32 SM70 startup/profile/cache checks passed; scoped pre-commit and repository CI passed. Not verified: full model loading or GPU execution. No numerical policy or runtime acceleration default changed. Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…pandable segments NCCL registers the buffers of collectives captured in a CUDA graph and assumes each one lies in a single physical allocation. With PyTorch expandable segments a tensor can span several, and a later graph replay can fault inside the collective (pytorch/pytorch#158029; NVIDIA's CUDA graph guide gives NCCL_GRAPH_REGISTER=0 as the workaround). NCCL caches the setting on first use, so env_override, which runs on `import vllm` before any communicator exists, now defaults it to 0 whenever the allocator configuration PyTorch selects enables expandable segments. An explicit NCCL_GRAPH_REGISTER is kept. On 4x V100 PCIe with TP4 and DCP2 (custom all-reduce is disabled there, so every collective in a decode graph is NCCL), an NCCL AllGather in the decode graph faulted with an out-of-range address in 7 of 10 server starts; with the setting, 3 of 3 starts ran through the same step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Leon <tomsel1010.us2@gmail.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
NCCL graph-buffer registration assumes a compatible allocation layout. With PyTorch expandable segments, that assumption can fail during graph replay.
Changes
Integrate #712 with the original author retained. Choose the safe existing NCCL override only when expandable segments are explicitly enabled, before communicator initialization. Preserve explicit user choices and allocator-variable precedence.
Closes #712.
Test Plan and Result
The scoped merge gate passed: 32 SM70 startup/profile/cache regression tests, 8 allocator/default/import regressions, and scoped pre-commit checks. PyTorch 2.10 allocator headers were checked for variable precedence. GPU collectives and model execution were not exercised by the CPU checks.
Prepared with AI assistance; source changes and test results were reviewed.