refactor: remove devices related gflags.#1948
Open
XuZhang99 wants to merge 5 commits into
Open
Conversation
XuZhang99
requested review from
Clement-Wang26,
DongheJin,
DragonFive,
JimHsiung,
Kang-Meng,
RobbieLeung,
liujinguang0125,
liutongxuan,
walsonyang,
xiao-yu-chen,
yingxudeng,
yq33victor and
zhang-minchao
as code owners
July 15, 2026 11:32
weizhehuang0827
previously approved these changes
Jul 15, 2026
Collaborator
|
在 PR1929 的实际合入提交上叠加了 PR1948 的两个提交,并完成 Qwen3.5-2B MTP 冒烟和性能验证。 测试条件
功能验证
服务端性能结果
叠加后的数值略慢,但两次重复间存在明显波动,尤其 MTP5 两次 E2E 分别为 158.498 ms 和 155.098 ms。PR1948 本身没有修改 draft、verify 或 decode 热路径,因此估计这里是测试波动。 仅从 PR1929 相关功能兼容性看,应该是没问题的。 |
XuZhang99
force-pushed
the
feat/auto-device-discovery
branch
from
July 17, 2026 02:06
2a11f70 to
79da539
Compare
…s/--device_id flags Remove the --devices, --device_id, and --draft_devices CLI flags along with their c_api/cc_api/pybind surfaces. Device selection now defaults to "auto": each process enumerates its visible devices via Platform::device_count(), honoring *_VISIBLE_DEVICES masks. Add a unified Platform::current_device() (NPU/CUDA/MLU/DCU/MUSA). Rework DistManager::setup_multi_node_workers for multi-process serving: the parsed devices vector is now the visible-card pool rather than "cards this process uses". Each process runs exactly one worker bound to devices[node_rank], so world_size == nnodes instead of devices.size()*nnodes. This keeps every process able to see all cards (so ATB can build the cross-card HCCL comm group) while deriving worker cardinality from nnodes/node_rank. Verified: 2-process TP=2 Qwen3-8B serves correctly, gsm8k mean_acc=0.9234. Update launch docs/examples to select devices via *_VISIBLE_DEVICES instead of --devices. Co-Authored-By: Claude <noreply@anthropic.com>
XuZhang99
force-pushed
the
feat/auto-device-discovery
branch
from
July 17, 2026 02:37
79da539 to
e844363
Compare
The CP-config validation in Master computed global_world_size as nnodes * devices.size(), which double-counts now that the devices vector is the visible-card pool rather than the per-process worker list. Multi-process serving runs one worker per process, so the global world size is the node count. Aligns this path with DistManager::setup_multi_node_workers. Co-Authored-By: Claude <noreply@anthropic.com>
…re used Multi-process serving requires every process to see all cards (so ATB can build the cross-card HCCL comm group); each process then binds the card selected by its --node_rank. Masking each process to a single card (<VAR>_VISIBLE_DEVICES=$DEVICE) makes the visible-device pool size 1 and trips the node_rank bounds check for rank>=1. Revert the launch snippets to invoke the binary without a per-process mask (no mask means all cards are visible), distinguishing processes only by --node_rank. Preserve the numactl CPU-affinity $DEVICE usage and fix stray backslash-continuation lines. Co-Authored-By: Claude <noreply@anthropic.com>
The earlier flag-removal pass dropped the global export <VAR>_VISIBLE_DEVICES line and left dead START_DEVICE/DEVICE shell vars. Restore the global export (the intended way to pick cards now that --devices is gone; a process sees the masked cards and binds by --node_rank) and remove the dead vars. Keep DEVICE in glm_5/kimi2_5 where numactl still uses it for CPU affinity. Multi-card cookbook examples that never had a global export stay bare (no mask means all cards visible). Co-Authored-By: Claude <noreply@anthropic.com>
Select the card this process owns from the visible pool by node_rank, so options.devices() holds exactly one device (the card this process uses) instead of the whole visible pool. This mirrors the historical single-element devices semantics (previously set via --devices="npu:$DEVICE"): DeviceNameUtils:: to_string(devices) prints "npu:N" rather than "npu:0,npu:1", and DeviceMonitor registers only this process's card. dist_manager reverts to deriving each_node_ranks/world_size from devices.size()==1, so world_size==nnodes. Verified: 2-process TP=2 Qwen3-8B, node_0 devices=npu:0, node_1 devices=npu:1, world_size=2, inference correct. Co-Authored-By: Claude <noreply@anthropic.com>
DongheJin
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove the
--devices,--device_id, and--draft_devicesCLI flags along with theirc_api/cc_api/pybindsurfaces. Device selection now defaults to"auto": each process enumerates its visible devices viaPlatform::device_count(), honoring*_VISIBLE_DEVICESmasks. Add a unifiedPlatform::current_device() (NPU/CUDA/MLU/DCU/MUSA).Rework
DistManager::setup_multi_node_workersfor multi-process serving: the parsed devices vector is now the visible-card pool rather than "cards this process uses". Each process runs exactly one worker bound to devices[node_rank], soworld_size == nnodesinstead ofdevices.size()*nnodes. This keeps every process able to see all cards (so ATB can build the cross-card HCCL comm group) while deriving worker cardinality from nnodes/node_rank. Verified: 2-process TP=2 Qwen3-8B serves correctly, gsm8k mean_acc=0.9234.Update launch docs/examples to select devices via *_VISIBLE_DEVICES instead of --devices.
Description
Related Issues
Change Type
Pull Request Checklist
Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.
PR Title and Commit Messages
<type>: <subject>.Pre-commit Checks
pre-commitby runningpip install pre-commitor an equivalent command.pre-commit install.pre-commit run --all-filesand fixed any reported issues.Self Review
.agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.mainbranch.Build and Test Coverage
python setup.py build testhas passed on a CUDA machine.python setup.py build testhas passed on an NPU machine.python setup.py build testhas passed on an MLU machine.Reviewer Notes