Skip to content

refactor: remove devices related gflags.#1948

Open
XuZhang99 wants to merge 5 commits into
xLLM-AI:mainfrom
XuZhang99:feat/auto-device-discovery
Open

refactor: remove devices related gflags.#1948
XuZhang99 wants to merge 5 commits into
xLLM-AI:mainfrom
XuZhang99:feat/auto-device-discovery

Conversation

@XuZhang99

@XuZhang99 XuZhang99 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Remove the --devices, --device_id, and --draft_devices CLI flags along with their c_api/cc_api/pybind surfaces. Device selection now defaults to "auto": each process enumerates its visible devices via Platform::device_count(), honoring *_VISIBLE_DEVICES masks. Add a unified Platform::current_device() (NPU/CUDA/MLU/DCU/MUSA).

Rework DistManager::setup_multi_node_workers for multi-process serving: the parsed devices vector is now the visible-card pool rather than "cards this process uses". Each process runs exactly one worker bound to devices[node_rank], so world_size == nnodes instead of devices.size()*nnodes. This keeps every process able to see all cards (so ATB can build the cross-card HCCL comm group) while deriving worker cardinality from nnodes/node_rank. Verified: 2-process TP=2 Qwen3-8B serves correctly, gsm8k mean_acc=0.9234.

Update launch docs/examples to select devices via *_VISIBLE_DEVICES instead of --devices.

Description

Related Issues

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

@pjgao

pjgao commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

在 PR1929 的实际合入提交上叠加了 PR1948 的两个提交,并完成 Qwen3.5-2B MTP 冒烟和性能验证。

测试条件

  • Target TP2,Draft Body TP1,Draft LMHead TP2
  • graph 开启,graph double buffer 关闭,prefix cache 关闭
  • 输出 25 token
  • warmup 8 条后,20 条请求执行 3 轮,共统计 60 条
  • MTP4、MTP5 分别完整执行两次

功能验证

  • PR1948 可以正确从可见设备中自动发现并映射两个 rank
  • Draft Body TP1 配置正常生效
  • 每轮均为 60/60 请求成功,没有启动、推理或退出异常

服务端性能结果

配置 PR1929 基线 E2E / TTFT / TPOT 叠加 PR1948 两轮均值 E2E / TTFT / TPOT
MTP4 152.940 / 78.500 / 3.105 ms 154.801 / 78.983 / 3.156 ms
MTP5 154.497 / 80.017 / 3.098 ms 156.798 / 79.608 / 3.216 ms

叠加后的数值略慢,但两次重复间存在明显波动,尤其 MTP5 两次 E2E 分别为 158.498 ms 和 155.098 ms。PR1948 本身没有修改 draft、verify 或 decode 热路径,因此估计这里是测试波动。

仅从 PR1929 相关功能兼容性看,应该是没问题的。

…s/--device_id flags

Remove the --devices, --device_id, and --draft_devices CLI flags along with
their c_api/cc_api/pybind surfaces. Device selection now defaults to "auto":
each process enumerates its visible devices via Platform::device_count(),
honoring *_VISIBLE_DEVICES masks. Add a unified Platform::current_device()
(NPU/CUDA/MLU/DCU/MUSA).

Rework DistManager::setup_multi_node_workers for multi-process serving: the
parsed devices vector is now the visible-card pool rather than "cards this
process uses". Each process runs exactly one worker bound to
devices[node_rank], so world_size == nnodes instead of devices.size()*nnodes.
This keeps every process able to see all cards (so ATB can build the
cross-card HCCL comm group) while deriving worker cardinality from
nnodes/node_rank. Verified: 2-process TP=2 Qwen3-8B serves correctly,
gsm8k mean_acc=0.9234.

Update launch docs/examples to select devices via *_VISIBLE_DEVICES instead
of --devices.

Co-Authored-By: Claude <noreply@anthropic.com>
@XuZhang99
XuZhang99 force-pushed the feat/auto-device-discovery branch from 79da539 to e844363 Compare July 17, 2026 02:37
XuZhang99 and others added 4 commits July 17, 2026 12:29
The CP-config validation in Master computed global_world_size as
nnodes * devices.size(), which double-counts now that the devices vector is
the visible-card pool rather than the per-process worker list. Multi-process
serving runs one worker per process, so the global world size is the node
count. Aligns this path with DistManager::setup_multi_node_workers.

Co-Authored-By: Claude <noreply@anthropic.com>
…re used

Multi-process serving requires every process to see all cards (so ATB can
build the cross-card HCCL comm group); each process then binds the card
selected by its --node_rank. Masking each process to a single card
(<VAR>_VISIBLE_DEVICES=$DEVICE) makes the visible-device pool size 1 and trips
the node_rank bounds check for rank>=1. Revert the launch snippets to invoke
the binary without a per-process mask (no mask means all cards are visible),
distinguishing processes only by --node_rank. Preserve the numactl CPU-affinity
$DEVICE usage and fix stray backslash-continuation lines.

Co-Authored-By: Claude <noreply@anthropic.com>
The earlier flag-removal pass dropped the global export <VAR>_VISIBLE_DEVICES
line and left dead START_DEVICE/DEVICE shell vars. Restore the global export
(the intended way to pick cards now that --devices is gone; a process sees the
masked cards and binds by --node_rank) and remove the dead vars. Keep DEVICE in
glm_5/kimi2_5 where numactl still uses it for CPU affinity. Multi-card cookbook
examples that never had a global export stay bare (no mask means all cards
visible).

Co-Authored-By: Claude <noreply@anthropic.com>
Select the card this process owns from the visible pool by node_rank, so
options.devices() holds exactly one device (the card this process uses) instead
of the whole visible pool. This mirrors the historical single-element devices
semantics (previously set via --devices="npu:$DEVICE"): DeviceNameUtils::
to_string(devices) prints "npu:N" rather than "npu:0,npu:1", and DeviceMonitor
registers only this process's card. dist_manager reverts to deriving
each_node_ranks/world_size from devices.size()==1, so world_size==nnodes.

Verified: 2-process TP=2 Qwen3-8B, node_0 devices=npu:0, node_1 devices=npu:1,
world_size=2, inference correct.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants