[Bugfix][SM70] Read the quantization SM70 gate from the worker's own device

[Peuqui]

Purpose

is_exact_sm70_cuda_platform() decides whether a worker takes the native SM70
routes. It is read by nine call sites: the TurboMind NVFP4 and MXFP4 MoE
selectors, the compressed-tensors W8A16-FP8 scheme, mxfp4.py, and four gates
in modelopt.py.

It asked device 0:

return current_platform.is_cuda() and current_platform.is_device_capability((7, 0))

is_device_capability defaults to device_id=0, i.e. the first entry of the
visibility list — not the device this worker builds its layers on. On a node
that mixes card generations, every rank therefore gets the same answer, and it
is the wrong one for all but one card generation.

Measured on 2x Quadro RTX 8000 (sm75) + 3x Tesla V100 (sm70),
CUDA_DEVICE_ORDER=PCI_BUS_ID, calling the real function once per device after
torch.cuda.set_device(i):

CUDA_VISIBLE_DEVICES=0,2,1,3 (Turing first — our serving layout):

device	card	capability	gate says
0	Quadro RTX 8000	(7, 5)	False	ok
1	Quadro RTX 8000	(7, 5)	False	ok
2	Tesla V100	(7, 0)	False	wrong
3	Tesla V100	(7, 0)	False	wrong

CUDA_VISIBLE_DEVICES=1,3,0,2 (Volta first):

device	card	capability	gate says
0	Tesla V100	(7, 0)	True	ok
1	Tesla V100	(7, 0)	True	ok
2	Quadro RTX 8000	(7, 5)	True	wrong
3	Quadro RTX 8000	(7, 5)	True	wrong

Both directions are harmful and both are silent. With a Turing card first the
Volta workers lose the SM70 kernels they are the only ones that can run. With a
Volta card first the Turing workers claim them.

The fix reads the capability of the device the worker is on:

if not current_platform.is_cuda():
    return False
return current_platform.is_device_capability(
    (7, 0), device_id=torch.accelerator.current_device_index()
)

torch.accelerator.current_device_index() is the idiom already used across the
tree (distributed/weight_transfer/, eplb_state.py, logits_processor.py).
This is the same bug class as #514, which fixed the SM70 config defaults; the
quantization gate was not part of that change.

Homogeneous nodes are unaffected: there every device has the same capability,
so device 0 and the worker's device always agreed.

Results

Same probe, same hardware, with the fix — all eight rows correct:

CUDA_VISIBLE_DEVICES	device 0	1	2	3
0,2,1,3	sm75 False	sm75 False	sm70 True	sm70 True
1,3,0,2	sm70 True	sm70 True	sm75 False	sm75 False

Test Plan

Environment: checkout at origin/main 4f19ef7 with the compiled extensions of a
1Cat-vLLM 1.5.0 wheel linked in (see Limitations).

python -m pytest tests/quantization/test_sm70_platform_gate_per_device.py -q

# affected directory, unit-test subset: the 35 files that neither boot an
# engine nor download a checkpoint (see Limitations for what is left out),
# run with and without the change to get a clean delta
#   (the subset is tests/quantization/test_*.py minus the files that
#    reference LLM(/vllm_runner/VllmRunner, minus test_quark.py and
#    test_mixed_precision.py)
HF_HUB_OFFLINE=1 python -m pytest $SUBSET -q

# every other test in the tree that stubs is_device_capability with a
# narrow lambda, again with and without the change
HF_HUB_OFFLINE=1 python -m pytest \
    tests/kernels/test_sm70_deepseek_v4_fp16_gemv.py \
    tests/v1/spec_decode/test_dflash_mrv2_config.py \
    tests/v1/cudagraph/test_sm70_mtp_split_cudagraphs.py \
    tests/v1/attention/test_gdn_metadata_builder.py \
    tests/v1/sample/test_topk_topp_sampler.py \
    tests/models/qwen4_exp/test_qsa_ops.py \
    tests/v1/worker/test_gpu_model_runner_v2_greedy.py \
    tests/v1/worker/test_qwen4_exp_v2.py -q

# counter-check: sm70_turbomind.py reverted to origin/main, tests kept
python -m pytest tests/quantization/test_sm70_platform_gate_per_device.py -q

pre-commit run --files vllm/model_executor/layers/quantization/sm70_turbomind.py \
    tests/quantization/test_sm70_platform_gate_per_device.py \
    tests/quantization/test_sm70_mxfp4_moe.py
pre-commit run mypy-3.10 --hook-stage manual --files $SAME_THREE_FILES

Test Result

New file: 5 passed.

Quantization unit subset, with this change: 643 passed, 29 failed,
8 skipped. The same 35 files on unmodified origin/main with the new tests
kept: 640 passed, 32 failed, 8 skipped. Diffing the two failure lists:

• newly failing: none
• newly passing: the three new tests that assert the per-device behaviour

The 29 failures are identical on both trees and environmental: 23 are
ValueError("type fp8e4nv not supported in this architecture") — Triton has no
FP8 on Volta/Turing — and six are CPU/CUDA device mismatches in test fixtures
(test_per_token_kv_cache.py, test_sm70_modelopt_nvfp4_kernel_oracle.py,
test_sm70_modelopt_nvfp4.py, test_sm70_mxfp4_moe.py).

Widening the call signature does break exactly one existing stub, which this PR
therefore updates. test_mxfp4_sm70_platform_gate_is_exact replaced
current_platform.is_device_capability with a lambda taking only capability,
so passing device_id raised TypeError. The two lambdas now take
device_id=0, matching what tests/config/test_sm70_gates_any_visible_device.py
already does since #514. The test's intent is unchanged, and that file goes from
31 to 32 passed.

Every other test in the tree that stubs is_device_capability with a narrow
lambda: 217 passed, 44 failed, 1 skipped both with and without the change,
with an empty diff between the two failure lists. None of those failures mention
device_id; they are CUDA/kernel failures on this bench.

Counter-check with vllm/model_executor/layers/quantization/sm70_turbomind.py
reverted to origin/main, the new tests kept: 3 failed, 2 passed —
test_gate_is_true_on_the_volta_worker,
test_gate_does_not_answer_for_device_zero and
test_moe_route_selection_follows_the_gate. The two that still pass are the
Turing case (which device 0 happens to answer correctly in that fixture) and
the non-CUDA case.

pre-commit over both files: ruff check, ruff format, typos, mypy-local, SPDX
headers, root lazy imports, forbidden imports, the "Prevent new 'torch.cuda'
APIs call" check, config-docstring check, attention-backend docs and the
boolean-ops check all Passed. pre-commit run mypy-3.10 --hook-stage manual:
Passed.

Not a duplicate

Checked on 2026-09-08 against every open PR. One open PR touches
vllm/model_executor/layers/quantization/sm70_turbomind.py: #561
("[Kernel] Reduce DFlash2 weight and scale memory on SM70"). Its only hunk in
that file is at line 447, in apply_prepared_linear; this change is at line 68.
is_exact_sm70_cuda_platform and is_device_capability do not occur in #561.
No open issue mentions the quantization capability gate.

Limitations

The device ordering matters for this fix, and it is worth stating explicitly:
is_device_capability resolves its device_id through the visibility list to
an NVML index, which is PCI ordered, while torch's device index follows
CUDA_DEVICE_ORDER. The two agree under CUDA_DEVICE_ORDER=PCI_BUS_ID, which
vLLM already warns to set when it detects different devices in the system, and
which this bench sets. Without it the two orderings can disagree on a
heterogeneous box — but that is a pre-existing property of is_device_capability
throughout the tree, not something this change introduces. I measured both with
and without it and kept the tree's own idiom rather than reaching past the
platform API.

The directory run above is the unit-test subset. Sixteen files in
tests/quantization/ boot a full engine or pull a checkpoint — test_cpu_offload.py
alone downloads 7.8 GB — and two more (test_quark.py, test_mixed_precision.py)
need lm_eval, which is not installed here. Those are excluded, and the runs are
pinned with HF_HUB_OFFLINE=1 so nothing downloads silently. None of them touch
sm70_turbomind.py.

The measurements above call the real function on real hardware but do not boot a
model; the routes it gates are exercised by the unit tests. I have not built
current main from source on this hardware — that is a multi-hour CUDA build.

AI assistance: this change was developed with Claude (Anthropic) as a coding
assistant. Every changed line was reviewed by me and the test runs above were
executed on my hardware; I can defend the change end to end.

🤖 Generated with Claude Code