Repository navigation
Multi-vendor CN GPU adaptation: device pipeline, warmup option, real-model validation tooling - #40
Merged
Merged
Conversation
Wire pymss up to the domestic accelerator families on top of the new `pymss_core.device` capability profiles. - new `pymss.backends`: the only place that knows the `--device` vocabulary (`SUPPORTED_DEVICES`) and the recurrent-fallback installer - `separator`: resolve vendor aliases (`biren`/`supa`, `iluvatar`, `metax`, `enflame`/`gcu`, `mlu`, `hygon`), extend the `auto` probe chain to CUDA -> NPU -> GCU -> MLU -> SUPA -> MPS -> CPU, dispatch private device types through a generic `_select_private_device`, and gate DataParallel on the profile instead of `device.type == "cuda"` - `utils`: autocast goes through `amp_supported()` and falls back to a null context on torch < 2.4 or when a vendor backend refuses the device type; the two hardcoded device-type tuples now use capability queries - `cli`: one definition of `--device` choices for all subcommands - `vr_separator` / uvr `spec_utils`: AMP, channels_last and STFT selection follow the profile - lower the torch floor from 2.7.1 to 2.4 and require pymss-core >= 0.1.15 - docs: `docs/cn-gpu.md` compatibility matrix, install steps, debug env vars and the on-card verification checklist; README/README_CN sections - tests: `test/test_backends.py` covers the alias table, CLI wiring, CPU-only fallbacks and vendor resolution
…ation
OpenI CloudBrain training jobs cannot be driven interactively, so ship a
startup script that collects the evidence a vendor bring-up needs:
- environment / torch build / visible devices, including torch.{npu,gcu,mlu,supa}
- installed vendor stack (adapted torch + SDK bindings)
- resolved pymss device profile and detection inputs
- STFT/ISTFT round trip through both the native and the fallback path
- recurrent forward through both the native and the CPU fallback path
- pytest summary of the unit suite
- optional --install-tests for a first run on a bare image
Every section is isolated, so a missing operator is reported instead of
killing the job. The report is written into c2net_context.output_path and
uploaded, which makes it downloadable from the task page.
…iner A training task only receives the code repository of the project it was created from, so pymss_core was not importable. Install it from the platform git (falling back to GitHub) and editable-install the checked out pymss before probing; --no-install skips both.
…tory .gitignore ignores scripts/ (line 181), so the probe never got committed and CloudBrain rejected it as a missing startup file. tools/ is tracked.
…file Verified on a real BIREN106M with torch 2.7.0+cpu / torch-br 1.7.0.20700 (SUPA 1.9.0.0.rc1): - torch.stft(return_complex=True) raises 'does not support return_complex=True now'; the rfft based composition reconstructs to 9.5e-07 -> complex_fft=False - torch.hann_window(..., device='supa') dies with a SUPA kernel launch error; windows must be built on CPU (hann_window_npu already does) - nn.LSTM runs natively, nn.GRU raises for the missing aten::_thnn_fused_gru_cell -> DeviceProfile.rnn_blocked + rnn_op_blocked() let the fallback be decided per cell instead of disabling recurrent kernels for the whole device - bf16 matmul and autocast work, 31.75 GiB visible on a single card
…at16 autocast_helper() requested dtype=torch.float16 for every device. BIREN br10x has no float16 matmul at all (torch.autocast raises 'Current SUPA Device does not support float16'), so the RuntimeError guard silently disabled AMP there. Ask the profile through autocast_dtype() and get bfloat16 on BIREN.
pyproject declares requires-python >= 3.10 but plugins/manifest.py imported tomllib unconditionally, which only exists from 3.11. The BIREN SUPA image ships Python 3.10, so 'import pymss' died before any model could load. Fall back to tomli and declare it for python_version < 3.11.
_resolve_public_device calls requires_plugin() on the --device biren/supa/gcu/ mlu/... path but separator only imported device_profile, install_recurrent_ fallback and prepare_device_alias, so the first real BIREN run died with NameError. Covered by tests for every alias plus the missing-plugin message.
…rent one separator now asks install_conv_fallback() at init, so a BIREN run stops dying with a SUDNN abort inside the first dilated resblock.
Debug tasks can only be created through the web UI, but their JupyterLab has a standard REST + websocket API. jlab_exec drives it (xsrf cookie, TLS retries, kernel reuse, --new-kernel for a blocked kernel) so a container can be operated without a screen. cn_gpu_bench measures the shapes pymss models use and encodes the two lessons from the BIREN session: refuse to run while another inference holds the card (two 17 GiB processes on 31.75 GiB produced a bogus 263 s/batch), and never build a full-resolution CPU control group (im2col OOM-killed the notebook server).
Same vendored-kernel gap as the core fix: mask * complex_spec is unsupported on SUPA, so the mask is applied on the real/imaginary planes.
Follows the core change: complex device->host copies fail on BIREN.
BIREN has no complex host->device copy.
BIREN rejects float16 autocast at runtime ('Please switch dtype to
bfloat16'); the device profile already carries bfloat16 for SUPA.
Per-shape SUDNN search (220 s first call vs 10-15 ms steady) and the knob matrix that identified eager conv kernels as the only effective switch, the operator gaps that had to be worked around (complex mul/copies, replication_pad2d, UpsampleBilinear bf16), the end-to-end result (388 s -> 10 s) and the rule that CPU fallbacks are only for vendor crashes and wrong results.
One device run is not evidence: BIREN has returned silent zeros, so the script reports rel (vs CPU), repeat (device run 1 vs run 2, which exposes nondeterminism) and the non-zero element count.
vendor_op_probe.py sweeps the operators each model family needs (subprocess per probe, CPU comparison so silent miscomputes show up), and vendor_model_probe.py runs every family from the parity harness with an optional per-op trace that names the operator that actually failed.
Published BR106M bf16 figures disagree (120 vs 170 TFLOPS), so the benchmark measures the achievable dense matmul peak on the card itself, counts each model's FLOPs with torch.utils.flop_counter and reports MFU against the measured peak -- the models run fp32 here, so the fp32 peak is the honest denominator.
The per-model benchmark can now wrap each forward in vendor bf16 autocast, and vendor_bench_e2e.py times a real separation with PYMSS_AMP=0/1 so the AMP path is judged on audio-seconds per wall-second instead of intuition.
The operator table now lists what actually breaks (complex reshape/cat/abs, grouped conv1d, GroupNorm, 5-D index_add_, F.pad, bf16 interpolation, and the FFT sizes that silently return zeros or autotune for ten minutes), and bf16 autocast is documented as enabled -- the earlier note said it was off, which was wrong. The family table adds the verification criteria that catch silent failures: relative error against CPU, run-to-run repeatability, and a non-zero count.
vendor_bench_all.py runs the peak measurement, the per-family table in fp32 and bf16, and the end-to-end separation serially -- two processes on this card make every number meaningless -- and writes a markdown report.
The diagnostic script that actually found scnet's 605 s LSTM and bandit's mis-broadcast lived in /tmp and was lost when the container restarted after its four-hour limit, so it is now a repository tool: rolling list of the last operators, slow-call reporting, a wall-clock cap instead of waiting out a ten-minute planner stall, the full vendor message, and --repeat to catch failures that only appear on the second run (scnet passes once and fails on the repeat).
Times stft, the model forward and istft separately on realistic audio lengths so an optimisation can be attributed to a stage rather than to 'the model', and reports audio-seconds per wall-second.
Two failures were unreadable: scnet passes twice under the tracer yet fails under the verifier (which also runs a CPU reference first), and legacy_v3_demucs never returns from a single call, so a wall-clock cap checked between operators cannot fire. The verifier can now print the child's stderr tail, and the tracer arms faulthandler so a stuck call dumps the Python stack it is stuck in.
scnet fails inside torch.supa.synchronize() after the forward, so the failing operator is asynchronous -- the per-operator tracer hides it precisely because it syncs after every call. --modules syncs after every submodule instead, which names the module and keeps the queue shallow. A child killed by the outer timeout now keeps whatever it printed, and the faulthandler stack is only a fallback.
A CPU forward in the same process makes the first device forward of scnet and legacy_v3_demucs fail with 'ErrorCode: 11, Sudnn Error' -- and the identical retry succeeds every time, with the right numbers. That is vendor first-call state, not a property of the model, so the reference now runs in a separate process writing a .pt file, which also matches how the models are used in practice.
Splitting the processes made the device child regenerate its input with the device RNG, which does not reproduce the CPU stream, so every family compared two different signals and reported rel ~ 1.0. The saved file now carries both.
…eference scnet is 5.8% off overall while its STFT is exact to 7e-07, so the drift is inside the network and a final-output comparison cannot name it. Two processes dump per-module statistics (sum/absmax/absmean/std, byte-bounded) and the tool reports the modules whose statistics leave tolerance, largest first.
… kernel scaffold - vendor_rope_* probes: shape sweeps (dim0<=1024, context poisoning on failure), seq-accuracy (halves convention, 0.167% flat), model A/B v1-v5, integration check, infer2/supa_rope/fwd rejections with evidence. - vendor_phase_profile: sync-boundary attribution, freq/time split; freq attention bucket dissection. vendor_roofline: ceilings + utilization. - vendor_attention_chain_probe: regroup bit-identity A/B. - vendor_rope_freq_cost: chunked-op cost breakdown (7x1.77ms short-seq). - vendor_rope_srtc_kernel: first end-to-end SRTC compile->load->launch from Python (kernel executes and writes); open: fp32 table pointer semantics (reads 0), NUMA-16 layout addressing, vectorization.
…a errors) + FFN docs
…acks compile on first use)
These were session diagnostics during SUPA optimization; conclusions live in docs/cn-gpu.md and BR_OPTIMIZATION_STATE.md. Durable tools kept: jlab_exec, vendor_realmodel_check, vendor_full_profile, vendor_model_verify, vendor_perf, vendor_mfu, vendor_op_trace, vendor_layer_diff, and friends.
…ice resolver, and clean manifest imports
Resolve conflicts by keeping the profile-driven device layer while preserving main's XPU work (PR pymss-project#35): - pymss/cli.py: --device choices come from SUPPORTED_DEVICES for every subcommand, so xpu and the vendor aliases cannot drift apart again. - pymss/separator.py: keep both _VENDOR_ALIASES and _validate_device_ids; auto order is cuda -> xpu -> npu -> gcu/mlu/supa -> mps -> cpu; explicit --device xpu keeps the "XPU-enabled PyTorch build" guard. - pymss/utils.py: _autocast stays profile-driven. FP32_INFERENCE_DEVICE_TYPES is dropped because DeviceProfile.amp is now the only source of truth (XPU reports amp=False). _finalize_overlap and the chunked fast path use device_profile().is_accelerator, which covers xpu. - pymss/modules/vocal_remover/vr_separator.py: one AMP policy via pymss.utils._autocast instead of a duplicate autocast call site. - spec_utils.py: keep the profile-based accelerator test. - pymss/plugins/manifest.py: keep math plus the tomllib/tomli fallback. - pyproject.toml: 2.4.0, pymss-core>=0.1.15, packaging>=24.0 from main. Note: test_graph_auto_node_uses_the_first_id_with_an_explicit_gpu_runtime fails identically on clean origin/main (pymss.graph.nodes imports register_node("input_audio") outside the builtin bootstrap), so it is a pre-existing main-branch defect, not a merge regression.
- add `musa`/`moore` to the vendor aliases and to the ValueError message, so explicit `--device musa` resolves instead of raising - include musa in the auto-probe order (gcu, mlu, supa, musa) so `--device auto` selects torch_musa when present - add test/test_backends_musa.py, pinning the DeviceProfile(MOORE) flags that were measured on a real MTT S70 (torch 2.5.0 + torch_musa 2.5.0, MUSA SDK 4.2.0)
The htdemucs checkpoints store half-precision weights. With AMP disabled the convolutions then receive fp32 activations and abort with `Input type (float) and bias type (c10::Half) should be the same`. Cast low-precision parameters/buffers to float32 whenever inference runs in float32, not only on CPU. Verified on MTT S70: HTDemucs4 6-stem, rel 2.7e-05 against the CPU reference.
feat(separator): 支持摩尔线程 MUSA(MTT S70 实测 11 个模型族全通)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
pymss 的多国产 GPU 适配:设备解析管线扩展、首调延迟优化(warmup)、真实模型验证工具链与文档。与 pymss-core 的
cn-gpu-pr分支配套。变更内容
separator.py):vendor 私有设备走_select_private_device(计数探针回退到插件模块);rocm/npu别名;cuda/cpu/mps 原路径语义不变(仅校验加固)inference.warmup选项:加载时 1 秒静音空跑,把国产栈按形状 JIT 编译税前置到启动期backends.py新增 + VR 分离器设备自省修复tools/):vendor_realmodel_check.py(--device五卡别名、per-model 子进程隔离、11 类型真实模型矩阵)、jlab_exec.py(云脑远程执行)等 11 个耐用工具docs/cn-gpu.md五卡实测矩阵、选型建议、JIT 首调税方法论实测:11 种模型类型 × 真实权重(10s 音频,秒)
三卡全部 11/11 通过。MR-V100 零修复一次通过;N260 transformer 类全场最快;冷热差异来自国产栈按形状 JIT 编译(首次调用含编译税),
warmup=True可前置消化(scnet 首调 23.9s→3.2s,−87%)。测试
test_backends.py新增;真实模型矩阵 JSON 三份存档(Z100L / N260 / MR-V100)