Skip to content

Multi-vendor CN GPU adaptation: device pipeline, warmup option, real-model validation tooling - #40

Merged
baicai-1145 merged 66 commits into
pymss-project:mainfrom
baicai-1145:cn-gpu-pr
Nov 9, 2026
Merged

baicai-1145 merged 66 commits into
pymss-project:mainfrom
baicai-1145:cn-gpu-pr

Conversation

@baicai-1145

Copy link
Copy Markdown
Member

概述

pymss 的多国产 GPU 适配:设备解析管线扩展、首调延迟优化(warmup)、真实模型验证工具链与文档。与 pymss-core 的 cn-gpu-pr 分支配套。

变更内容

  • 设备解析(separator.py):vendor 私有设备走 _select_private_device(计数探针回退到插件模块);rocm / npu 别名;cuda/cpu/mps 原路径语义不变(仅校验加固)
  • inference.warmup 选项:加载时 1 秒静音空跑,把国产栈按形状 JIT 编译税前置到启动期
  • backends.py 新增 + VR 分离器设备自省修复
  • 验证工具链(tools/):vendor_realmodel_check.py(--device 五卡别名、per-model 子进程隔离、11 类型真实模型矩阵)、jlab_exec.py(云脑远程执行)等 11 个耐用工具
  • 文档:docs/cn-gpu.md 五卡实测矩阵、选型建议、JIT 首调税方法论

实测:11 种模型类型 × 真实权重(10s 音频,秒)

模型类型 海光 Z100L 沐曦 N260 天数 MR-V100
mel_band_roformer 2.26 1.07 1.62
bs_roformer 6.37 1.39 3.35
bandit_v2 3.72 1.42 3.78
scnet 6.01 1.68 23.9(热态 3.2)
legacy_demucs 5.82 41.8 3.94
legacy_tasnet 3.17 7.54 3.99
htdemucs 9.60 7.22 7.55
mdx23c 4.61 7.51 8.2(热态 2.6)
bandit 11.29 9.92 13.4
apollo 11.62 26.7 16.6(热态 9.4)
vr 35.0 12.2 36.8(热态 0.43,batch16)

三卡全部 11/11 通过。MR-V100 零修复一次通过;N260 transformer 类全场最快;冷热差异来自国产栈按形状 JIT 编译(首次调用含编译税),warmup=True 可前置消化(scnet 首调 23.9s→3.2s,−87%)。

测试

  • 套件 382 passed / 222 skipped
  • test_backends.py 新增;真实模型矩阵 JSON 三份存档(Z100L / N260 / MR-V100)

Wire pymss up to the domestic accelerator families on top of the new
`pymss_core.device` capability profiles.

- new `pymss.backends`: the only place that knows the `--device` vocabulary
  (`SUPPORTED_DEVICES`) and the recurrent-fallback installer
- `separator`: resolve vendor aliases (`biren`/`supa`, `iluvatar`, `metax`,
  `enflame`/`gcu`, `mlu`, `hygon`), extend the `auto` probe chain to
  CUDA -> NPU -> GCU -> MLU -> SUPA -> MPS -> CPU, dispatch private device
  types through a generic `_select_private_device`, and gate DataParallel on
  the profile instead of `device.type == "cuda"`
- `utils`: autocast goes through `amp_supported()` and falls back to a null
  context on torch < 2.4 or when a vendor backend refuses the device type;
  the two hardcoded device-type tuples now use capability queries
- `cli`: one definition of `--device` choices for all subcommands
- `vr_separator` / uvr `spec_utils`: AMP, channels_last and STFT selection
  follow the profile
- lower the torch floor from 2.7.1 to 2.4 and require pymss-core >= 0.1.15
- docs: `docs/cn-gpu.md` compatibility matrix, install steps, debug env vars
  and the on-card verification checklist; README/README_CN sections
- tests: `test/test_backends.py` covers the alias table, CLI wiring, CPU-only
  fallbacks and vendor resolution
…ation

OpenI CloudBrain training jobs cannot be driven interactively, so ship a
startup script that collects the evidence a vendor bring-up needs:

- environment / torch build / visible devices, including torch.{npu,gcu,mlu,supa}
- installed vendor stack (adapted torch + SDK bindings)
- resolved pymss device profile and detection inputs
- STFT/ISTFT round trip through both the native and the fallback path
- recurrent forward through both the native and the CPU fallback path
- pytest summary of the unit suite
- optional --install-tests for a first run on a bare image

Every section is isolated, so a missing operator is reported instead of
killing the job. The report is written into c2net_context.output_path and
uploaded, which makes it downloadable from the task page.
…iner

A training task only receives the code repository of the project it was
created from, so pymss_core was not importable. Install it from the
platform git (falling back to GitHub) and editable-install the checked out
pymss before probing; --no-install skips both.
…tory

.gitignore ignores scripts/ (line 181), so the probe never got committed and
CloudBrain rejected it as a missing startup file. tools/ is tracked.
…file

Verified on a real BIREN106M with torch 2.7.0+cpu / torch-br 1.7.0.20700
(SUPA 1.9.0.0.rc1):

- torch.stft(return_complex=True) raises 'does not support return_complex=True
  now'; the rfft based composition reconstructs to 9.5e-07 -> complex_fft=False
- torch.hann_window(..., device='supa') dies with a SUPA kernel launch error;
  windows must be built on CPU (hann_window_npu already does)
- nn.LSTM runs natively, nn.GRU raises for the missing aten::_thnn_fused_gru_cell
  -> DeviceProfile.rnn_blocked + rnn_op_blocked() let the fallback be decided
  per cell instead of disabling recurrent kernels for the whole device
- bf16 matmul and autocast work, 31.75 GiB visible on a single card
…at16

autocast_helper() requested dtype=torch.float16 for every device.  BIREN br10x
has no float16 matmul at all (torch.autocast raises 'Current SUPA Device does
not support float16'), so the RuntimeError guard silently disabled AMP there.
Ask the profile through autocast_dtype() and get bfloat16 on BIREN.
pyproject declares requires-python >= 3.10 but plugins/manifest.py imported
tomllib unconditionally, which only exists from 3.11.  The BIREN SUPA image
ships Python 3.10, so 'import pymss' died before any model could load.  Fall
back to tomli and declare it for python_version < 3.11.
_resolve_public_device calls requires_plugin() on the --device biren/supa/gcu/
mlu/... path but separator only imported device_profile, install_recurrent_
fallback and prepare_device_alias, so the first real BIREN run died with
NameError.  Covered by tests for every alias plus the missing-plugin message.
…rent one

separator now asks install_conv_fallback() at init, so a BIREN run stops dying
with a SUDNN abort inside the first dilated resblock.
Debug tasks can only be created through the web UI, but their JupyterLab has a
standard REST + websocket API.  jlab_exec drives it (xsrf cookie, TLS retries,
kernel reuse, --new-kernel for a blocked kernel) so a container can be operated
without a screen.

cn_gpu_bench measures the shapes pymss models use and encodes the two lessons
from the BIREN session: refuse to run while another inference holds the card
(two 17 GiB processes on 31.75 GiB produced a bogus 263 s/batch), and never build
a full-resolution CPU control group (im2col OOM-killed the notebook server).
Same vendored-kernel gap as the core fix: mask * complex_spec is unsupported on
SUPA, so the mask is applied on the real/imaginary planes.
Follows the core change: complex device->host copies fail on BIREN.
BIREN has no complex host->device copy.
BIREN rejects float16 autocast at runtime ('Please switch dtype to
bfloat16'); the device profile already carries bfloat16 for SUPA.
Per-shape SUDNN search (220 s first call vs 10-15 ms steady) and the knob matrix
that identified eager conv kernels as the only effective switch, the operator
gaps that had to be worked around (complex mul/copies, replication_pad2d,
UpsampleBilinear bf16), the end-to-end result (388 s -> 10 s) and the rule that
CPU fallbacks are only for vendor crashes and wrong results.
One device run is not evidence: BIREN has returned silent zeros, so the script
reports rel (vs CPU), repeat (device run 1 vs run 2, which exposes
nondeterminism) and the non-zero element count.
vendor_op_probe.py sweeps the operators each model family needs (subprocess per
probe, CPU comparison so silent miscomputes show up), and vendor_model_probe.py
runs every family from the parity harness with an optional per-op trace that
names the operator that actually failed.
Published BR106M bf16 figures disagree (120 vs 170 TFLOPS), so the benchmark
measures the achievable dense matmul peak on the card itself, counts each
model's FLOPs with torch.utils.flop_counter and reports MFU against the measured
peak -- the models run fp32 here, so the fp32 peak is the honest denominator.
The per-model benchmark can now wrap each forward in vendor bf16 autocast, and
vendor_bench_e2e.py times a real separation with PYMSS_AMP=0/1 so the AMP path
is judged on audio-seconds per wall-second instead of intuition.
The operator table now lists what actually breaks (complex reshape/cat/abs,
grouped conv1d, GroupNorm, 5-D index_add_, F.pad, bf16 interpolation, and the FFT
sizes that silently return zeros or autotune for ten minutes), and bf16 autocast
is documented as enabled -- the earlier note said it was off, which was wrong.

The family table adds the verification criteria that catch silent failures:
relative error against CPU, run-to-run repeatability, and a non-zero count.
vendor_bench_all.py runs the peak measurement, the per-family table in fp32 and
bf16, and the end-to-end separation serially -- two processes on this card make
every number meaningless -- and writes a markdown report.
The diagnostic script that actually found scnet's 605 s LSTM and bandit's
mis-broadcast lived in /tmp and was lost when the container restarted after its
four-hour limit, so it is now a repository tool: rolling list of the last
operators, slow-call reporting, a wall-clock cap instead of waiting out a
ten-minute planner stall, the full vendor message, and --repeat to catch failures
that only appear on the second run (scnet passes once and fails on the repeat).
Times stft, the model forward and istft separately on realistic audio lengths so
an optimisation can be attributed to a stage rather than to 'the model', and
reports audio-seconds per wall-second.
Two failures were unreadable: scnet passes twice under the tracer yet fails under
the verifier (which also runs a CPU reference first), and legacy_v3_demucs never
returns from a single call, so a wall-clock cap checked between operators cannot
fire.  The verifier can now print the child's stderr tail, and the tracer arms
faulthandler so a stuck call dumps the Python stack it is stuck in.
scnet fails inside torch.supa.synchronize() after the forward, so the failing
operator is asynchronous -- the per-operator tracer hides it precisely because it
syncs after every call.  --modules syncs after every submodule instead, which
names the module and keeps the queue shallow.  A child killed by the outer timeout
now keeps whatever it printed, and the faulthandler stack is only a fallback.
A CPU forward in the same process makes the first device forward of scnet and
legacy_v3_demucs fail with 'ErrorCode: 11, Sudnn Error' -- and the identical retry
succeeds every time, with the right numbers. That is vendor first-call state, not
a property of the model, so the reference now runs in a separate process writing a
.pt file, which also matches how the models are used in practice.
Splitting the processes made the device child regenerate its input with the device
RNG, which does not reproduce the CPU stream, so every family compared two
different signals and reported rel ~ 1.0.  The saved file now carries both.
…eference

scnet is 5.8% off overall while its STFT is exact to 7e-07, so the drift is inside
the network and a final-output comparison cannot name it.  Two processes dump
per-module statistics (sum/absmax/absmean/std, byte-bounded) and the tool reports
the modules whose statistics leave tolerance, largest first.
baicai-1145 and others added 25 commits October 29, 2026 21:13
… kernel scaffold

- vendor_rope_* probes: shape sweeps (dim0<=1024, context poisoning on
  failure), seq-accuracy (halves convention, 0.167% flat), model A/B v1-v5,
  integration check, infer2/supa_rope/fwd rejections with evidence.
- vendor_phase_profile: sync-boundary attribution, freq/time split; freq
  attention bucket dissection. vendor_roofline: ceilings + utilization.
- vendor_attention_chain_probe: regroup bit-identity A/B.
- vendor_rope_freq_cost: chunked-op cost breakdown (7x1.77ms short-seq).
- vendor_rope_srtc_kernel: first end-to-end SRTC compile->load->launch
  from Python (kernel executes and writes); open: fp32 table pointer
  semantics (reads 0), NUMA-16 layout addressing, vectorization.
These were session diagnostics during SUPA optimization; conclusions live
in docs/cn-gpu.md and BR_OPTIMIZATION_STATE.md. Durable tools kept:
jlab_exec, vendor_realmodel_check, vendor_full_profile, vendor_model_verify,
vendor_perf, vendor_mfu, vendor_op_trace, vendor_layer_diff, and friends.
Resolve conflicts by keeping the profile-driven device layer while
preserving main's XPU work (PR pymss-project#35):

- pymss/cli.py: --device choices come from SUPPORTED_DEVICES for every
  subcommand, so xpu and the vendor aliases cannot drift apart again.
- pymss/separator.py: keep both _VENDOR_ALIASES and _validate_device_ids;
  auto order is cuda -> xpu -> npu -> gcu/mlu/supa -> mps -> cpu; explicit
  --device xpu keeps the "XPU-enabled PyTorch build" guard.
- pymss/utils.py: _autocast stays profile-driven. FP32_INFERENCE_DEVICE_TYPES
  is dropped because DeviceProfile.amp is now the only source of truth
  (XPU reports amp=False). _finalize_overlap and the chunked fast path use
  device_profile().is_accelerator, which covers xpu.
- pymss/modules/vocal_remover/vr_separator.py: one AMP policy via
  pymss.utils._autocast instead of a duplicate autocast call site.
- spec_utils.py: keep the profile-based accelerator test.
- pymss/plugins/manifest.py: keep math plus the tomllib/tomli fallback.
- pyproject.toml: 2.4.0, pymss-core>=0.1.15, packaging>=24.0 from main.

Note: test_graph_auto_node_uses_the_first_id_with_an_explicit_gpu_runtime
fails identically on clean origin/main (pymss.graph.nodes imports
register_node("input_audio") outside the builtin bootstrap), so it is a
pre-existing main-branch defect, not a merge regression.
- add `musa`/`moore` to the vendor aliases and to the ValueError message, so
  explicit `--device musa` resolves instead of raising
- include musa in the auto-probe order (gcu, mlu, supa, musa) so `--device auto`
  selects torch_musa when present
- add test/test_backends_musa.py, pinning the DeviceProfile(MOORE) flags that were
  measured on a real MTT S70 (torch 2.5.0 + torch_musa 2.5.0, MUSA SDK 4.2.0)
The htdemucs checkpoints store half-precision weights. With AMP disabled the
convolutions then receive fp32 activations and abort with
`Input type (float) and bias type (c10::Half) should be the same`.

Cast low-precision parameters/buffers to float32 whenever inference runs in
float32, not only on CPU. Verified on MTT S70: HTDemucs4 6-stem, rel 2.7e-05
against the CPU reference.
baicai-1145 and others added 2 commits November 9, 2026 03:21
feat(separator): 支持摩尔线程 MUSA(MTT S70 实测 11 个模型族全通)
@baicai-1145
baicai-1145 merged commit 7d599ea into pymss-project:main Nov 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants