Skip to content

feat(device): support --device tpu through torch_tpu - #43

Open
baicai-1145 wants to merge 1 commit into
pymss-project:mainfrom
baicai-1145:tpu
Open

baicai-1145 wants to merge 1 commit into
pymss-project:mainfrom
baicai-1145:tpu

Conversation

@baicai-1145

Copy link
Copy Markdown
Member

What

--device tpu support for Google TPU via torch_tpu (official PyTorch TPU backend, not torch_xla). Needs Python 3.13+ and torch 2.11–2.15.

Companion PR: pymss-project/pymss-core#14 — the device profile, the numerics fixes and the torch.compile unblocking all live there.

Changes

  • separator.py: tpu added to the vendor aliases and the public-device whitelist.
  • _accelerator_available(): availability goes through torch.accelerator, because torch.tpu.is_available / device_count are rejected as "internal to TorchTPU" (torch.tpu.set_device likewise). _select_private_device uses torch.accelerator.device_count for the same reason.
  • TPU is probed last in the auto path — importing torch_tpu starts the libtpu runtime, so every cheaper backend gets to answer first.
  • docs/tpu.md: the full measurement set (operator parity, istft/fold numerics, group_norm contiguity, matmul precision, per-family throughput, torch.compile numbers).
  • README section pointing at it.

Highlights from docs/tpu.md

TPU has no operator gaps — complex STFT, RNN, grouped/dilated conv, fold/unfold and SDPA are all native. The adaptation is about numerics:

issue measured
torch.istft 1.5e-03 rel and 32x slower → use istft_npu (3.8e-05, 205 ms)
F.fold/col2im bf16 accumulate 2.6e-03 over a whole track, 99 ms → exact scatter_add_ shim, 4.2 ms (24x faster)
F.group_norm non-contiguous raises view(): — SCNet and htdemucs both abort → .contiguous()
fp32 matmul/conv is bf16 2.4e-03 → matmul_precision="high" gives 1.4e-05 for free

After the fixes: legacy_hdemucs end-to-end deviation 1.60e-05, and repeated identical input gives 0.0.

Performance

Single forward, 5 s stereo audio, TPU v5e1:

family params fp32 bf16
legacy_demucs 136.9M 48 ms (104x rt) 43 ms
legacy_hdemucs 75.3M 72 ms (69x rt) 82 ms
legacy_v3_demucs 174.4M 88 ms (57x rt) 85 ms
apollo / scnet / mdx23c 4.5–17.8M 84–129 ms 58–111 ms
bandit / bandit_v2 22.9M 176 / 191 ms 211 / 206 ms
bs_roformer_hyperace 48.0M 251 ms (20x rt) 153 ms
bs_conformer 97.5M 759 ms (6.6x rt) 387 ms
bs_roformer 136.0M 700 ms (7.1x rt) 365 ms
mel_band_conformer 77.8M 943 ms (5.3x rt) 528 ms
mel_band_roformer 103.8M 1110 ms (4.5x rt) 621 ms

Two things worth knowing before tuning use_amp:

  • bf16 helps only the attention families (1.8–2.0x). On conv-heavy models it is 7–26% slower, because TPU's autocast inserts a cast per accelerated op and a cast costs ~200 µs while a small conv saves <20 µs.
  • Batching helps only the dispatch-bound families: scnet 2.56x and bandit 1.78x at bs=8, but bs_roformer gains just 7%.

Utilisation is 3.8–14.5% of the 50.2 TFLOPS fp32 peak and dispatch is only 20–46% of runtime, so the wall is per-op granularity (45.5 µs per dispatched op vs ~4 µs on CUDA), not FLOPs. A 4-minute song: bs_roformer 34 s fp32 / 18 s bf16.

torch_tpu is the official PyTorch TPU backend (not torch_xla) and needs
Python 3.13+ with torch 2.11-2.15. It renames PrivateUse1 to "tpu", and both
torch.tpu.is_available/device_count and torch.tpu.set_device are rejected as
"internal to TorchTPU", so availability and index selection go through
torch.accelerator instead.

TPU is probed last in the auto path: importing torch_tpu starts the libtpu
runtime, so every cheaper backend gets to answer first.

docs/tpu.md records the whole measurement set -- operator parity, the istft
and fold numerics fixes, group_norm contiguity, matmul precision, per-family
throughput and the torch.compile numbers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant