perf: faster GPU decode: one SSM_CONV for the depthwise conv, less host work per step by kittyandrew · Pull Request #102 · mudler/parakeet.cpp

kittyandrew · 2026-10-18T14:50:04Z

This makes GPU decode 24-31% faster than master on Vulkan and CUDA, across three models, with the CPU path unchanged. It has two parts. The first runs the conformer's depthwise conv as one SSM_CONV, and the second removes repeated host work and host-GPU waits from every call. Each change is its own commit, followed by a commit with the test that checks it.

What changes

The depthwise conv as one SSM_CONV (`13832af`)

The depthwise conv (Conv1d with groups = C) runs as im2col, a mul_mat and two transposes, and the batched encoder runs that once per item. On a GPU backend this PR computes it as one ggml_ssm_conv over the input padded to T + K - 1 frames. SSM_CONV sums x[t + k, c] * w[k, c] over k, which is NeMo's depthwise conv for the same left padding, and it takes the batch dimension directly.

The CPU keeps im2col. SSM_CONV was no faster there, and the CPU output stays bit-identical for the NeMo parity tests.
Only where the backend can run it. active_backend_supports asks ggml_backend_supports_op about the pad and the SSM_CONV, so no graph falls back to the scheduler (7-23% slower on CUDA per AGENTS.md). This also covers ggml-cuda's limit of a channel count that is a multiple of 128.
Only kernel widths 3, 4, 5, 9 and 15. ggml-cuda has kernels for these widths only and aborts on any other, although its supports_op does not check. Every published Parakeet uses 9.
The streaming encoder is unchanged. It has its own conv with a cache.

Both conv graphs now call one builder, build_depthwise_conv. The path choice is the pure function depthwise_conv_takes_ssm, so a test can run either path through the production code.

Less host work per call

These came out of profiling a 58 s clip on Vulkan.

5059837 Skip the attention mask when it masks nothing. For one clip with full context and no padding, the encoder filled and uploaded an all-zero T x T mask per layer. Softmax without a mask gives the same bits.
74fe432 Build the relative position encoding once per length. The table for T is a slice of the table for any longer T, so one table, grown to the longest clip so far, replaces the sines and cosines of every call.
6748975 Share GpuMel's DFT basis. Each transcription built a GpuMel, and each one computed the n_bins x n_fft basis again.
f6e161c Wait once per compute. Backend::compute now queues the reads of the captures and the output behind the graph and synchronizes once. A decode step reads up to five small tensors, and on Vulkan each separate wait cost about 21 us. The scheduler path keeps its synchronous reads.
6ed7b75 Drop the copies of the LSTM gate views. A 1-D view is already contiguous, so ggml_cont only added a kernel per gate.
29f5112 Run the prediction step and the joint as one graph. After each emitted token, the TDT greedy loop ran two computes with a read-back and an upload between them. The RNN-T loop and beam search still call the halves separately.
690050b Frame GpuMel's input without copying the signal twice. Each preemphasized, padded sample is computed where the framing reads it, with the same double arithmetic.
7dd6027 Read the relative shift through a strided view. NeMo's rel_shift is a pad, a reshape, two copies and a slice over the [2T-1, T, H] scores. A view that starts at T-1, with a row stride one element shorter, reads the same values in place. The batched builder keeps its version.

Speed

Decode time summed over 73 clips: the repo's speech fixture and 72 of my own recordings, 2 to 58 s long. Each number is the median of 2 rounds, each configuration in its own process, with the trees interleaved. The machine has an RTX 4070 (driver 595.104.02, CUDA 13.4) and a Ryzen 7 7700X; Vulkan ran on the RTX 4070 alone. Builds are Release with GGML_NATIVE=ON.

Backend	Model	Same text	master	this PR	Change
Vulkan	`fluid-speech-mini-q2k-q4k`	72/73	5,129 ms	3,650 ms	-28.8%
Vulkan	`tdt-0.6b-v2-q8_0`	73/73	5,161 ms	3,721 ms	-27.9%
Vulkan	`tdt_ctc-110m-f16`	71/73	3,320 ms	2,305 ms	-30.6%
CUDA	`fluid-speech-mini-q2k-q4k`	61/73	3,680 ms	2,792 ms	-24.1%
CUDA	`tdt-0.6b-v2-q8_0`	69/73	3,706 ms	2,644 ms	-28.7%
CUDA	`tdt_ctc-110m-f16`	73/73	2,244 ms	1,652 ms	-26.4%
CPU	`tdt_ctc-110m-f16`	73/73	160,905 ms	161,693 ms	+0.5%

The CPU row is one round, and +0.5% is noise: the CPU code path changes only by the test seams.

What the host-work commits add on top of SSM_CONV alone, on Vulkan:

Model	Same text	SSM_CONV alone	this PR	Change
`fluid-speech-mini-q2k-q4k`	73/73	4,504 ms	3,574 ms	-20.6%
`tdt-0.6b-v2-q8_0`	73/73	4,511 ms	3,572 ms	-20.8%
`tdt_ctc-110m-f16`	73/73	2,993 ms	2,315 ms	-22.7%

The first revision of this PR measured SSM_CONV alone on two integrated GPUs through Mesa: 13-15% faster on the Radeon in a Ryzen 7 7700X (RADV), and 6-7% on the Intel UHD in a Core i5-10210U (ANV). I did not rerun those with the host-work commits.

Transcripts

The host-work commits change no text: SSM_CONV alone and this PR agree on 73/73 clips for every model. The differences against master come from SSM_CONV's summation order. They are mostly punctuation and casing, and the 2-bit Fluid Speech Mini on CUDA is the most sensitive. The changed outputs move toward the other backends: with SSM_CONV, Fluid Speech Mini on CUDA matches the Vulkan text on 63 clips (master: 56).

On 1,400 clips from seven Open ASR Leaderboard sets, Fluid Speech Mini on Vulkan gave 6.56% WER on master and 6.58% with SSM_CONV alone. I did not rerun that with the host-work commits, since they change no text.

Tests

Every changed path has a test. The ones marked CI are model-independent and run in the build job (ctest -LE model). The new label gpu marks tests that run GPU code; without a GPU device they exit 77, so the CPU job reports them as skipped.

Change	Test	Checks	Where
SSM_CONV	`test_depthwise_conv`	Both paths through `build_depthwise_conv` on the CPU backend, against NeMo's conv computed on the host: causal and symmetric padding, B = 1 and 3, T from 1 to 50, K in {3, 4, 5, 9, 15}.	CI
SSM_CONV choice	`test_depthwise_conv_path`	The CPU, K = 7 and a backend that declines the ops all take im2col; the backend is asked only when it matters.	CI
SSM_CONV on GPUs	`test_depthwise_conv_gpu`	On every GPU device: SSM_CONV at every supported width, im2col at K = 7 and at C = 100 on CUDA; results match CPU im2col and NeMo within 1e-4.	`gpu`
Mask skip	`test_relpos_mask`	The mask equals NeMo's in padded, chunked and windowed cases, and is skipped only where it has no -inf entry.	CI
Position table	`test_pos_enc_cache`	Mixed lengths, a d_model change and back, and eight threads all match a fresh table, bit for bit.	CI
DFT basis	`test_mel_gpu_basis`	The shared basis equals the per-instance one; one object per size, across threads too.	CI
GpuMel on a GPU	`test_mel_gpu_device` (`test_mel_gpu --gpu`)	GpuMel on the first GPU device against MelFrontend, as production runs it. Fails on every GPU, on master too: see below.	`model`, `gpu`
Single wait	`test_backend_readback_gpu`	On every GPU device, large and small outputs read back exactly over 20 rounds, also through the scheduler.	`gpu`
LSTM views, fused step	`test_pred_joint_step`	The fused step equals step + step_logits bit for bit, `step()` matches a host LSTM within 1e-4, and an SOS step fetches the embedding table. `--gpu` runs it on a GPU.	`model`, `gpu`
GpuMel framing	`test_mel_gpu_frames`	The frames equal the old two-copy framing, bit for bit.	CI
Strided rel_shift	`test_rel_shift`	The view and `ac + rel_shift(bd)` equal NeMo's rel_shift on the host, bit for bit.	CI
rel_shift on GPUs	`test_relpos_attention_gpu`	On every GPU device, the strided add matches exactly, and softmax without a mask matches the zero mask within 1e-5.	`gpu`

For each test I put a bug into the production code, saw the test fail, and took the bug out again. The bugs included a swapped or one-off pad, a scrambled batch axis, a decision that ignores the backend, a mask skipped one frame early, the wrong slice of the position table, swapped LSTM gates, a fused step that ignores SOS, a framing shifted one sample and a rel_shift view with the wrong stride. One gap remains: on CUDA, removing the final synchronize does not fail test_backend_readback_gpu, so that test catches a missing wait on Vulkan only.

The seams that make this testable leave production's choices as they were: build_depthwise_conv and depthwise_conv_takes_ssm (conformer.hpp), relpos_attention_mask and rel_shift (relpos_attention.hpp), dft_basis and gpu_mel_frames (mel_gpu.hpp), pred_joint_step (tdt.hpp), and PredictionNet::embed_table_fetched.

Results. The CI build job (-DPARAKEET_BUILD_TESTS=ON -DGGML_NATIVE=OFF, ctest -LE model) passes on every commit: 62 tests, 52 pass and 10 skip. With the 110M model, 19 model tests run and 18 pass; test_relpos_attention_local_chunked fails with the same output on master. On the GPUs (RTX 4070 through Vulkan and CUDA, Radeon iGPU through Vulkan), every gpu test passes except test_mel_gpu_device. The worst differences measured were 7.6e-6 against a 1e-4 limit, and 6.0e-8 against 1e-5.

Found along the way, not changed here

GpuMel on a GPU differs from MelFrontend by up to 0.28 (Vulkan, RTX 4070) and 0.15 (iGPU, CUDA), means 0.011 and 0.0046, so the new test_mel_gpu_device fails its 0.05 limit. The same test linked against master's library gives the same numbers, and this PR's GpuMel inputs are bit-identical to master's (test_mel_gpu_basis, test_mel_gpu_frames). The difference most likely comes from the DFT matmul on the GPU; I have not traced it into the kernels. I left the test failing rather than fit a limit to one machine: production already decodes from these values on GPUs.
MelKernel reads freed memory on GPU builds if the weights are not realized first: realize_weights frees the host tensors (fix: release host weight staging buffer after device upload (47% memory saving) #53), and MelKernel reads a tensor's shape after the weight read that triggers it. Model::load realizes them first, so production is safe; test_mel_gpu crashed on GPU builds until it did the same.
On CUDA, 16 tests abort at exit ("driver shutting down"), because they free a CUDA buffer without calling pk::shutdown_backend. test_capi_concurrency expects three CPU backends, which a GPU build does not make.

I could not run Metal. ggml-metal has an SSM_CONV kernel; if its supports_op declines a shape, the conv keeps im2col, and test_depthwise_conv_gpu would check that on a Metal build.

The depthwise conv was im2col + mul_mat + two transposes, and the batched path ran it once per item and concatenated. On a GPU backend it is now one ggml_ssm_conv over the input padded to T + K - 1 frames, which also takes the batch dimension directly. Its sum is NeMo's conv for the same left padding, causal or not. The CPU keeps the im2col path: it was no faster there, and its output stays bit-identical for the NeMo parity tests. A GPU backend that cannot run the pad or the SSM_CONV for this shape (ggml_backend_supports_op, through a new active_backend_supports) keeps im2col as well, so no graph falls back to the scheduler. ggml-cuda's supports_op does not check the kernel width, but its kernels exist only for widths 3, 4, 5, 9 and 15 and abort on any other, so other widths keep im2col on every backend; every published Parakeet uses 9. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

localai-org-maint-bot

The new GPU path needs a checked-in numerical regression before merge. At this head, depthwise_conv_ssm returns null on CPU, so the reported CPU ctest runs exercise the old im2col path. The diff adds no test for the new SSM_CONV graph; the manual transcript and WER comparisons are useful evidence but do not protect its padding, batch layout, or fallback behavior against later changes.

Please add a focused comparison through the production convolution builder against the existing im2col/reference result for causal and symmetric padding, B=1 and B>1, and short sequences. Include a supported GPU case that proves SSM_CONV was selected, and an unsupported-width/backend case that proves fallback. A hardware-gated test can skip explicitly when the backend is unavailable; the CPU suite must not report that it validated this GPU path.

@mudler @richiejp: source review only. I have not reproduced the GPU measurements or established a numerical defect; the outstanding blocker is executable regression coverage for the changed path.

The SSM_CONV path only runs on GPU backends, so the CPU suite never exercised it. The depthwise conv now has one builder, build_depthwise_conv, which both conv module overloads call and which a test can force onto either path. The choice Auto makes is the pure function depthwise_conv_takes_ssm, and production chooses exactly as before. test_depthwise_conv runs both paths on the CPU backend against NeMo's conv computed on the host: causal and symmetric padding, one item and three, T from 1 (below K) to 50, at every width ggml-cuda has a kernel for. test_depthwise_conv_path covers the fallback: the CPU, K = 7 and a backend that declines the ops all take im2col. test_depthwise_conv_gpu runs Auto on every GPU device, checks that it took SSM_CONV (or im2col where the device declines, as ggml-cuda does for C = 100), and compares it with the CPU's im2col result. It exits 77 without a GPU device and carries the new label gpu. For the batched graph with B = 1, the im2col path now reads the input directly instead of a copy of it, as the single-clip graph already did. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

For one clip with full context and no padding, every entry of the relative-position attention mask is zero. The encoder still filled a T x T array on the CPU for each layer and uploaded it: 24 x 2 MB for a 58 s clip. Softmax without a mask gives the same result bit for bit, so pass none in that case. Padded, chunked and local-window masks are built as before. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

…hing The mask the single-clip attention builds is now its own function, relpos_attention_mask, which returns null for a full-context clip without padding. test_relpos_mask compares it with NeMo's mask over padded, unpadded, chunked-limited and sliding-window cases, and fails if it is skipped anywhere that mask has a -inf entry. It also checks on the CPU backend that softmax without a mask gives the same bits as softmax with the all-zero mask it replaces. test_relpos_attention_gpu, added with the rel_shift test, checks that on GPUs. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

rel_pos_encoding computed (2T-1) x d_model sines and cosines on every encoder call. Row p holds position (T-1)-p, so the table for T is a slice of the table for any longer T, bit for bit. Keep one table, grown to the longest clip so far, and copy the slice. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

rel_pos_encoding now returns a slice of one table grown to the longest clip so far. test_pos_enc_cache calls it for lengths in a mixed order (shorter after longer, repeats, T = 1), switches d_model and back, and calls it from eight threads at once. Every result must equal, bit for bit, the table computed directly for that length. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

A GpuMel is built for every transcription, and its constructor computed the n_bins x n_fft cosine and sine basis each time. The basis depends only on those two sizes, so build it once per size and share it. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

dft_basis is now declared in mel_gpu.hpp. test_mel_gpu_basis checks that each size's shared basis holds exactly what GpuMel's constructor computed per instance, that a size returns the same object every time and another size its own (two of the sizes share n_fft), and that threads asking at once all get one object. GpuMel only runs on GPU backends in production, but test_mel_gpu runs it on whatever backend the build picks, the CPU in CI. With --gpu it now runs it on the first GPU device and exits 77 without one; ctest registers that as test_mel_gpu_device, labelled model and gpu. The test now realizes the weights right after the load, as Model::load does. On a GPU, realize_weights frees the host tensors, and MelKernel reads a tensor's shape after the weight read that realizes them, so on GPU builds the test read freed memory: it crashed on some devices and read a wrong window on others. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

Backend::compute ran the graph, waited, then read every captured tensor and the output with a wait each. A decode step reads up to five small tensors, and on Vulkan each separate read cost about 21 us. Queue the reads behind the compute with the async API and synchronize once. The scheduler path keeps its synchronous reads; backends without async reads fall back to a plain read inside ggml. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

Backend::compute now queues the reads of the captures and the output behind the graph and waits once. On the CPU those reads stay synchronous, so only a GPU exercises the change. test_backend_readback_gpu runs, on every GPU device, graphs with a 4M-float output and small captures whose inputs change every round, and checks every value read back: a read that returned before the copy finished would hold the previous round's values. A graph with a custom op, which GPU backends lack, checks the scheduler path the same way. It exits 77 without a GPU device. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

PredictionNet::step copied each 1-D gate view of z with ggml_cont before its activation. A 1-D view is already contiguous, so each copy only added a kernel: eight per step on a two-layer prediction net. step_batch keeps its copies, since its 2-D views skip the other gates between columns. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

After each emitted token the TDT greedy loop ran the prediction net, read its output back, and uploaded it again for the joint: two computes, two waits. Split both graphs into builders, PredictionNet::build_step and Joint::build_step, and chain them into one graph whenever the prediction output is stale. Steps that reuse the cached output run the joint alone, as before. The RNN-T loop and beam search still call the halves separately. build_step fetches the host copy of the embedding table on every step, SOS included, as step() did. A warm-up on silence emits no token, and it must still pay that one-time read, or the first real transcription after a load does. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

tdt_greedy's one-graph prediction step and joint is now the function pred_joint_step, so a test can call what the decoder runs. test_pred_joint_step steps through SOS and three tokens, carrying the state, and checks three things. pred_joint_step must give what PredictionNet::step and Joint::step_logits give as two computes: the same bits on the CPU. PredictionNet::step, whose LSTM gates are now views without copies, must match the LSTM computed on the host in double from the GGUF weights within 1e-4. A fresh PredictionNet must have fetched its embedding table after an SOS step, which the new embed_table_fetched reports, so a warm-up on silence pays that read. It needs PARAKEET_TEST_GGUF. With --gpu, registered as test_pred_joint_step_gpu, it runs on the first GPU device instead, holds the fused step to 1e-4, and exits 77 without one. test_exact_batch already covers both changes on the CPU, bit for bit: step() against step_batch, which keeps its copies, and tdt_greedy against the batched decoder. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

GpuMel built the preemphasized signal and then a zero-padded copy, both in double, before cutting frames from it. For a 58 s clip that is two 7 MB arrays per call, and their allocation, zeroing and copies took 7 of the mel stage's 16 ms. Compute each preemphasized, padded sample where the framing loop reads it, with the same double arithmetic, so the frames are unchanged bit for bit. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

GpuMel's framing is now the function gpu_mel_frames. test_mel_gpu_frames compares its frame matrix, bit for bit, with the one GpuMel built before from a preemphasized copy and a padded copy of the signal: clips from 1 sample to just over a second, around the hop and n_fft boundaries, with preemphasis on and off. test_mel_gpu cannot catch such a change by itself: its 0.05 limit let a framing bug that skipped the preemphasis of one sample come within 0.005 of passing. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

The single-clip attention built NeMo's rel_shift as pad, reshape, drop a row, copy, slice, copy: three passes over the [2T-1, T, H] position scores, 32 MB per layer for a 58 s clip. The result at (key j, query q) is bd[T-1-q+j, q], so a view starting at T-1 whose row stride is one element shorter reads it in place. The values and the add that consumes them are unchanged. The batched builder keeps its version. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

The view is now the function rel_shift. test_rel_shift computes NeMo's rel_shift on the host (pad a column, reshape, drop a row, reshape, slice) and checks, on the CPU backend and bit for bit, both the view copied out and the scores ac + rel_shift(bd) that build_graph adds, for T from 1 to 33 and one or four heads. test_relpos_attention_gpu runs those scores on every GPU device, since the device's add now reads a non-contiguous input, and checks that softmax without a mask equals softmax with the all-zero mask and the CPU's softmax. It exits 77 without a GPU device. Assisted-by: Claude:claude-opus-5-5 [Claude Code]

kittyandrew · 2026-10-24T21:24:15Z

Written by an AI agent (claude-code/claude-opus-5-5), ownership of @kittyandrew.

Thanks, fair point: the CPU suite never ran the SSM_CONV graph. The new head adds executable coverage for it.

Through the production builder. The depthwise conv is now one function, build_depthwise_conv, which both conv module overloads call and which a test can force onto either path. Production still decides the same way, through depthwise_conv_takes_ssm.
SSM_CONV against im2col and NeMo (test_depthwise_conv, model-independent, runs in CI). Both paths run on the CPU backend against NeMo's conv computed on the host. It covers causal and symmetric padding, B = 1 and B = 3, T from 1 (below K) to 50, and K in {3, 4, 5, 9, 15}. It runs ggml-cpu's SSM_CONV kernel, not a GPU's, and says so.
Fallback (test_depthwise_conv_path, runs in CI). The CPU, K = 7 and a backend that declines the ops all take im2col. The backend is not even asked on the CPU or at K = 7.
GPU, hardware-gated (test_depthwise_conv_gpu, new label gpu). It runs Auto on every GPU device of the build and checks that it built SSM_CONV. At C = 100, where ggml-cuda declines the op, it checks for im2col. It compares the result with the CPU's im2col result. Without a GPU device it exits 77 with a message, so the CPU job reports it as skipped.

I planted a bug for each test (a swapped pad, a scrambled batch axis, items read in reverse, a decision that ignores the backend) and saw it fail. The numbers are in the PR body.

On this machine the GPU test ran on an RTX 4070 through Vulkan and through CUDA, and on a Radeon iGPU through Vulkan. Auto chose SSM_CONV everywhere except K = 7, and except C = 100 on CUDA. Results match CPU im2col within 7.2e-7 (limit 1e-4).

The head also folds in the nine host-overhead commits from our fork (one is squashed into the commit it fixed). Each comes with a test of its own, held to the same bar. Against master, the whole PR is 24-31% faster on Vulkan and CUDA across three models; the PR body has the tables. One new GPU check, test_mel_gpu_device, fails on every GPU, and also on master: GpuMel's DFT matmul differs there from the CPU front end by up to 0.28, against a 0.05 limit. I describe it in the body rather than loosen the limit quietly.

kittyandrew mentioned this pull request Oct 18, 2026

Run Fluid Speech Mini on the GPU on Linux through parakeet.cpp and Vulkan watzon/sayso#26

Merged

4 tasks done

localai-org-maint-bot requested changes Oct 22, 2026

View reviewed changes

kittyandrew added 16 commits October 24, 2026 22:33

kittyandrew changed the title ~~perf: run the conformer's depthwise conv as one SSM_CONV on GPU backends~~ perf: faster GPU decode: one SSM_CONV for the depthwise conv, less host work per step Oct 24, 2026

Navigation Menu

Repository navigation

perf: faster GPU decode: one SSM_CONV for the depthwise conv, less host work per step - #102

perf: faster GPU decode: one SSM_CONV for the depthwise conv, less host work per step#102
kittyandrew wants to merge 17 commits into
mudler:masterfrom
kittyandrew:depthwise-ssm-conv

kittyandrew commented Oct 18, 2026 •

edited

Loading

Uh oh!

localai-org-maint-bot left a comment

Uh oh!

kittyandrew commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kittyandrew commented Oct 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

The depthwise conv as one SSM_CONV (13832af)

Less host work per call

Speed

Transcripts

Tests

Found along the way, not changed here

Uh oh!

localai-org-maint-bot left a comment

Choose a reason for hiding this comment

Uh oh!

kittyandrew commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kittyandrew commented Oct 18, 2026 •

edited

Loading

The depthwise conv as one SSM_CONV (`13832af`)