Repository navigation
Run Fluid Speech Mini on the GPU on Linux through parakeet.cpp and Vulkan - #26
Conversation
…lkan A new engine kind, ParakeetCpp, names one GGUF file in a Hugging Face repository at a pinned revision, plus the Kroko archive for the live preview. The engine downloads that file next to the archives, with the same staging and .complete marker, and runs its final pass on parakeet.cpp's C API with ggml's Vulkan backend. A warm-up after the load compiles the GPU shaders before the first dictation. Without a usable GPU the model is hidden: on Linux, model_runs_here probes Vulkan for a discrete GPU, then an integrated one, and never counts a CPU device such as lavapipe. The app passes the chosen GPU to the engine as GGML_VK_VISIBLE_DEVICES. A load without a GPU still fails, with the new reason no_gpu. With a GPU and English, onboarding offers the model second. build.rs fetches parakeet.cpp and its ggml as two pinned archives checked by SHA-256, applies parakeet.cpp's ggml patches and links it statically, so the engine needs only libvulkan.so.1. On Ubuntu 22.04, scripts/linux-vulkan-deps.sh builds the Vulkan headers and glslc that ggml needs, and the release job caches them. The Nix package gets vulkan-loader as a build input, so autoPatchelf resolves the engine's libvulkan.so.1. Co-Authored-By: claude-code/claude-opus-5-5 <noreply@anthropic.com>
Onboarding click-throughI ran the packaged build of 588ebf4 (
One gap: the Fluid Speech Mini card doesn't say it needs a GPU. On a machine without one, the model just doesn't appear. |
… changes Fluid Speech Mini on an integrated GPU was about three times slower than Parakeet TDT v2 on the same computer's CPU: 4.0 s for 58 s of speech on a Ryzen 7 7700X's Radeon, and 13.5 s on a Core i5-10210U laptop's Intel UHD, where its CPU took 4.7 s. The app's probe and the engine's check now count only a discrete GPU. The engine checks the device parakeet.cpp would pick, so a laptop that lists its integrated GPU first refuses too, and the no_gpu text says a dedicated graphics card. A GGUF's folder in the store is now its commit, not its file name, so a new revision of the same file downloads again instead of loading the old weights. The Hugging Face override is SAYSO_HF_ENDPOINT, as in the portable engine. The build dependencies are Linux-only, so macOS and Windows build what they did before. The single-file download shares the archive's staging code, a GGUF no longer goes through the sherpa-onnx file layout, and the Linux catalog has its own Fluid Speech Mini description. The docs, the engine's module docs and CONTRIBUTING's list of packaging triggers follow. Co-Authored-By: claude-code/claude-opus-5-5 <noreply@anthropic.com>
Integrated GPUs are too slow, so e2cea27 counts only discrete GPUsI measured a laptop GPU for question 1: a ThinkPad with a Core i5-10210U and Intel UHD Graphics (Comet Lake GT2, Mesa ANV). I used the same 12 recordings as the table in the description, 3 fresh engine processes per row, and medians.
That is about 3 times slower than the CPU model, as on the 7700X's Radeon (4.0 s for 58 s against 1.5 s). The text matched the RTX 4070 on 9 of 12 recordings. So e2cea27 counts only a discrete GPU, in the app's probe and in the engine's check. Question 1 in the description now asks whether that is the rule you want. I ran both machines again on the packaged build of e2cea27, through AT-SPI, with every Vulkan driver visible:
e2cea27 also makes a GGUF's folder its commit, so a new revision downloads again. It renames the override to |
The engine links both statically, and their MIT License asks for the license text in every copy. The texts from the pinned commits sit in crates/sayso-engine/licenses, and the tarball installs them under share/licenses/sayso, which the deb, rpm, AUR, Nix, AppImage and Flatpak packages carry. The Models page keeps its credit line. The engine's download tests now serve the GGUF from the same test server as the archives, through its redirect, as Hugging Face's resolve URLs redirect too. Co-Authored-By: claude-code/claude-opus-5-5 <noreply@anthropic.com>
@watzon, one question is left before this can leave draft: should an integrated GPU get Fluid Speech Mini? Your rule was a discrete GPU first, then an integrated one. This PR now counts only a discrete GPU, because both integrated GPUs I could measure were slower than the CPU model on the same computer. I moved the other questions to "Postponed" in the description; none of them block this PR. The final pass, with 5 s of GPU idle before each clip, medians of 3 fresh engine processes:
The options I see:
If option 1 is fine, this PR is done from my side. If you prefer option 2, I can add it here. |
The conv patch from the issue is now sent upstream as mudler/parakeet.cpp#102: 4-15% faster decode on NVIDIA (Vulkan, CUDA) and on AMD and Intel integrated GPUs, with the same WER on the 1,400-clip set (6.58 against 6.56). This PR does not depend on it. Once it merges, the pin in |
|
@kittyandrew, I'm going with option 1: discrete GPUs only. Thanks for measuring this. On both integrated GPUs you tested, the model was 2.6-2.9 times slower than Parakeet TDT v2 on the CPU. A 58 s dictation took 13.5 s on the laptop's GPU and 4.7 s on its CPU. Option 2 would show a slower model to people who can't tell it's slower, and option 3 needs a number from a large integrated GPU that nobody has measured yet. If someone with a Radeon 780M or a recent Intel Xe can run your benchmark later, I'm happy to add integrated GPUs with a limit based on that data. So this part is done. I'll review the code next and run the manual test and clippy jobs before I merge. |
sounds good |
Closes #25. This follows the answers in #25 (comment).
Note
Fluid Speech Mini is offered only with a discrete GPU: both integrated GPUs I measured were about 3 times slower than the CPU model. That is safe to merge as is. Whether integrated GPUs should get the model is your call, in the comments (#26 (comment)). The untested points are under "Not proven yet".
What changes
sayso-core). A newEngineKind::ParakeetCpp { repo, revision, file, preview_archive }names one GGUF file in a Hugging Face repository at a pinned commit, plus the Kroko archive for the live preview. Fluid Speech Mini joins the Linux catalog with idfluid-speech-mini, the same id as on macOS. Its Linux entry has its own WER (6.56, measured on Linux, see below), a link to the GGUF repository, its own description ("runs on the graphics card"), and a credit line for parakeet.cpp and ggml (MIT).EngineKind::needs_gpumarks it.download.rs,engine.rs). The engine fetches<repo>/resolve/<revision>/<file>into a folder named by the revision, with the same staging folder and.completemarker as the archives, so a new revision downloads again.SAYSO_HF_ENDPOINToverrides the host, as in the portable engine.parakeet.rs). Aparakeet_ggufrecipe loads the file through parakeet.cpp's C API. Unless the device parakeet.cpp picks is a discrete GPU, the load fails with the new reasonno_gpu, and the app shows "This model needs a dedicated graphics card with a Vulkan driver, and Sayso found none. Choose a model that runs on the CPU in Models." There is no CPU fallback. After the load, a warm-up decodes 1, 30 and 60 s of silence, so the driver compiles its shaders before the first dictation.sayso-app). On Linux,model_runs_hereprobes Vulkan withash(already in the lock through wgpu): it counts only a discrete GPU, not an integrated one and not a CPU device such as llvmpipe or lavapipe. Without one, the model is hidden. The app passes the chosen GPU to the engine asGGML_VK_VISIBLE_DEVICESwhen it starts it, unless the user set that variable. With a discrete GPU and English as the dictation language, onboarding offers Fluid Speech Mini as its second model; Parakeet TDT v2 stays first.build.rs). No submodule. Theparakeetfeature, on by default and Linux only, downloads two pinned GitHub archives and checks their SHA-256: parakeet.cpp5c9b0fband its ggml submodulee705c5f, which the archive leaves out. It applies parakeet.cpp's own ggml patches withpatch(its CMake applies them only to a git checkout), and writes a marker file last, so a tree that the CI cache emptied is downloaded again.SAYSO_PARAKEET_CPP_DIRbuilds a local tree instead. The engine links parakeet.cpp statically and needs onlylibvulkan.so.1at run time.--no-default-featuresbuilds without it. The build dependencies are Linux-only, so macOS and Windows build what they did before.scripts/linux-deps.shaddscmakeandpatch, andglslcwhere apt has it. Ubuntu 22.04 has noglslcpackage and Vulkan headers too old for ggml (1.3.204), so the newscripts/linux-vulkan-deps.shbuilds Vulkan-Headers v1.4.363 and shaderc v2026.4'sglslcfrom source.linux.ymlcaches the result per architecture, keyed by the script. LunarG's SDK has no Linux aarch64 tarball (itslatest.jsonlists only linux, mac, windows and warm, and the aarch64 URL returns 404), so the source build covers both runners.vulkan-loaderjoinsbuildInputsinpackaging/nix/package.nix, so autoPatchelf can resolve the engine'slibvulkan.so.1. It stays inruntimeDependenciesfor the app, which opens it at run time.docs/linux.md,docs/local-models.md,docs/benchmark.md(a Linux section),CONTRIBUTING.md,README.md, and the reason table insayso-engine-client/NOTES.md.How you checked it
On NixOS, x86_64, Rust 1.98.1, Ryzen 7 7700X with an RTX 4070 (driver 595.104.02) and the CPU's integrated Radeon:
cargo test --workspacecargo clippy --workspace --all-targets -- -D warningsscripts/check-deps.shNew tests: parsing the engine kind and rejecting bad repository, revision and file names; a GGUF download from a local server, and a new revision downloading again; the engine binary refusing a GPU model with
no_gpuwhenVK_DRIVER_FILESnames no driver; the GPU pick (only a discrete GPU counts); and the onboarding choices with and without a GPU.By hand:
VK_DRIVER_FILESset to each driver in turn, the app's probe chose the RTX 4070 from all drivers and nothing from the Radeon alone, lavapipe alone, or no driver. With the Radeon's driver listed first, it still chose the RTX 4070, and the engine refused when only the Radeon was visible to it.no_gpu.docs/benchmark.md's 1,400 clips, with the same harness andSAYSO_ENGINE_PATHset to this engine:ubuntu:22.04(Docker, x86_64),linux-deps.shthenlinux-vulkan-deps.shbuilt the tools in 106 s (a 6.9 MB cache), and parakeet.cpp's Vulkan build compiled all its shaders. A control run with 22.04's own headers failed inggml-vulkan.cpp, as expected. The 24.04 apt-only path, as intest-linux, also built.packaging/nix/package.nixfrom a tarball of this branch (scripts/bundle-linux.sh). Before the fix below, autoPatchelf failed: it addsruntimeDependenciesto RPATH but does not use them to resolve aNEEDEDentry, and the engine now needslibvulkan.so.1directly. Withvulkan-loaderalso inbuildInputs, the packaged engine found it, loaded the model on the RTX 4070 and transcribed, with an empty environment.Not proven yet
mainsaves its own on its first run. Nothing ran the GPU model on an arm GPU.The warm-up and the download. You asked for the warm-up to run while the user waits for the download. The engine loads a model only when all of its files are on disk, so the 14 s shader compile starts when the download ends. Continue is enabled at once, though, and the first step that needs the model is the practice step (6 of 7), three steps later. In both click-throughs the load took 13-14 s after the download, so a user who spends that long on those three steps never waits. I left the engine as it is.
Postponed
These do not block this pull request. Each can be a follow-up if you want it.
glslc(2023.8) lacks a few extensions that shaderc v2026.4 has, sotest-linuxand contributor builds on 24.04 build fewer shader variants than the release, and only the release build covers the integer-dot shaders.test-linuxhas no GPU, so it never runs them; it stays as it is.share/licenses/saysoof every package, and the Models page keeps the credit line. The packages still ship only Sayso's own LICENSE for the other libraries, such as onnxruntime (MIT).The conv patch from the issue is not in this pull request and nothing depends on it. With
SAYSO_PARAKEET_CPP_DIRpointing at a patched tree, the same engine is 10-24% faster per dictation, with the same text on 12 of 12 recordings and the same memory. Sending it to parakeet.cpp, as you suggested, is a separate step.