feat(huddle): make STT model selectable + add multilingual Parakeet v3 (#2478) by artile · Pull Request #2520 · block/buzz

artile · 2026-07-23T14:15:50Z

Summary

Huddle transcription was hard-pinned to an English-only STT model, so non-English speech transcribed as garbage (#2478). There was no language setting, env override, or model-swap path — the loader knew exactly one model.

This makes the huddle STT model selectable, and ships a multilingual option alongside the English default.

Closes the "no model-swap path" root cause in #2478; see the coverage note below for what is and isn't covered by the bundled multilingual model.

What changed

huddle/models.rs — a small STT model registry. STT_MODELS is the single source of truth for each model: id, download URL, archive SHA-256, expected files, sherpa-onnx family, manifest version, license text, and language coverage.
select_stt_model(override, locale) — pure, unit-tested selection. Precedence (issue Huddle transcription is English-only: non-English speech (e.g. Korean) produces garbled transcripts #2478 options 2 + 3):
1. BUZZ_STT_MODEL env override, when it names a known model (case-insensitive),
2. else a non-English system locale → the multilingual model,
3. else the English default.
Two models ship, both with a pinned SHA-256:

id model family languages

parakeet-en (default) Parakeet TDT-CTC 110M NeMo CTC English

parakeet-v3 Parakeet TDT 0.6B v3 transducer 25 European (auto-detected)
huddle/stt.rs configures sherpa-onnx per family: NeMo CTC (single model.int8.onnx) or transducer (encoder/decoder/joiner.int8.onnx); tokens.txt is shared. Downloaded on demand via the existing flow.
Per-model checksum is now Option. The size cap, safe extraction, and expected-files verification always apply; the English default stays hash-pinned. Both shipped models are pinned today (the v3 hash was computed from the k2-fsa release archive, since upstream publishes none).

Model choice — why Parakeet v3, and the honest coverage note

The issue suggested NVIDIA Parakeet v3, and a sherpa-onnx export exists (sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8), so it is what this PR bundles. It adds 25 European languages with automatic language-ID.

Parakeet v3 covers European languages only. The concrete case in #2478 is Korean, which v3 does not cover. This PR removes the architectural root cause (the English-only hard pin) and adds the multilingual model + per-family config path; a CJK model (e.g. SenseVoice-Small, KO/EN/JA/ZH/yue) is now a one-entry follow-up in the registry — the transducer/CTC plumbing it would reuse is already here. Happy to add it in a follow-up if you'd like Korean in this same PR instead.

Testing

cargo check -p buzz-desktop — clean.
cargo clippy -p buzz-desktop — clean, no warnings.
cargo fmt — clean.
cargo test -p buzz-desktop huddle::models — 8 passed, 0 failed. New tests cover: English default, non-English locale → multilingual, override precedence + case-insensitivity, unknown/empty override fallback, registry invariants (unique ids, English default hash-pinned, a multilingual model exists), license-sidecar readiness, and per-family (transducer) readiness.

I can't run a live non-English transcription in CI here (it needs the ~465 MB v3 model + a huddle), but the selection/config/readiness paths are unit-tested and the archive contents/subdir/files were verified against the k2-fsa release. Glad to help test Korean once a CJK model lands (ties back to #2478).

Notes

Zero unsafe; no panics/unwraps on the new paths.
No new dependencies (locale is read from standard POSIX env vars).
The English model stays the default and is unchanged for existing users.

Refs #2478

…block#2478) Huddle transcription was hard-pinned to an English-only STT model, so non-English speech transcribed as garbage (block#2478). There was no language setting, env override, or model-swap path — the loader knew one model. Make the huddle STT model selectable behind a small registry: - `STT_MODELS` in huddle/models.rs is the single source of truth for each model: id, download URL, archive checksum, expected files, sherpa-onnx family, manifest version, license, and language coverage. - `select_stt_model` picks a model from the `BUZZ_STT_MODEL` override, then the system locale (non-English -> multilingual), then the English default. The function is pure and unit-tested. - Two models ship: the existing Parakeet TDT-CTC 110M (English, default) and Parakeet TDT 0.6B v3 (multilingual, 25 European languages). Both carry a pinned SHA-256; the v3 hash was computed from the k2-fsa release archive. - huddle/stt.rs configures the sherpa-onnx recognizer per family: NeMo CTC (single model.int8.onnx) or transducer (encoder/decoder/joiner). tokens.txt is shared. The archive checksum is now optional per model (size cap, safe extraction, and expected-files verification always apply); the English default stays pinned. Coverage note: Parakeet v3 covers European languages only. Korean/CJK — the concrete case in block#2478 — is a follow-up: the registry makes adding a CJK model (e.g. SenseVoice-Small) one data entry plus a family already modelled. Zero unsafe, no panics/unwraps on the new paths. New model selection logic is covered by unit tests in huddle::models. Refs block#2478 Signed-off-by: Taras Kornichuk <dread9ko@gmail.com>

LauraGPT

Thanks for opening the model-selection path. I tested the exact language from #2478 and found one correctness gap: the current locale fallback still routes ko-KR to Parakeet v3, although v3 does not support Korean.

I verified the official sherpa-onnx SenseVoice export end to end:

archive: sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2
size: 163002883 bytes
SHA-256: 7d1efa2138a65b0b488df37f8b89e3d91a60676e416f515b952358d83dfd347e
required files: model.int8.onnx, tokens.txt
coverage: zh/en/ja/ko/yue
live CPU check with sherpa-onnx 1.13.4: the packaged 4.608 s Korean sample decoded in 0.211 s, detected <|ko|>, and produced normal Korean text. The packaged zh/ja/en/yue samples also decoded successfully.

The smallest complete fix is to add a SenseVoice family and registry entry, then select models by declared primary-language tags: zh/ja/ko/yue -> SenseVoice, Parakeet's supported European tags -> v3, en -> the existing English default. Unknown non-English locales should not silently select a model that does not support them.

One licensing detail: SenseVoice is under the FunASR Model Open Source License 1.1, not CC-BY-4.0. The archive includes a LICENSE pointer to the FunASR license, so the generic license_text comments should not assume every model is CC-BY.

LauraGPT · 2026-07-23T14:49:38Z

+            .unwrap_or("")
+            .to_ascii_lowercase();
+        if !lang.is_empty() && lang != "en" {
+            if let Some(model) = STT_MODELS.iter().find(|m| m.multilingual) {


This selects the first multilingual model for every non-English locale. In the reported case, ko-KR therefore selects parakeet-v3, but the registry itself documents that v3 has no Korean coverage. Please dispatch on supported primary-language tags rather than a boolean (and add a regression test for ko-KR). If SenseVoice is deferred, at minimum only auto-select v3 for its 25 declared language tags so this PR does not replace one unsupported model with another.

LauraGPT · 2026-07-23T14:49:38Z

+    // Coverage note: Parakeet v3 covers European languages only. CJK/Korean
+    // (the concrete case in #2478) is not covered by this model; the registry
+    // makes adding a CJK model (e.g. SenseVoice-Small) a follow-up — one data
+    // entry, no engine change beyond a family already handled here.


SenseVoice is not one data entry with the currently handled families: sherpa-onnx 1.13.4 exposes it through cfg.model_config.sense_voice, so this needs a SttFamily::SenseVoice arm in stt.rs. The configuration is a single model.int8.onnx path plus language = "auto" and use_itn = true; it should not be configured as NemoCtc or Transducer.

LauraGPT · 2026-07-23T15:23:51Z

I implemented the complete SenseVoice follow-up and opened a collaboration PR directly against your current branch: artile#1

It is based exactly on 9236cb855d7f2b09d6508f35a9f412f14563ca71, so merging it will update this PR without a rebase or cherry-pick. The patch adds the SenseVoice sherpa-onnx family, routes zh/yue/ja/ko to the verified int8 model, limits Parakeet v3 auto-selection to its declared European languages, and uses the correct FunASR model license.

Validation completed on the exact patch:

official zh/en/ja/ko/yue samples decoded successfully on CPU with sherpa-onnx 1.13.4;
Korean sample: 4.608 s audio, 0.211 s decode, detected <|ko|>;
cargo test --manifest-path desktop/src-tauri/Cargo.toml huddle:: -- --nocapture: 102 passed, 0 failed;
Clippy with -D warnings, rustfmt check, and git diff --check: passed.

LauraGPT · 2026-07-26T06:38:44Z

@artile, the #2478 reporter has now explicitly confirmed that Korean coverage is the acceptance criterion and asked for the SenseVoice path to be included in this PR.

The ready collaboration PR is still cleanly based on your exact head 9236cb855: artile#1. It adds the verified SenseVoice registry/family path and CJK locale routing, with 102 huddle tests, Clippy, rustfmt, and real zh/en/ja/ko/yue CPU inference already green. Merging that collaboration PR would update this contributor branch directly and let the reporter test the actual Korean fix without opening a competing upstream PR.

Berenger-Wooclap · 2026-08-02T19:23:10Z

Data point from a French self-hosted deployment, plus a Whisper comparison that may be useful for the registry design.

Same symptom as the Korean report in #2478, in French: "Tu me reçois Smart ? Est-ce que ça marche ?" comes back as "do you feelar Miss Mart? Is it walking".

I benchmarked the currently bundled model against this PR's parakeet-v3, against Whisper, and against one multilingual NeMo CTC model, on four French sentences. sherpa-onnx 1.13.4 (the version Buzz vendors), 16 kHz mono, 2 threads, offline decoding, Apple Silicon. All five models received byte-identical audio.

model	family	exact transcripts	RTF	download
`parakeet-tdt-ctc-110m-en` (current)	NeMo CTC	0/4	0.015	~126 MiB (weights)
`nemo-fast-conformer-ctc-…-20k`	NeMo CTC	1/4	0.023	414 MiB
`parakeet-tdt-0.6b-v3` (this PR)	transducer	2/4	0.049	465 MiB
`whisper-small` int8, `language=fr`	whisper	1/4	0.289	609 MiB
`whisper-medium` int8, `language=fr`	whisper	2/4	0.867	1841 MiB

"Exact" is strict: character-for-character, accents and punctuation included.

The current model is unusable on all four. For "Peux-tu vérifier les échecs du DAG Airflow de cette nuit et me dire si c'est une erreur transitoire ?" it returns:

feel like the dager flow the secondly there is second from itall

parakeet-v3 returns two of the four character-exact, and its only errors are English proper nouns and jargon ("Smart" → "marr", "DAG Airflow" → "daggerflow"):

Le canal est archivé, donc l'agent ne peut pas répondre dans le fil.

On Whisper. It came up as an alternative, so I measured it rather than guessing. parakeet-v3 matches whisper-medium on accuracy while being ~18× faster and ~4× smaller. At RTF 0.867, whisper-medium is close to real time, which is a poor fit for the short VAD-segmented utterances a huddle produces — that latency lands on every turn. Whisper also needs the language declared up front, whereas v3 does automatic language ID, which matters for the code-switching case in #3720. So for European languages this PR's choice looks like the right one.

One option worth noting for the registry. sherpa-onnx-nemo-fast-conformer-ctc-be-de-en-es-fr-hr-it-pl-ru-uk-20k is multilingual and NeMo CTC, so it loads through the existing single-family code path with no plumbing at all. It is meaningfully worse than v3 (1/4 vs 2/4, and it drops words), so it is not a substitute for this PR — but it may be useful as a low-risk entry, or for anyone who needs a stopgap before this lands.

Caveats, stated plainly. This is macOS say synthetic speech, not live mic input, so treat absolute numbers as indicative; the relative contrast on identical audio is the point. Four sentences is enough to separate "garbage" from "exact", not to publish a WER. Voice quality mattered more than I expected: two low-quality synthetic voices produced garbage from every model including v3, and I initially misread that as a v3 weakness — everything above was re-run on a single clean voice.

On the Korean acceptance criterion in #2478: this PR does not cover it, but it is what makes covering it possible, and it already fixes the user-facing bug for the 25 languages it does cover. Since #3349 (closed) showed the sherpa-onnx Whisper wiring works, a whisper family entry also looks like a natural later addition reusing this PR's per-family path rather than replacing the default.

Happy to re-run any of this, test a CJK model if one is added, or provide the raw outputs.

LauraGPT · 2026-08-03T14:31:55Z

I have carried this work forward in #4534 on the current main, preserving Taras as the author of the registry commit and retaining both DCO sign-offs. The new PR includes the completed SenseVoice path from artile/buzz#1, so ko/zh/ja/yue are covered and the #2478 reporter’s Korean acceptance criterion is met. Current validation is 146 huddle tests, Clippy with -D warnings, and real CPU decoding of all five official SenseVoice language samples. Maintainers can review #4534 as the complete superseding branch; no action is needed on the collaboration PR.

artile requested a review from a team as a code owner July 23, 2026 14:15

This was referenced Jul 23, 2026

feat(huddle): huddle-transcript signed event kind + SDK builder (#2513) #2522

Open

Huddle transcription is English-only: non-English speech (e.g. Korean) produces garbled transcripts #2478

Open

artile force-pushed the feat/multilingual-huddle-stt branch from d6c3482 to 9236cb8 Compare July 23, 2026 14:32

LauraGPT reviewed Jul 23, 2026

View reviewed changes

LauraGPT mentioned this pull request Jul 23, 2026

feat(huddle): add SenseVoice for CJK locales artile/buzz#1

Closed

LauraGPT mentioned this pull request Aug 3, 2026

feat(huddle): add selectable multilingual STT with SenseVoice #4534

Open

Navigation Menu

Uh oh!

feat(huddle): make STT model selectable + add multilingual Parakeet v3 (#2478) - #2520

feat(huddle): make STT model selectable + add multilingual Parakeet v3 (#2478)#2520
artile wants to merge 1 commit into
block:mainfrom
artile:feat/multilingual-huddle-stt

artile commented Jul 23, 2026

Uh oh!

LauraGPT left a comment

Uh oh!

LauraGPT Jul 23, 2026

Uh oh!

LauraGPT Jul 23, 2026

Uh oh!

LauraGPT commented Jul 23, 2026

Uh oh!

LauraGPT commented Jul 26, 2026

Uh oh!

Berenger-Wooclap commented Aug 2, 2026

Uh oh!

LauraGPT commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

id	model	family	languages
`parakeet-en` (default)	Parakeet TDT-CTC 110M	NeMo CTC	English
`parakeet-v3`	Parakeet TDT 0.6B v3	transducer	25 European (auto-detected)

Uh oh!

Conversation

artile commented Jul 23, 2026

Summary

What changed

Model choice — why Parakeet v3, and the honest coverage note

Testing

Notes

Uh oh!

LauraGPT left a comment

Choose a reason for hiding this comment

Uh oh!

LauraGPT Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

LauraGPT Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

LauraGPT commented Jul 23, 2026

Uh oh!

LauraGPT commented Jul 26, 2026

Uh oh!

Berenger-Wooclap commented Aug 2, 2026

Uh oh!

LauraGPT commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants