From 0773660f1a5cda61055b6c12198fb568b9a0d599 Mon Sep 17 00:00:00 2001 From: Jhin Lee Date: Wed, 23 Sep 2026 18:13:15 -0400 Subject: [PATCH 1/5] docs: match speech-to-text docs to the validated scope State where the STT pack has passed and which native targets and backends have no STT run, document the edge-fixture outcomes and the cancel/memory budgets, narrow MP3/FLAC and five-minute recordings to what real-model checks cover, fix the LiteRT snippet that failed analysis, restore the orphaned chat-app README sentence, and add the missing [#594](https://github.com/leehack/llamadart/pull/594) changelog entry. Part of [#325](https://github.com/leehack/llamadart/issues/325). --- CHANGELOG.md | 8 ++- doc/cross_platform_validation.md | 7 +- example/chat_app/README.md | 16 +++-- lib/src/core/speech/speech_platform_stub.dart | 2 +- tool/testing/test_matrix.dart | 4 +- website/docs/changelog/recent-releases.md | 8 ++- website/docs/examples/chat-app.md | 9 ++- website/docs/guides/speech-to-text.md | 72 +++++++++++++++---- website/docs/platforms/support-matrix.md | 5 +- 9 files changed, 102 insertions(+), 29 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a5ec06e53..5a8acf5f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,8 +25,14 @@ concatenation crossing the 30-second preprocessing boundary, the last three derived from the locked `jfk.wav`. Every GGUF STT pack run executes the four checks and each one gates `functional_pass`; the LiteRT-ASR and TTS packs - pass no edge fixtures and keep their existing check count + pass no edge fixtures ([#325](https://github.com/leehack/llamadart/issues/325)). +- Raise every `stt`, `tts` and `litert-asr` speech validation pack run from 8 + to 15 lifecycle checks: an immediate cancel, three + cancel/dispose/load/generate cycles, and bounds that fail the run when a + cancellation takes over 500 ms to end its task or the peak resident set + exceeds 1.10x the one sampled after the first generation + ([#594](https://github.com/leehack/llamadart/pull/594)). - Force greedy `topK: 1` for zero-temperature LiteRT-LM Web generation, matching the native clamp ([#548](https://github.com/leehack/llamadart/issues/548)). diff --git a/doc/cross_platform_validation.md b/doc/cross_platform_validation.md index 51f2cd80c..5f205e3eb 100644 --- a/doc/cross_platform_validation.md +++ b/doc/cross_platform_validation.md @@ -993,7 +993,12 @@ had not seen the task finish when it cancelled; it cannot show that the generation had begun. Exceeding any budget fails the run; if resident memory cannot be sampled, the memory bound records `SKIP` with a reason, and that is the only check a passing run may leave unmeasured. -GGUF STT additionally compares file and bytes inputs. +GGUF STT additionally compares file and bytes inputs and runs four generated +edge fixtures: digital silence must fail with the typed empty-transcript +`LlamaSpeechException`, a truncated RIFF must yield an inexact non-empty +transcript or `LlamaAudioFormatException`, a 44.1 kHz stereo copy of +`jfk.wav` must yield the reference, and three concatenated copies (33 s) must +yield it three times. TTS rejects silent, nonfinite or truncated output; playability is not a listening-quality assertion. Its first playable audio is the final buffer, never a progress callback. The voice report preserves the diff --git a/example/chat_app/README.md b/example/chat_app/README.md index 205dadbc5..44e717e84 100644 --- a/example/chat_app/README.md +++ b/example/chat_app/README.md @@ -11,9 +11,10 @@ A Flutter chat application demonstrating real-world usage of llamadart with UI. - 📋 **Clipboard attachments**: Paste screenshots or copied image/audio files with `Cmd/Ctrl+V`, or use **Paste attachment** from the attachment menu on touch devices. Plain-text paste continues to work normally. -- 🎙️ **Whole-file transcription**: Compatible native GGUF ASR models can - transcribe a selected file or capture a foreground microphone recording, - then transcribe it after **Stop & transcribe**. +- 🎙️ **Whole-file transcription**: Compatible GGUF ASR models, on native and + on Web with bridge assets `v0.1.30+`, can transcribe a selected file or + capture a foreground microphone recording, then transcribe it after + **Stop & transcribe**. - 📝 **Live dictation**: Native chat models, including generic audio-chat models, can use a separately installed LiteRT sidecar to show confirmed and pending English text while the user speaks, then place the final text in the @@ -119,7 +120,11 @@ flutter test --run-skipped -t local-only \ for up to five minutes; **Stop & transcribe** finalizes it, runs whole-file STT, and deletes the native file or revokes the browser blob. Capture is foreground-only and cancelling discards the temporary - recording. This Qwen path remains whole-file rather than live. For native chat + recording. Real-model checks cover only WAV input of at most 33 seconds. + A long recording can fill the preset's 4,096-token context; transcription + then completes with a truncated transcript and no error + ([#636](https://github.com/leehack/llamadart/issues/636)). + This Qwen path remains whole-file rather than live. For native chat models, including generic audio-chat models, the composer can separately install a live-dictation model/tokenizer and expose **Live transcription** on Android, iOS, macOS, and Windows. Moonshine Tiny is the recommended @@ -159,6 +164,7 @@ flutter test --run-skipped -t local-only \ supported. It remains hidden on Linux because the recorder plugin can report startup before its required external tools are ready; selected-file transcription still works there. + - The complete Qwen3-ASR model/projector, microphone, and final-transcript flow has passed on a physical Pixel using CPU inference and in the iOS Simulator. This is not yet physical-iPhone or Windows validation. - The native Gemma 4 E2B presets expose **Ask with voice** when either the @@ -167,7 +173,7 @@ flutter test --run-skipped -t local-only \ 30 seconds; **Stop & ask** sends the encoded WAV bytes through normal multimodal chat and asks the model to answer the spoken request. This is not `SpeechToTextEngine`: it has no transcript, timestamp, confidence, or live - partial-text contract. Qwen3-ASR continues to use the separate five-minute + partial-text contract. Qwen3-ASR continues to use the separate **Stop & transcribe** flow, and takes precedence for models declared as ASR profiles. - The voice-question UI is code-supported on Android, iOS, macOS, and diff --git a/lib/src/core/speech/speech_platform_stub.dart b/lib/src/core/speech/speech_platform_stub.dart index 7b6d57a5d..ef73712d0 100644 --- a/lib/src/core/speech/speech_platform_stub.dart +++ b/lib/src/core/speech/speech_platform_stub.dart @@ -13,7 +13,7 @@ bool get speechToTextSupportsFileInput => true; /// Whether byte inputs require explicit encoding metadata on this platform. bool get speechToTextRequiresEncodedAudioFormat => false; -/// Encoded formats validated for prompt-adapted speech input. +/// Encoded formats accepted for prompt-adapted speech input. Set get speechToTextEncodedAudioFormats => const { 'wav', 'mp3', diff --git a/tool/testing/test_matrix.dart b/tool/testing/test_matrix.dart index d602ee66e..b5c7cdc25 100644 --- a/tool/testing/test_matrix.dart +++ b/tool/testing/test_matrix.dart @@ -71,7 +71,9 @@ const List testMatrixRows = [ tier: 'platform', mode: 'opt-in; local model-backed', covers: - 'Locked Qwen3-ASR/Qwen3-TTS and dedicated LiteRT CPU ASR, cancellation, reload, typed guards, WER and audio timing', + 'Locked Qwen3-ASR/Qwen3-TTS and dedicated LiteRT CPU ASR, cancellation, ' + 'reload, typed guards, WER, audio timing, Qwen3-ASR edge fixtures, and ' + 'cancel-latency and peak-memory bounds', command: 'dart run tool/testing/run_local_e2e.dart --scenario validation-speech-stt; ' 'dart run tool/testing/run_local_e2e.dart --scenario validation-speech-tts; ' diff --git a/website/docs/changelog/recent-releases.md b/website/docs/changelog/recent-releases.md index 127352928..14e5e6c64 100644 --- a/website/docs/changelog/recent-releases.md +++ b/website/docs/changelog/recent-releases.md @@ -33,8 +33,14 @@ For canonical full release notes, use: concatenation crossing the 30-second preprocessing boundary, the last three derived from the locked `jfk.wav`. Every GGUF STT pack run executes the four checks and each one gates `functional_pass`; the LiteRT-ASR and TTS packs - pass no edge fixtures and keep their existing check count + pass no edge fixtures ([#325](https://github.com/leehack/llamadart/issues/325)). +- Raise every `stt`, `tts` and `litert-asr` speech validation pack run from 8 + to 15 lifecycle checks: an immediate cancel, three + cancel/dispose/load/generate cycles, and bounds that fail the run when a + cancellation takes over 500 ms to end its task or the peak resident set + exceeds 1.10x the one sampled after the first generation + ([#594](https://github.com/leehack/llamadart/pull/594)). - Force greedy `topK: 1` for zero-temperature LiteRT-LM Web generation, matching the native clamp ([#548](https://github.com/leehack/llamadart/issues/548)). diff --git a/website/docs/examples/chat-app.md b/website/docs/examples/chat-app.md index fd591d634..0a796e476 100644 --- a/website/docs/examples/chat-app.md +++ b/website/docs/examples/chat-app.md @@ -74,7 +74,11 @@ flutter test current LiteRT-LM chat bundles. Microphone capture is enabled on Android, iOS, macOS, Windows, and supported secure browser origins; Linux capture remains disabled pending a safe external-recorder preflight, while - selected-file transcription remains available there. + selected-file transcription remains available there. Recordings stop at five + minutes, but real-model checks cover only WAV input of at most 33 seconds. A + long recording can fill the preset's 4,096-token context; transcription then + completes with a truncated transcript and no error + ([#636](https://github.com/leehack/llamadart/issues/636)). - Experimental live English dictation for native chat models, including generic audio-chat models, through independently installed, checksum-pinned LiteRT sidecars. Moonshine Tiny is the recommended 54 MB default; Parakeet @@ -95,8 +99,7 @@ flutter test bytes through ordinary multimodal chat so the model can answer the spoken request. It does not use `SpeechToTextEngine` and provides no transcript, timestamp, confidence, or live-partial contract. Qwen3-ASR keeps the separate - five-minute **Stop & transcribe** workflow and takes precedence for ASR - profiles. + **Stop & transcribe** workflow and takes precedence for ASR profiles. - A dedicated cross-platform Qwen3-TTS mode backed by `TextToSpeechEngine`. Type an utterance, optionally choose a language and select or record speaker-reference audio, then cancel synthesis, automatically play the diff --git a/website/docs/guides/speech-to-text.md b/website/docs/guides/speech-to-text.md index 95ffe18e9..52ae3867a 100644 --- a/website/docs/guides/speech-to-text.md +++ b/website/docs/guides/speech-to-text.md @@ -20,7 +20,7 @@ multimodal models. | Runtime | Generic audio-input chat | Typed `SpeechToTextEngine` | Text to speech | | --- | --- | --- | --- | -| Native llama.cpp / GGUF | Model + projector dependent | Experimental Qwen3-ASR adapter; complete WAV/MP3/FLAC file or bytes, final transcript only | Experimental Qwen3-TTS adapter; see [Text to Speech](./text-to-speech) | +| Native llama.cpp / GGUF | Model + projector dependent | Experimental Qwen3-ASR adapter; complete WAV/MP3/FLAC file or bytes (real-model validation covers WAV only), final transcript only | Experimental Qwen3-TTS adapter; see [Text to Speech](./text-to-speech) | | WebGPU / GGUF | Bridge + model dependent | Experimental Qwen3-ASR adapter with bridge assets `v0.1.30+`; complete WAV bytes, final transcript only | Unsupported | | Native LiteRT-LM | Separate `.litertlm` audio chat remains bundle dependent | Experimental dedicated CPU ASR through `SpeechToTextEngine.liteRtLm`; mono 16 kHz float PCM, partial/final text, streaming input | Unsupported | | LiteRT-LM Web | Unsupported | Unsupported | Unsupported | @@ -51,7 +51,7 @@ final recognizer = SpeechToTextEngine.liteRtLm( final capabilities = await recognizer.capabilities; if (!capabilities.isSupported) { - throw StateError(capabilities.unsupportedReason); + throw StateError(capabilities.unsupportedReason!); } final session = await recognizer.startStream(); @@ -69,6 +69,7 @@ for (final chunk in mono16KhzFloatPcmChunks) { await session.finish(); final completion = await session.done; await events.cancel(); +print(completion.state); ``` The input contract is mono 16 kHz normalized `Float32List` PCM. The public @@ -152,9 +153,10 @@ preflight fails before a task can start. After startup, `events` is a single-subscription stream: runtime failure is emitted as a stream error and the same terminal condition is available through `task.done`. -Native llama.cpp currently decodes WAV, MP3, and FLAC file or byte inputs. Raw -PCM remains unsupported for that prompt adapter because projector sample rates -are model-specific. Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a +Native llama.cpp accepts WAV, MP3, and FLAC file or byte inputs. Only WAV is +validated with a real model; MP3 and FLAC tests check only that the adapter +accepts the formats, and no test decodes them. Raw PCM remains unsupported for +that prompt adapter because projector sample rates are model-specific. Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a complete mono 16 kHz float buffer, or the incremental session shown above. WebGPU accepts encoded WAV bytes only. Browser file pickers must read the @@ -236,9 +238,10 @@ LiteRT-LM chat bundles. Live dictation uses the native LiteRT-LM streaming STT API with an app-managed sidecar; it is not a capability of the selected chat bundle and remains unavailable on Web. ASR microphone recordings are capped at five minutes, cancelled when the app is -backgrounded, and deleted on native or revoked on Web after transcription. This -remains a whole-file workflow: it does not produce live partial transcripts -while the user speaks. The recorder requests 16 kHz mono WAV, but hardware or +backgrounded, and deleted on native or revoked on Web after transcription. +The repository's real-model checks cover at most 33 seconds of audio; see +[Known limits](#known-limits). This remains a whole-file workflow: it does not +produce live partial transcripts while the user speaks. The recorder requests 16 kHz mono WAV, but hardware or the browser may choose another valid sample rate; the downstream decoder reads the WAV metadata. The live sidecar path instead requests PCM16 mono 16 kHz streaming, preserves @@ -278,6 +281,38 @@ native and Web. Its immutable artifact revision, byte sizes, and SHA-256 digests are pinned. Native downloads verify both files before selection; Web uses origin-scoped Cache Storage and the pinned source metadata. +## Validated behavior + +The `validation-speech-stt` pack +(`dart run tool/testing/run_local_e2e.dart --scenario validation-speech-stt`) +runs the checksum-locked Qwen3-ASR 0.6B Q8_0 model and projector on `jfk.wav`, +an 11-second English WAV, as a file and as bytes. It compares transcripts by +word error rate, ignoring case and punctuation. It also sends four generated +WAV inputs as bytes: + +| Input | Required outcome | +| --- | --- | +| 3 s of digital silence | `LlamaSpeechException` with the message `Speech recognition produced an empty transcript.` | +| The first 20,044 bytes of `jfk.wav`, whose RIFF header declares more audio than the bytes carry | A non-empty transcript that is not the reference, or `LlamaAudioFormatException`. Every recorded run returned a short transcript and no error. | +| `jfk.wav` resampled to 44.1 kHz stereo | The reference transcript | +| `jfk.wav` three times (33 s), crossing the projector's 30-second audio chunk | The reference three times | + +Each run then repeats three cancel/dispose/load/generate cycles. It fails if +either budget is exceeded: + +- **Cancellation, 500 ms**: from `cancel()` to the task's terminal state, for + every cancel issued as soon as `transcribe` returns and every cancel issued + after half the duration of the most recent completed generation. This bounds + when the task ends for its caller, not when native work stops. +- **Memory, 1.10x**: the largest whole-process resident set sampled after the + checks that follow the first generation, as a multiple of the one sampled + right after that generation. + +The pack has passed on macOS arm64 with CPU and with Metal on native +`v0.4.1-1`, and on Linux x64 CPU (AMD EPYC) with an earlier llamadart tree on +native `v0.4.1`. The Metal runs report the Metal backend; the pack does not +verify GPU execution. + ## Known limits - Qwen3-ASR may emit a leading `language English` marker. llamadart @@ -287,12 +322,21 @@ uses origin-scoped Cache Storage and the pinned source metadata. diarization. Incremental audio and partial text are LiteRT-LM-only. - Inference backend correctness and performance remain device dependent; establish a CPU baseline before claiming GPU support for a deployment. -- The local real-model smoke has passed on macOS arm64 CPU, and a separate - chat-app/manual smoke has passed on Metal. The full chat-app - model/projector, microphone, and final-transcript flow has also passed on a - physical Pixel using CPU inference and in the iOS Simulator. A physical - iPhone and Windows remain unverified; Linux keeps selected-file STT but not - microphone capture. Web requires `v0.1.30+`, a browser with enough memory for +- The repository's real-model checks use WAV input of at most 33 s; none uses + MP3 or FLAC. Longer input is unvalidated. The audio prompt grows with + duration (3,890 tokens for 297 s in + [#636](https://github.com/leehack/llamadart/issues/636)). If it fits in + `contextSize` but leaves too little room for the transcript, the task + completes with a truncated transcript and no error. +- Beyond [Validated behavior](#validated-behavior), chat-app microphone + transcription on CPU has passed on a physical Pixel and in the iOS Simulator + ([#328](https://github.com/leehack/llamadart/pull/328)), and CPU file + transcription plus chat-app microphone transcription have passed on a + physical iPad ([#462](https://github.com/leehack/llamadart/pull/462)). No STT + run covers Android x64, Linux arm64, Windows arm64 or x64, macOS x86_64, or a + physical iPhone, nor the Vulkan, CUDA, HIP, OpenCL, or BLAS backends. Linux + keeps selected-file STT but not microphone capture. +- Web requires `v0.1.30+`, a browser with enough memory for the roughly 1.02 GB model/projector pair, and the targeted `web-speech-to-text-smoke` validation row. That row verifies both browser file selection and Chromium fake-device microphone capture with the same WAV diff --git a/website/docs/platforms/support-matrix.md b/website/docs/platforms/support-matrix.md index 0c307c9ab..0f1610ceb 100644 --- a/website/docs/platforms/support-matrix.md +++ b/website/docs/platforms/support-matrix.md @@ -19,8 +19,9 @@ availability below is for the pinned/default artifacts. Speech support is narrower than general runtime availability. llama.cpp/GGUF has an experimental whole-file `SpeechToTextEngine` adapter when the caller explicitly selects the Qwen3-ASR profile and the loaded projector reports audio -capability. Native accepts WAV, MP3, and FLAC file or byte inputs. WebGPU bridge -assets `v0.1.30+` opt into the validated Qwen3-ASR path with WAV bytes only; +capability. Native accepts WAV, MP3, and FLAC file or byte inputs; real-model +validation covers WAV only. WebGPU bridge assets `v0.1.30+` opt into the +validated Qwen3-ASR path with WAV bytes only; older or custom runtimes stay unsupported unless the host explicitly declares the capability. Native llama.cpp and WebGPU bridge assets `v0.1.33+` also expose experimental Qwen3-TTS synthesis through the separate typed From 35074e59254e1d16bd8a0df244d3edb187a91b05 Mon Sep 17 00:00:00 2001 From: Jhin Lee Date: Wed, 23 Sep 2026 18:20:49 -0400 Subject: [PATCH 2/5] docs: scope the MP3/FLAC test note to native --- website/docs/guides/speech-to-text.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/docs/guides/speech-to-text.md b/website/docs/guides/speech-to-text.md index 52ae3867a..1021c90cf 100644 --- a/website/docs/guides/speech-to-text.md +++ b/website/docs/guides/speech-to-text.md @@ -154,8 +154,8 @@ single-subscription stream: runtime failure is emitted as a stream error and the same terminal condition is available through `task.done`. Native llama.cpp accepts WAV, MP3, and FLAC file or byte inputs. Only WAV is -validated with a real model; MP3 and FLAC tests check only that the adapter -accepts the formats, and no test decodes them. Raw PCM remains unsupported for +validated with a real model; native tests check only that the adapter accepts +MP3 and FLAC, and no test decodes them. Raw PCM remains unsupported for that prompt adapter because projector sample rates are model-specific. Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a complete mono 16 kHz float buffer, or the incremental session shown above. From 2bd6c969efcfd678b393d14eaefa12ac03ef93c1 Mon Sep 17 00:00:00 2001 From: Jhin Lee Date: Wed, 23 Sep 2026 18:51:59 -0400 Subject: [PATCH 3/5] docs: drop the false 30-second Qwen3-ASR chunk claim --- CHANGELOG.md | 7 +++---- packages/llamadart_validation/assets/speech/README.md | 8 ++++---- .../lib/src/speech_edge_fixtures.dart | 2 +- website/docs/changelog/recent-releases.md | 7 +++---- website/docs/guides/speech-to-text.md | 2 +- 5 files changed, 12 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a8acf5f0..d287dfa45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,10 +22,9 @@ - Extend the GGUF speech-to-text validation pack with four synthetic edge fixtures built in-process, so no extra audio is stored: generated digital silence, plus a truncated RIFF, a stereo 44.1 kHz re-encode and a 33-second - concatenation crossing the 30-second preprocessing boundary, the last three - derived from the locked `jfk.wav`. Every GGUF STT pack run executes the four - checks and each one gates `functional_pass`; the LiteRT-ASR and TTS packs - pass no edge fixtures + concatenation, the last three derived from the locked `jfk.wav`. Every GGUF + STT pack run executes the four checks and each one gates `functional_pass`; + the LiteRT-ASR and TTS packs pass no edge fixtures ([#325](https://github.com/leehack/llamadart/issues/325)). - Raise every `stt`, `tts` and `litert-asr` speech validation pack run from 8 to 15 lifecycle checks: an immediate cancel, three diff --git a/packages/llamadart_validation/assets/speech/README.md b/packages/llamadart_validation/assets/speech/README.md index 13f17026b..d2502acf0 100644 --- a/packages/llamadart_validation/assets/speech/README.md +++ b/packages/llamadart_validation/assets/speech/README.md @@ -17,10 +17,10 @@ reference or threshold to match the output of a failing run. The `stt` pack also builds four edge fixtures in process, so no extra audio is stored: digital silence, plus three derived from `jfk.wav` — a truncated RIFF whose header declares more audio than the bytes carry, a stereo 44.1 kHz -re-encode, and a 33-second concatenation crossing the projector's 30-second -`audio_chunk_len`. Their required outcomes are measured behavior, not -aspiration, except that the truncated RIFF may equally be rejected with a typed -`LlamaAudioFormatException` should decoding start validating the header. +re-encode, and a 33-second concatenation. Their required outcomes are measured +behavior, not aspiration, except that the truncated RIFF may equally be +rejected with a typed `LlamaAudioFormatException` should decoding start +validating the header. `speech-results.json` records each generated fixture's SHA-256 under `edge_fixtures`, keyed by the same id as its check. diff --git a/packages/llamadart_validation/lib/src/speech_edge_fixtures.dart b/packages/llamadart_validation/lib/src/speech_edge_fixtures.dart index 38cb94f3f..61143c1ca 100644 --- a/packages/llamadart_validation/lib/src/speech_edge_fixtures.dart +++ b/packages/llamadart_validation/lib/src/speech_edge_fixtures.dart @@ -129,7 +129,7 @@ List buildSpeechEdgeFixtures(Uint8List fixture) { seconds: concatenated.length / 16000, contract: SpeechEdgeContract.repeatedReference, referenceRepeats: repeats, - rationale: 'Audio crossing the 30-second preprocessing boundary.', + rationale: 'Audio over 30 seconds must be transcribed in full.', ), ]; for (final entry in fixtures) { diff --git a/website/docs/changelog/recent-releases.md b/website/docs/changelog/recent-releases.md index 14e5e6c64..fc7bff3f8 100644 --- a/website/docs/changelog/recent-releases.md +++ b/website/docs/changelog/recent-releases.md @@ -30,10 +30,9 @@ For canonical full release notes, use: - Extend the GGUF speech-to-text validation pack with four synthetic edge fixtures built in-process, so no extra audio is stored: generated digital silence, plus a truncated RIFF, a stereo 44.1 kHz re-encode and a 33-second - concatenation crossing the 30-second preprocessing boundary, the last three - derived from the locked `jfk.wav`. Every GGUF STT pack run executes the four - checks and each one gates `functional_pass`; the LiteRT-ASR and TTS packs - pass no edge fixtures + concatenation, the last three derived from the locked `jfk.wav`. Every GGUF + STT pack run executes the four checks and each one gates `functional_pass`; + the LiteRT-ASR and TTS packs pass no edge fixtures ([#325](https://github.com/leehack/llamadart/issues/325)). - Raise every `stt`, `tts` and `litert-asr` speech validation pack run from 8 to 15 lifecycle checks: an immediate cancel, three diff --git a/website/docs/guides/speech-to-text.md b/website/docs/guides/speech-to-text.md index 1021c90cf..c8f416f85 100644 --- a/website/docs/guides/speech-to-text.md +++ b/website/docs/guides/speech-to-text.md @@ -295,7 +295,7 @@ WAV inputs as bytes: | 3 s of digital silence | `LlamaSpeechException` with the message `Speech recognition produced an empty transcript.` | | The first 20,044 bytes of `jfk.wav`, whose RIFF header declares more audio than the bytes carry | A non-empty transcript that is not the reference, or `LlamaAudioFormatException`. Every recorded run returned a short transcript and no error. | | `jfk.wav` resampled to 44.1 kHz stereo | The reference transcript | -| `jfk.wav` three times (33 s), crossing the projector's 30-second audio chunk | The reference three times | +| `jfk.wav` three times (33 s) | The reference three times | Each run then repeats three cancel/dispose/load/generate cycles. It fails if either budget is exceeded: From e6849aa81faa1228e907033b4a2c4e4e93cfb3f6 Mon Sep 17 00:00:00 2001 From: Jhin Lee Date: Wed, 23 Sep 2026 18:51:59 -0400 Subject: [PATCH 4/5] docs: scope speech limits to what was measured --- example/chat_app/README.md | 5 +++-- .../lib/src/speech_runner.dart | 3 ++- website/docs/examples/chat-app.md | 8 ++++---- website/docs/guides/speech-to-text.md | 18 +++++++++--------- 4 files changed, 18 insertions(+), 16 deletions(-) diff --git a/example/chat_app/README.md b/example/chat_app/README.md index 44e717e84..bb14a09ba 100644 --- a/example/chat_app/README.md +++ b/example/chat_app/README.md @@ -121,8 +121,9 @@ flutter test --run-skipped -t local-only \ STT, and deletes the native file or revokes the browser blob. Capture is foreground-only and cancelling discards the temporary recording. Real-model checks cover only WAV input of at most 33 seconds. - A long recording can fill the preset's 4,096-token context; transcription - then completes with a truncated transcript and no error + On native macOS arm64 CPU, 297 seconds of audio filled a 4,096-token + context, the preset's size, and transcription completed with a truncated + transcript and no error ([#636](https://github.com/leehack/llamadart/issues/636)). This Qwen path remains whole-file rather than live. For native chat models, including generic audio-chat models, the composer can separately diff --git a/packages/llamadart_validation/lib/src/speech_runner.dart b/packages/llamadart_validation/lib/src/speech_runner.dart index b3cc3dc50..73065f763 100644 --- a/packages/llamadart_validation/lib/src/speech_runner.dart +++ b/packages/llamadart_validation/lib/src/speech_runner.dart @@ -9,7 +9,8 @@ import 'runner.dart' show redactDiagnostic; import 'speech_edge_fixtures.dart'; /// Word edit distance divided by reference words; insertions can exceed 1.0. -/// Normalization ignores case and ASCII punctuation, preserving Unicode words. +/// Normalization lowercases, replaces `.,!?:;"—–` with spaces, then splits on +/// whitespace. double speechWordErrorRate(String reference, String actual) { List words(String text) => text .toLowerCase() diff --git a/website/docs/examples/chat-app.md b/website/docs/examples/chat-app.md index 0a796e476..2287d12ab 100644 --- a/website/docs/examples/chat-app.md +++ b/website/docs/examples/chat-app.md @@ -75,10 +75,10 @@ flutter test iOS, macOS, Windows, and supported secure browser origins; Linux capture remains disabled pending a safe external-recorder preflight, while selected-file transcription remains available there. Recordings stop at five - minutes, but real-model checks cover only WAV input of at most 33 seconds. A - long recording can fill the preset's 4,096-token context; transcription then - completes with a truncated transcript and no error - ([#636](https://github.com/leehack/llamadart/issues/636)). + minutes, but real-model checks cover only WAV input of at most 33 seconds. On + native macOS arm64 CPU, 297 seconds of audio filled a 4,096-token context, the + preset's size, and transcription completed with a truncated transcript and no + error ([#636](https://github.com/leehack/llamadart/issues/636)). - Experimental live English dictation for native chat models, including generic audio-chat models, through independently installed, checksum-pinned LiteRT sidecars. Moonshine Tiny is the recommended 54 MB default; Parakeet diff --git a/website/docs/guides/speech-to-text.md b/website/docs/guides/speech-to-text.md index c8f416f85..b1530a23f 100644 --- a/website/docs/guides/speech-to-text.md +++ b/website/docs/guides/speech-to-text.md @@ -286,9 +286,9 @@ uses origin-scoped Cache Storage and the pinned source metadata. The `validation-speech-stt` pack (`dart run tool/testing/run_local_e2e.dart --scenario validation-speech-stt`) runs the checksum-locked Qwen3-ASR 0.6B Q8_0 model and projector on `jfk.wav`, -an 11-second English WAV, as a file and as bytes. It compares transcripts by -word error rate, ignoring case and punctuation. It also sends four generated -WAV inputs as bytes: +an 11-second English WAV, as a file and as bytes. It scores each transcript +against the reference by word error rate, after lowercasing both and replacing +`.,!?:;"—–` with spaces. It also sends four generated WAV inputs as bytes: | Input | Required outcome | | --- | --- | @@ -325,17 +325,17 @@ verify GPU execution. - The repository's real-model checks use WAV input of at most 33 s; none uses MP3 or FLAC. Longer input is unvalidated. The audio prompt grows with duration (3,890 tokens for 297 s in - [#636](https://github.com/leehack/llamadart/issues/636)). If it fits in - `contextSize` but leaves too little room for the transcript, the task + [#636](https://github.com/leehack/llamadart/issues/636)). On native, if it + fits in `contextSize` but leaves too little room for the transcript, the task completes with a truncated transcript and no error. - Beyond [Validated behavior](#validated-behavior), chat-app microphone transcription on CPU has passed on a physical Pixel and in the iOS Simulator ([#328](https://github.com/leehack/llamadart/pull/328)), and CPU file transcription plus chat-app microphone transcription have passed on a - physical iPad ([#462](https://github.com/leehack/llamadart/pull/462)). No STT - run covers Android x64, Linux arm64, Windows arm64 or x64, macOS x86_64, or a - physical iPhone, nor the Vulkan, CUDA, HIP, OpenCL, or BLAS backends. Linux - keeps selected-file STT but not microphone capture. + physical iPad ([#462](https://github.com/leehack/llamadart/pull/462)). No + Qwen3-ASR run covers Android x64, Linux arm64, Windows arm64 or x64, macOS + x86_64, or a physical iPhone, nor the Vulkan, CUDA, HIP, OpenCL, or BLAS + backends. Linux keeps selected-file STT but not microphone capture. - Web requires `v0.1.30+`, a browser with enough memory for the roughly 1.02 GB model/projector pair, and the targeted `web-speech-to-text-smoke` validation row. That row verifies both browser From 5022c0b536410862e950394354009d4968e057e2 Mon Sep 17 00:00:00 2001 From: Jhin Lee Date: Wed, 23 Sep 2026 20:31:40 -0400 Subject: [PATCH 5/5] docs: cite the Linux x64 speech pack run at native v0.4.1-1 validation-speech-stt passed 20/20 twice on Linux x64 CPU (AMD EPYC 7B12) at 1584e7d4f with native v0.4.1-1, replacing the earlier native v0.4.1 run in the guide's validated scope. Also rewraps two long guide lines. --- website/docs/guides/speech-to-text.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/website/docs/guides/speech-to-text.md b/website/docs/guides/speech-to-text.md index cf6b1d41a..399bc568f 100644 --- a/website/docs/guides/speech-to-text.md +++ b/website/docs/guides/speech-to-text.md @@ -162,8 +162,9 @@ the same terminal condition is available through `task.done`. Native llama.cpp accepts WAV, MP3, and FLAC file or byte inputs. Only WAV is validated with a real model; native tests check only that the adapter accepts MP3 and FLAC, and no test decodes them. Raw PCM remains unsupported for -that prompt adapter because projector sample rates are model-specific. Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a -complete mono 16 kHz float buffer, or the incremental session shown above. +that prompt adapter because projector sample rates are model-specific. +Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a complete mono 16 kHz +float buffer, or the incremental session shown above. WebGPU accepts encoded WAV bytes only. Browser file pickers must read the selected file into memory and use `SpeechAudioBytesInput`; local filesystem @@ -247,9 +248,9 @@ ASR microphone recordings are capped at five minutes, cancelled when the app is backgrounded, and deleted on native or revoked on Web after transcription. The repository's real-model checks cover at most 33 seconds of audio; see [Known limits](#known-limits). This remains a whole-file workflow: it does not -produce live partial transcripts while the user speaks. The recorder requests 16 kHz mono WAV, but hardware or -the browser may choose another valid sample rate; the downstream decoder reads -the WAV metadata. +produce live partial transcripts while the user speaks. The recorder requests +16 kHz mono WAV, but hardware or the browser may choose another valid sample +rate; the downstream decoder reads the WAV metadata. The live sidecar path instead requests PCM16 mono 16 kHz streaming, preserves samples split across arbitrary byte-chunk boundaries, applies one in-flight worker push at a time, and caps each session at five minutes. It is currently @@ -314,10 +315,9 @@ either budget is exceeded: checks that follow the first generation, as a multiple of the one sampled right after that generation. -The pack has passed on macOS arm64 with CPU and with Metal on native -`v0.4.1-1`, and on Linux x64 CPU (AMD EPYC) with an earlier llamadart tree on -native `v0.4.1`. The Metal runs report the Metal backend; the pack does not -verify GPU execution. +With native `v0.4.1-1`, the pack has passed on macOS arm64 with CPU and with +Metal, and on Linux x64 with CPU (AMD EPYC 7B12). The Metal runs report the +Metal backend; the pack does not verify GPU execution. ## Known limits