Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,16 @@
- Extend the GGUF speech-to-text validation pack with four synthetic edge
fixtures built in-process, so no extra audio is stored: generated digital
silence, plus a truncated RIFF, a stereo 44.1 kHz re-encode and a 33-second
concatenation crossing the 30-second preprocessing boundary, the last three
derived from the locked `jfk.wav`. Every GGUF STT pack run executes the four
checks and each one gates `functional_pass`; the LiteRT-ASR and TTS packs
pass no edge fixtures and keep their existing check count
concatenation, the last three derived from the locked `jfk.wav`. Every GGUF
STT pack run executes the four checks and each one gates `functional_pass`;
the LiteRT-ASR and TTS packs pass no edge fixtures
([#325](https://github.com/leehack/llamadart/issues/325)).
- Raise every `stt`, `tts` and `litert-asr` speech validation pack run from 8
to 15 lifecycle checks: an immediate cancel, three
cancel/dispose/load/generate cycles, and bounds that fail the run when a
cancellation takes over 500 ms to end its task or the peak resident set
exceeds 1.10x the one sampled after the first generation
([#594](https://github.com/leehack/llamadart/pull/594)).
- Force greedy `topK: 1` for zero-temperature LiteRT-LM Web generation, matching
the native clamp
([#548](https://github.com/leehack/llamadart/issues/548)).
Expand Down
7 changes: 6 additions & 1 deletion doc/cross_platform_validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -993,7 +993,12 @@ had not seen the task finish when it cancelled; it cannot show that the
generation had begun. Exceeding any budget fails the run; if resident memory
cannot be sampled, the memory bound records `SKIP` with a reason, and that is
the only check a passing run may leave unmeasured.
GGUF STT additionally compares file and bytes inputs.
GGUF STT additionally compares file and bytes inputs and runs four generated
edge fixtures: digital silence must fail with the typed empty-transcript
`LlamaSpeechException`, a truncated RIFF must yield an inexact non-empty
transcript or `LlamaAudioFormatException`, a 44.1 kHz stereo copy of
`jfk.wav` must yield the reference, and three concatenated copies (33 s) must
yield it three times.
TTS rejects silent, nonfinite or truncated output; playability is not a
listening-quality assertion. Its first playable audio is
the final buffer, never a progress callback. The voice report preserves the
Expand Down
17 changes: 12 additions & 5 deletions example/chat_app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@ A Flutter chat application demonstrating real-world usage of llamadart with UI.
- 📋 **Clipboard attachments**: Paste screenshots or copied image/audio files
with `Cmd/Ctrl+V`, or use **Paste attachment** from the attachment menu on
touch devices. Plain-text paste continues to work normally.
- 🎙️ **Whole-file transcription**: Compatible native GGUF ASR models can
transcribe a selected file or capture a foreground microphone recording,
then transcribe it after **Stop & transcribe**.
- 🎙️ **Whole-file transcription**: Compatible GGUF ASR models, on native and
on Web with bridge assets `v0.1.30+`, can transcribe a selected file or
capture a foreground microphone recording, then transcribe it after
**Stop & transcribe**.
- 📝 **Live dictation**: Native chat models, including generic audio-chat
models, can use a separately installed LiteRT sidecar to show confirmed and
pending English text while the user speaks, then place the final text in the
Expand Down Expand Up @@ -119,7 +120,12 @@ flutter test --run-skipped -t local-only \
for up to five minutes; **Stop & transcribe** finalizes it, runs whole-file
STT, and deletes the native file or revokes the browser blob.
Capture is foreground-only and cancelling discards the temporary
recording. This Qwen path remains whole-file rather than live. For native chat
recording. Real-model checks cover only WAV input of at most 33 seconds.
On native macOS arm64 CPU, 297 seconds of audio filled a 4,096-token
context, the preset's size, and transcription completed with a truncated
transcript and no error
([#636](https://github.com/leehack/llamadart/issues/636)).
This Qwen path remains whole-file rather than live. For native chat
models, including generic audio-chat models, the composer can separately
install a live-dictation model/tokenizer and expose **Live transcription**
on Android, iOS, macOS, and Windows. Moonshine Tiny is the recommended
Expand Down Expand Up @@ -159,6 +165,7 @@ flutter test --run-skipped -t local-only \
supported. It remains hidden on Linux because the recorder plugin can
report startup before its required external tools are ready; selected-file
transcription still works there.
- The complete Qwen3-ASR model/projector, microphone, and final-transcript
flow has passed on a physical Pixel using CPU inference and in the iOS
Simulator. This is not yet physical-iPhone or Windows validation.
- The native Gemma 4 E2B presets expose **Ask with voice** when either the
Expand All @@ -167,7 +174,7 @@ flutter test --run-skipped -t local-only \
30 seconds; **Stop & ask** sends the encoded WAV bytes through normal
multimodal chat and asks the model to answer the spoken request. This is not
`SpeechToTextEngine`: it has no transcript, timestamp, confidence, or live
partial-text contract. Qwen3-ASR continues to use the separate five-minute
partial-text contract. Qwen3-ASR continues to use the separate
**Stop & transcribe** flow, and takes precedence for models declared as
ASR profiles.
- The voice-question UI is code-supported on Android, iOS, macOS, and
Expand Down
2 changes: 1 addition & 1 deletion lib/src/core/speech/speech_platform_stub.dart
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ bool get speechToTextSupportsFileInput => true;
/// Whether byte inputs require explicit encoding metadata on this platform.
bool get speechToTextRequiresEncodedAudioFormat => false;

/// Encoded formats validated for prompt-adapted speech input.
/// Encoded formats accepted for prompt-adapted speech input.
Set<String> get speechToTextEncodedAudioFormats => const <String>{
'wav',
'mp3',
Expand Down
8 changes: 4 additions & 4 deletions packages/llamadart_validation/assets/speech/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ reference or threshold to match the output of a failing run.
The `stt` pack also builds four edge fixtures in process, so no extra audio is
stored: digital silence, plus three derived from `jfk.wav` — a truncated RIFF
whose header declares more audio than the bytes carry, a stereo 44.1 kHz
re-encode, and a 33-second concatenation crossing the projector's 30-second
`audio_chunk_len`. Their required outcomes are measured behavior, not
aspiration, except that the truncated RIFF may equally be rejected with a typed
`LlamaAudioFormatException` should decoding start validating the header.
re-encode, and a 33-second concatenation. Their required outcomes are measured
behavior, not aspiration, except that the truncated RIFF may equally be
rejected with a typed `LlamaAudioFormatException` should decoding start
validating the header.
`speech-results.json` records each generated fixture's SHA-256 under
`edge_fixtures`, keyed by the same id as its check.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ List<SpeechEdgeFixture> buildSpeechEdgeFixtures(Uint8List fixture) {
seconds: concatenated.length / 16000,
contract: SpeechEdgeContract.repeatedReference,
referenceRepeats: repeats,
rationale: 'Audio crossing the 30-second preprocessing boundary.',
rationale: 'Audio over 30 seconds must be transcribed in full.',
),
];
for (final entry in fixtures) {
Expand Down
3 changes: 2 additions & 1 deletion packages/llamadart_validation/lib/src/speech_runner.dart
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ import 'runner.dart' show redactDiagnostic;
import 'speech_edge_fixtures.dart';

/// Word edit distance divided by reference words; insertions can exceed 1.0.
/// Normalization ignores case and ASCII punctuation, preserving Unicode words.
/// Normalization lowercases, replaces `.,!?:;"—–` with spaces, then splits on
/// whitespace.
double speechWordErrorRate(String reference, String actual) {
List<String> words(String text) => text
.toLowerCase()
Expand Down
4 changes: 3 additions & 1 deletion tool/testing/test_matrix.dart
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,9 @@ const List<TestMatrixRow> testMatrixRows = <TestMatrixRow>[
tier: 'platform',
mode: 'opt-in; local model-backed',
covers:
'Locked Qwen3-ASR/Qwen3-TTS and dedicated LiteRT CPU ASR, cancellation, reload, typed guards, WER and audio timing',
'Locked Qwen3-ASR/Qwen3-TTS and dedicated LiteRT CPU ASR, cancellation, '
'reload, typed guards, WER, audio timing, Qwen3-ASR edge fixtures, and '
'cancel-latency and peak-memory bounds',
command:
'dart run tool/testing/run_local_e2e.dart --scenario validation-speech-stt; '
'dart run tool/testing/run_local_e2e.dart --scenario validation-speech-tts; '
Expand Down
13 changes: 9 additions & 4 deletions website/docs/changelog/recent-releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,16 @@ For canonical full release notes, use:
- Extend the GGUF speech-to-text validation pack with four synthetic edge
fixtures built in-process, so no extra audio is stored: generated digital
silence, plus a truncated RIFF, a stereo 44.1 kHz re-encode and a 33-second
concatenation crossing the 30-second preprocessing boundary, the last three
derived from the locked `jfk.wav`. Every GGUF STT pack run executes the four
checks and each one gates `functional_pass`; the LiteRT-ASR and TTS packs
pass no edge fixtures and keep their existing check count
concatenation, the last three derived from the locked `jfk.wav`. Every GGUF
STT pack run executes the four checks and each one gates `functional_pass`;
the LiteRT-ASR and TTS packs pass no edge fixtures
([#325](https://github.com/leehack/llamadart/issues/325)).
- Raise every `stt`, `tts` and `litert-asr` speech validation pack run from 8
to 15 lifecycle checks: an immediate cancel, three
cancel/dispose/load/generate cycles, and bounds that fail the run when a
cancellation takes over 500 ms to end its task or the peak resident set
exceeds 1.10x the one sampled after the first generation
([#594](https://github.com/leehack/llamadart/pull/594)).
- Force greedy `topK: 1` for zero-temperature LiteRT-LM Web generation, matching
the native clamp
([#548](https://github.com/leehack/llamadart/issues/548)).
Expand Down
9 changes: 6 additions & 3 deletions website/docs/examples/chat-app.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,11 @@ flutter test
current LiteRT-LM chat bundles. Microphone capture is enabled on Android,
iOS, macOS, Windows, and supported secure browser origins; Linux capture
remains disabled pending a safe external-recorder preflight, while
selected-file transcription remains available there.
selected-file transcription remains available there. Recordings stop at five
minutes, but real-model checks cover only WAV input of at most 33 seconds. On
native macOS arm64 CPU, 297 seconds of audio filled a 4,096-token context, the
preset's size, and transcription completed with a truncated transcript and no
error ([#636](https://github.com/leehack/llamadart/issues/636)).
- Experimental live English dictation for native chat models, including
generic audio-chat models, through independently installed, checksum-pinned
LiteRT sidecars. Moonshine Tiny is the recommended 54 MB default; Parakeet
Expand All @@ -95,8 +99,7 @@ flutter test
bytes through ordinary multimodal chat so the model can answer the spoken
request. It does not use `SpeechToTextEngine` and provides no transcript,
timestamp, confidence, or live-partial contract. Qwen3-ASR keeps the separate
five-minute **Stop & transcribe** workflow and takes precedence for ASR
profiles.
**Stop & transcribe** workflow and takes precedence for ASR profiles.
- A dedicated cross-platform Qwen3-TTS mode backed by `TextToSpeechEngine`.
Type an utterance, optionally choose a language and select or record
speaker-reference audio, then cancel synthesis, automatically play the
Expand Down
78 changes: 61 additions & 17 deletions website/docs/guides/speech-to-text.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ multimodal models.

| Runtime | Generic audio-input chat | Typed `SpeechToTextEngine` | Text to speech |
| --- | --- | --- | --- |
| Native llama.cpp / GGUF | Model + projector dependent | Experimental Qwen3-ASR adapter; complete WAV/MP3/FLAC file or bytes, final transcript only | Experimental Qwen3-TTS adapter; see [Text to Speech](./text-to-speech) |
| Native llama.cpp / GGUF | Model + projector dependent | Experimental Qwen3-ASR adapter; complete WAV/MP3/FLAC file or bytes (real-model validation covers WAV only), final transcript only | Experimental Qwen3-TTS adapter; see [Text to Speech](./text-to-speech) |
| WebGPU / GGUF | Bridge + model dependent | Experimental Qwen3-ASR adapter with bridge assets `v0.1.30+`; complete WAV bytes, final transcript only | Unsupported |
| Native LiteRT-LM | Separate `.litertlm` audio chat remains bundle dependent | Experimental dedicated CPU ASR through `SpeechToTextEngine.liteRtLm`; mono 16 kHz float PCM, partial/final text, streaming input | Unsupported |
| LiteRT-LM Web | Unsupported | Unsupported | Unsupported |
Expand Down Expand Up @@ -51,7 +51,7 @@ final recognizer = SpeechToTextEngine.liteRtLm(

final capabilities = await recognizer.capabilities;
if (!capabilities.isSupported) {
throw StateError(capabilities.unsupportedReason);
throw StateError(capabilities.unsupportedReason!);
}

final session = await recognizer.startStream();
Expand All @@ -69,6 +69,7 @@ for (final chunk in mono16KhzFloatPcmChunks) {
await session.finish();
final completion = await session.done;
await events.cancel();
print(completion.state);
```

The input contract is mono 16 kHz normalized `Float32List` PCM. The public
Expand Down Expand Up @@ -158,10 +159,12 @@ preflight fails before a task can start. After startup, `events` is a
single-subscription stream: runtime failure is emitted as a stream error and
the same terminal condition is available through `task.done`.

Native llama.cpp currently decodes WAV, MP3, and FLAC file or byte inputs. Raw
PCM remains unsupported for that prompt adapter because projector sample rates
are model-specific. Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a
complete mono 16 kHz float buffer, or the incremental session shown above.
Native llama.cpp accepts WAV, MP3, and FLAC file or byte inputs. Only WAV is
validated with a real model; native tests check only that the adapter accepts
MP3 and FLAC, and no test decodes them. Raw PCM remains unsupported for
that prompt adapter because projector sample rates are model-specific.
Dedicated LiteRT-LM accepts `SpeechAudioPcmInput` for a complete mono 16 kHz
float buffer, or the incremental session shown above.

WebGPU accepts encoded WAV bytes only. Browser file pickers must read the
selected file into memory and use `SpeechAudioBytesInput`; local filesystem
Expand Down Expand Up @@ -242,11 +245,12 @@ LiteRT-LM chat bundles. Live dictation uses the native LiteRT-LM streaming STT
API with an app-managed sidecar; it is not a capability of the selected chat
bundle and remains unavailable on Web.
ASR microphone recordings are capped at five minutes, cancelled when the app is
backgrounded, and deleted on native or revoked on Web after transcription. This
remains a whole-file workflow: it does not produce live partial transcripts
while the user speaks. The recorder requests 16 kHz mono WAV, but hardware or
the browser may choose another valid sample rate; the downstream decoder reads
the WAV metadata.
backgrounded, and deleted on native or revoked on Web after transcription.
The repository's real-model checks cover at most 33 seconds of audio; see
[Known limits](#known-limits). This remains a whole-file workflow: it does not
produce live partial transcripts while the user speaks. The recorder requests
16 kHz mono WAV, but hardware or the browser may choose another valid sample
rate; the downstream decoder reads the WAV metadata.
The live sidecar path instead requests PCM16 mono 16 kHz streaming, preserves
samples split across arbitrary byte-chunk boundaries, applies one in-flight
worker push at a time, and caps each session at five minutes. It is currently
Expand Down Expand Up @@ -284,6 +288,37 @@ native and Web. Its immutable artifact revision, byte sizes, and SHA-256
digests are pinned. Native downloads verify both files before selection; Web
uses origin-scoped Cache Storage and the pinned source metadata.

## Validated behavior

The `validation-speech-stt` pack
(`dart run tool/testing/run_local_e2e.dart --scenario validation-speech-stt`)
runs the checksum-locked Qwen3-ASR 0.6B Q8_0 model and projector on `jfk.wav`,
an 11-second English WAV, as a file and as bytes. It scores each transcript
against the reference by word error rate, after lowercasing both and replacing
`.,!?:;"—–` with spaces. It also sends four generated WAV inputs as bytes:

| Input | Required outcome |
| --- | --- |
| 3 s of digital silence | `LlamaSpeechException` with the message `Speech recognition produced an empty transcript.` |
| The first 20,044 bytes of `jfk.wav`, whose RIFF header declares more audio than the bytes carry | A non-empty transcript that is not the reference, or `LlamaAudioFormatException`. Every recorded run returned a short transcript and no error. |
| `jfk.wav` resampled to 44.1 kHz stereo | The reference transcript |
| `jfk.wav` three times (33 s) | The reference three times |

Each run then repeats three cancel/dispose/load/generate cycles. It fails if
either budget is exceeded:

- **Cancellation, 500 ms**: from `cancel()` to the task's terminal state, for
every cancel issued as soon as `transcribe` returns and every cancel issued
after half the duration of the most recent completed generation. This bounds
when the task ends for its caller, not when native work stops.
- **Memory, 1.10x**: the largest whole-process resident set sampled after the
checks that follow the first generation, as a multiple of the one sampled
right after that generation.

With native `v0.4.1-1`, the pack has passed on macOS arm64 with CPU and with
Metal, and on Linux x64 with CPU (AMD EPYC 7B12). The Metal runs report the
Metal backend; the pack does not verify GPU execution.

## Known limits

- Qwen3-ASR may emit a leading `language English<asr_text>` marker. llamadart
Expand All @@ -293,12 +328,21 @@ uses origin-scoped Cache Storage and the pinned source metadata.
diarization. Incremental audio and partial text are LiteRT-LM-only.
- Inference backend correctness and performance remain device dependent;
establish a CPU baseline before claiming GPU support for a deployment.
- The local real-model smoke has passed on macOS arm64 CPU, and a separate
chat-app/manual smoke has passed on Metal. The full chat-app
model/projector, microphone, and final-transcript flow has also passed on a
physical Pixel using CPU inference and in the iOS Simulator. A physical
iPhone and Windows remain unverified; Linux keeps selected-file STT but not
microphone capture. Web requires `v0.1.30+`, a browser with enough memory for
- The repository's real-model checks use WAV input of at most 33 s; none uses
MP3 or FLAC. Longer input is unvalidated. The audio prompt grows with
duration (3,890 tokens for 297 s in
[#636](https://github.com/leehack/llamadart/issues/636)). On native, if it
fits in `contextSize` but leaves too little room for the transcript, the task
completes with a truncated transcript and no error.
- Beyond [Validated behavior](#validated-behavior), chat-app microphone
transcription on CPU has passed on a physical Pixel and in the iOS Simulator
([#328](https://github.com/leehack/llamadart/pull/328)), and CPU file
transcription plus chat-app microphone transcription have passed on a
physical iPad ([#462](https://github.com/leehack/llamadart/pull/462)). No
Qwen3-ASR run covers Android x64, Linux arm64, Windows arm64 or x64, macOS
x86_64, or a physical iPhone, nor the Vulkan, CUDA, HIP, OpenCL, or BLAS
backends. Linux keeps selected-file STT but not microphone capture.
- Web requires `v0.1.30+`, a browser with enough memory for
the roughly 1.02 GB model/projector pair, and the targeted
`web-speech-to-text-smoke` validation row. That row verifies both browser
file selection and Chromium fake-device microphone capture with the same WAV
Expand Down
Loading
Loading