Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
1eb4c49
feat(text): remove filler words before dictation delivery
softmaxe Nov 11, 2026
ed23502
feat(dictation): skip cleanup without cancelling transcription
softmaxe Nov 11, 2026
2747b4d
feat(audio): add private microphone recording and playback test
softmaxe Nov 11, 2026
1ac1731
feat(audio): filter dictation silence with embedded Silero VAD
softmaxe Nov 11, 2026
de976b6
refactor(ui): give transcription options their own settings section
softmaxe Nov 11, 2026
366d7c6
fix(text): preserve filler boundaries and snippet precedence
softmaxe Nov 11, 2026
b931f4d
fix(history): send original audio when retry finds no speech
softmaxe Nov 11, 2026
823242d
fix(audio): cap finalized microphone test clips at ten seconds
softmaxe Nov 11, 2026
2d2152f
test(pipeline): tolerate scheduling delays when skipping cleanup
softmaxe Nov 11, 2026
b89bf9e
fix(text): resolve punctuation at the seam of removed fillers
softmaxe Nov 11, 2026
1076370
fix(text): keep the text's own commas, casing and code around fillers
softmaxe Nov 11, 2026
9991efc
fix(text): protect CJK-quoted mentions and literals from filler edits
softmaxe Nov 11, 2026
157f48e
chore(dev): add a pill scene that loops one whole dictation
softmaxe Nov 12, 2026
78531a0
fix(ui): keep the pill continuous from recording to delivery
softmaxe Nov 12, 2026
dc153a1
chore(dev): replay measured server timings in the pill flow scene
softmaxe Nov 12, 2026
07bbd7a
fix(ui): pace the polishing label and Skip by how long cleanup runs
softmaxe Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
453 changes: 451 additions & 2 deletions Cargo.lock

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ url = "2"

# Audio
cpal = "0.18"
hound = "3.5"
tract-onnx = { version = "=0.23.10", default-features = false }

# Text
ferrous-opencc = "0.4"
Expand All @@ -80,7 +82,6 @@ block2 = "0.6"
base64 = "0.22"

[dev-dependencies]
hound = "3.5"
tempfile = "3"
ttf-parser = "0.25"
tiny_http = "0.12"
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,18 @@ typing in.
recording (key combinations toggle with one press), or hold to talk. A
floating recording pill shows a live waveform and the elapsed time, with a
cancel button beside it; click the pill to finish or press Escape to cancel.
During polishing, Skip inserts the recognized text immediately with local
corrections and snippets still applied.
If the server fails, the recording is kept so you can resend it.
- **Self-hosted models.** Speech recognition goes to your own
`/audio/transcriptions` endpoint; optional cleanup goes to a
`/chat/completions` endpoint, with an editable prompt and a Prompt Studio to
test it. Test Connection checks each server and lists its models.
- **Skip silence locally.** Embedded Silero VAD trims pauses and non-speech
sound from dictation uploads, with buffers around speech. History and recovery
retain the original recording; Send anyway can bypass a no-speech result once.
- **Fewer hesitation sounds.** Optional local filler-word removal works even
with text cleanup off, while preserving the raw transcript in History.
- **Gets better with use.** A dictionary supplies spelling hints, corrections
you make right after a paste are learned, and Snippets expand short phrases
into longer text. Removing a word or snippet can be undone.
Expand All @@ -45,6 +52,8 @@ typing in.
sets the period beside the one before it; highlights such as active days,
busiest day and time spoken; and a Habits section with your streak calendar
and weekly rhythm. Everything is computed on device.
- **Test your microphone.** Record up to ten seconds and play it back from
Settings > Dictation. The clip stays in memory and is never saved or sent.
- **Fits into macOS.** Menu bar icon, standard app menu, start at login
(macOS 13+), optional media pause while recording, English and Simplified
Chinese UI, and light, dark, Tokyo Night or automatic themes.
Expand Down
8 changes: 8 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,16 @@
- **随处听写。** 可用 Globe/Fn 键、单个修饰键(如右 Command)或任意组合键触发。
单键双击进入免手持录音(组合键按一次即开始或结束),或按住说话。录音时会出现
悬浮条,显示实时波形和已录时长,旁边有取消按钮;点击悬浮条结束录音,按 Esc 取消。
润色时可点击“跳过”直接插入识别结果,本机纠错和片段展开仍会执行。
如果服务器出错,录音会被保留,可以重新发送。
- **自托管模型。** 语音识别请求发送到你自己的 `/audio/transcriptions` 接口;可选的
文本整理请求发送到 `/chat/completions` 接口,提示词可编辑,并可在 Prompt Studio
中测试。“测试连接”可检查每个服务器并列出它的模型。
- **本机过滤静音。** 内置 Silero VAD 在上传听写录音前去掉停顿和非人声,并保留
语音前后的缓冲。历史和恢复使用原始录音;未检测到人声时可选择“仍然发送”,
为这次重发跳过过滤。
- **减少犹豫音。** 可在本机移除“呃”、um、uh 等语气词,无需开启文本整理;
历史记录仍保留服务器返回的原始转录。
- **越用越准。** 词典为识别提供拼写提示;粘贴后立即做出的修改会被自动学习;
片段(Snippets)可以把短语展开为更长的文本。删除词条或片段后可以撤销。
- **历史记录。** 用 Cmd+K 搜索、查看原始转录、用保存的音频重新转录、撤销删除,
Expand All @@ -36,6 +42,8 @@
- **统计。** 最近 7 天、30 天或全部时间的字数、听写次数和每分钟字数,每项都与
上一周期对比;趋势图把当前周期与上一周期并排显示;亮点包括活跃天数、最忙的一天
和说话时长;“习惯”部分展示连续天数日历和每周节奏。全部在本机计算。
- **测试麦克风。** 在“设置 > 听写”录制最长十秒的声音并试听,音频只保留在内存中,
不会保存、转写或发送到服务器。
- **融入 macOS。** 菜单栏图标、标准应用菜单、开机启动(macOS 13+)、可选在录音时暂停媒体播放、
英文和简体中文界面,以及浅色、深色、Tokyo Night 或自动主题。
- **键盘与旁白。** 所有控件都能用 Tab 访问并带有旁白(VoiceOver)名称,列表和图表
Expand Down
45 changes: 45 additions & 0 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,3 +126,48 @@ Do not reset TCC permissions, change system trust, or delete Keychain items duri
- Tab through theme choices, buttons, switches, inputs, and dropdowns. Check focus, hover, selected, disabled, and destructive states. The filled action must retain a clearly contrasting label; switches must remain distinguishable by both thumb position and track color.
- With `WHISPER_USER_DATA_DIR` pointing to a temporary directory, use `WHISPER_DEV_SCENE` in a debug build to inspect `pill:recording`, `pill:processing`, `pill:preparing`, `pill:error`, and `pill:fallback`. Tokyo Night uses the same indigo surface family as the workspace, muted blue voice activity, and distinct error/recovery colors. Light and Dark retain their dark floating HUD.
- Automated tests cover preference serialization, system-appearance resolution, reading contrast, focus indicators, and selected/action text. They complement visual review; they do not prove native interaction or rendering quality.

## Filler-word removal regression check

- In Settings > Speech to text, toggle Remove filler words, restart, and verify the preference persists. Existing settings enable it by default.
- With cleanup off and English selected, dictate “Um, we should go”; delivery should read “We should go” while History's raw transcript retains the hesitation. With Auto selected, “um carro” must remain intact.
- Check Chinese hesitation runs and the exceptions 打呃 and 呃逆. Newlines, ordinary words, and the non-filler Chinese words 嗯, 啊, 额, 那个, 然后, and 就是 must remain.
- Check adjacent fillers leave one space between surviving English words, preserve newlines, and remove filler-owned punctuation including ellipses. `foo uh,uh,bar` should become `foo bar`, `hello. Um... we go` should become `hello. We go`, and `好的,呃。我们走` should become `好的。我们走`.
- Check URLs, emails, backtick-wrapped tokens, member access, attached code punctuation and assignments remain intact (`https://uh.edu/a`, `user@hm.com`, `uh_count`, `uh-oh`, `let uh = 1;`). A spaced hesitation dash (`um - I think`) must still filter in Auto.
- Check filler-only input, including `uh!` and `呃。`, produces no-audio feedback. Punctuation-only server text with no removed filler keeps its existing delivery behavior. Whole snippets must match the raw transcript first (`uh oh` and even the filler-only trigger `hmm`), then the filtered transcript (`Um, my address` for `my address`). Turn removal off and verify the transcript is unchanged. File uploads must continue returning raw server text.
- Tab to the switch and activate it with the keyboard; verify its VoiceOver label in both interface languages and all three themes.

## Skip cleanup regression check

- Delay the cleanup server, dictate, and verify Transcribing becomes Polishing only after ASR completes. Skip must appear inside the capsule while Cancel remains beside it. Skip is unavailable during ASR.
- Click Skip and verify immediate insertion of the recognized text with filler removal, Chinese script conversion, known corrections, and snippets. There must be no cleanup failure notice and no second paste when the delayed reply arrives. Cancel must still discard the whole operation.
- Tab to Skip and activate it with Return or Space. Check VoiceOver's “Skip cleanup” label and the “Insert without cleanup” hover hint. Check smooth width changes and emergence, then repeat with Reduce motion for immediate changes.
- Use `WHISPER_DEV_SCENE=pill:cleanup` with isolated data to inspect Light, Dark and Tokyo Night, and both interface languages.
- The pill renames itself "Polishing" after 400 ms of cleanup and offers Skip after 1.5 s. A Dictation the server finishes faster must go from "Transcribing" to the check with neither flashing past. Replay measured timings with `WHISPER_DEV_FLOW_MS=<transcribe>,<polish>` (for example `130,170`).
- Use `WHISPER_DEV_SCENE=pill:flow` to watch one whole Dictation on a loop: recording, transcribing, polishing, Skip leaving, and the delivered check. The meter and the label must never overlap, and the pill must not read "Transcribing" again after it has read "Polishing".

## Microphone loopback regression check

- In Settings > Dictation, check the microphone control order: Test, Refresh, device picker. Test must use the same resolved device as the next dictation, including Auto, lid changes, unavailable saved devices, and the system-default fallback.
- Test retains at most ten seconds with a live level meter, countdown, and Stop. Even if UI polling is delayed beyond the deadline, manual stop and auto stop must truncate the retained samples to ten seconds at the capture sample rate. Stop early, play, stop playback, replay, and Record again. The progress bar, duration and silence warning should reflect only the retained memory clip. A silent clip remains playable and shows a warning.
- Deny microphone permission, disconnect the selected device, and test a device delivering no frames. Check the named error and Try again. Disconnect output during playback and check its error as well.
- Start dictation with the hotkey during recording and playback. The test must close and yield audio before dictation begins. While dictation records, prepares, or processes, Test is disabled with an explanation. No test may play dictation cues or pause media.
- Close Settings, change section, change device, and drop/reopen the sheet during microphone startup, recording and playback. Each must release the streams and clip, and no old result may reopen the strip. No test audio may appear in History, recovery, storage or requests.
- Check Tab, Return/Space, VoiceOver names, Light/Dark/Tokyo Night, both languages, and Reduce motion. Use `settings:mic-recording`, `settings:mic-ready` and `settings:mic-error` dev scenes with isolated data for visual checks.
- `cargo test -- --ignored live_microphone_loopback` checks real capture and output with microphone authorization; ordinary lifecycle tests use an injected clock and hardware-free audio.

## Local silence filtering regression check

- In Settings > Speech to text, verify Skip silence defaults on, persists, and is named for VoiceOver. Check both languages, all themes, and keyboard activation.
- Record speech with pauses and background noise. Only the uploaded copy should be trimmed, preserving a 480 ms lookback and tail. Confirm adopted pre-roll is filtered too. History, recovery, recording duration and Insights words per minute must use the original recording.
- Try silence and broadband noise: VAD should avoid a server request and show the existing no-audio wording with a kept recording. Send anyway must send the original once, while Cancel and Record New still discard it. Turn filtering off and verify upload bytes remain untrimmed. File uploads remain outside this path.
- Retry History audio and verify filtering applies with current settings while the stored audio and duration remain unchanged. When speech is detected, send the trimmed copy; when no speech is detected, send the original stored WAV because retry explicitly requests that recording. Live dictation must still avoid a request and offer Send anyway. Check model/inference failures fail open and a short recording is sent intact.
- Automated tests exercise production frame retention with scripted probabilities, detector failures, partial final frames, a real speech fixture in silence and broadband noise, and independent ONNX Runtime reference probabilities. `cargo test --release sixty_second_recording_timing -- --nocapture` logs the 60-second latency without a brittle CI timing limit.

### VAD model spike and fixture provenance

`tract-onnx` 0.23.10 was checked against its current crate API and tested before integration. The general v6.2.2 op15 model fails static typing at a decoder `If` (branch ranks differ), while the official fixed-shape 16 kHz OpenVINO ONNX export loads directly. Both official exports give identical reference probabilities on the fixture. Anonymous in-memory node names are made unique before tract optimization so debug validation and release inference use the same graph; weights and connections remain unchanged. Tract identifies 83 of its 90 frames as speech at >0.3; three seconds of silence and seeded broadband noise identify zero speech frames. A separately evaluated `silero-vad-pure` 0.1.1 produced incorrect low speech probabilities and is not a dependency.

The small `tests/fixtures/vad-speech.wav` was generated locally with macOS `say -v Samantha` saying “Please send the design review to Thursday afternoon.” and converted with `afconvert -f WAVE -d LEI16@16000 -c 1`. It is 16 kHz mono PCM16, 93 KB, and contains no user recording. `vad-probabilities.json` records the official model's CPU ONNX Runtime probabilities for consecutive 512-sample frames, using zero initial `[2,1,128]` state, 64 samples of left context, and zero padding on the final frame. The production tract test compares every probability within 0.00002.

On the implementation machine (Apple M2 Pro, macOS 27.2.1), the isolated release timing test processed 60 seconds in **116.397 ms**, including first model initialization. The release executable grew from **44,016,400 bytes** before VAD to **67,235,808 bytes** with VAD (+23,219,408 bytes, about 22.14 MiB). These are executable sizes, not app bundles. `cargo build --release --locked --offline` passed; `otool -L` listed only Apple/system dependencies, with no ONNX Runtime or other new distributed dylib. No release packaging or signing was run.
32 changes: 32 additions & 0 deletions docs/third-party-notices.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,35 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

## Silero VAD v6.2.2

The main executable embeds the unmodified 16 kHz fixed-shape ONNX model
[`silero_vad_openvino_16k.onnx`](https://github.com/snakers4/silero-vad/blob/60b7ffa243625ebdc1070275a29f18c87843786a/src/silero_vad/data/silero_vad_openvino_16k.onnx)
from the official v6.2.2 release. The OpenVINO export is a standard ONNX graph;
Whisper executes it with pure Rust `tract-onnx` 0.23.10, without OpenVINO or
ONNX Runtime. It is 1,288,203 bytes, SHA-256
`7776b81ad1b0350c15d7f1555943b9232eb53e9ca5d989c6d0cea9ebc8664d87`.
No model download or external resource is used at runtime.

MIT License

Copyright (c) 2020-present Silero Team

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
31 changes: 28 additions & 3 deletions locales/en.json
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,12 @@
"serverTitle": "Server",
"modelHelp": "The model your server should run. Test Connection lists the ones it offers.",
"serverModels": "Models on this server",
"protocolTag": "OpenAI-compatible · no auth"
"protocolTag": "OpenAI-compatible · no auth",
"removeFillerWords": "Remove filler words",
"removeFillerWordsDescription": "Drops hesitation sounds such as “um” and “uh” from transcripts. Works without text cleanup.",
"optionsTitle": "Transcription options",
"skipSilence": "Skip silence",
"skipSilenceDescription": "Removes pauses and non-speech sound before the recording is sent, keeping a short buffer so words are not cut off. Runs on this Mac."
},
"llms": {
"server": {
Expand Down Expand Up @@ -330,7 +335,8 @@
"viewTranscript": "View Transcript",
"recordNew": "Record New",
"resend": "Resend Recording",
"openSettings": "Settings…"
"openSettings": "Settings…",
"sendAnyway": "Send anyway"
},
"noAudio": {
"title": "We didn’t catch that",
Expand Down Expand Up @@ -368,7 +374,11 @@
"cancel": "Cancel",
"cancelDictation": "Cancel Dictation",
"dismiss": "Close",
"closesIn": "Closes in {{seconds}} s; hover to keep open"
"closesIn": "Closes in {{seconds}} s; hover to keep open",
"polishing": "Polishing",
"skip": "Skip",
"skipHint": "Insert without cleanup",
"skipCleanup": "Skip cleanup"
},
"keptRecording": "Your {{time}} recording is kept until you close this card."
},
Expand Down Expand Up @@ -757,5 +767,20 @@
"copy": "Copy Text",
"copiedHint": "Copied. Press ⌘V to paste."
}
},
"microphoneTest": {
"test": "Test",
"stop": "Stop",
"play": "Play",
"recordAgain": "Record again",
"close": "Close microphone test",
"tryAgain": "Try again",
"privacy": "Stays on this Mac. Never transcribed or saved.",
"silent": "No sound was picked up. Check the selected microphone.",
"permission": "Microphone permission is missing.",
"device": "The microphone is unavailable or disconnected.",
"noSignal": "The microphone did not deliver an input signal.",
"output": "Audio output is unavailable.",
"busy": "Finish or cancel dictation before testing the microphone."
}
}
Loading
Loading