vad-benchmarks.md

VAD benchmarks

This page lists every voice activity detection (VAD) measurement made for parakeet.cpp so far: how each one was built, what it measures, the numbers, and how to run it again. For the API and the model files, see vad.md.

Read the caveats before you quote a number. Every reference here is coarse, the machine was shared, and some runs were not on a quiet machine. A difference that is smaller than the noise is not claimed.

The detectors:

Name	What it is
Silero	Silero VAD 6.2.3 (MIT), 32 ms frames, from its own small GGUF (F32 or F16)
Ultra Q8_0	VAD head of moondream/parakeet-ultra, GGUF in Q8_0, 80 ms frames
Redux packed	VAD head of moondream/parakeet-redux, packed ternary GGUF, 80 ms frames
whisper.cpp Silero	Silero VAD inside whisper.cpp (whisper_vad_*), used as a comparison
ONNX Silero	The official Silero ONNX file run with onnxruntime (CPU, one thread), used as the reference

Summary

Accuracy is the frame level F1 (percent) against a coarse reference, see Metrics. Speed is audio seconds per wall second ("x real time") on a 300 s clip. Higher is faster.

Detector	F1, clean synthetic	F1, pink noise 0 dB	F1, TED-LIUM talks	Speed	Peak memory	File size
Silero (matched settings)	94.1	91.7	91.8	about 660x on 1 thread, parakeet-cli vad, includes process start and model load	60 MB (parakeet-cli vad, F32 and F16)	2.2 MB (F32), 1.3 MB (F16)
Silero (its own defaults)	94.4	91.9	92.1	not timed separately	same as above	same as above
Ultra Q8_0 head, full ASR GGUF	95.0	93.4	93.8	216x, 8 threads (older run); 91.6x on 1 thread and 266.6x on 8 threads in the slice run, busy machine	905 MiB loaded, 1134 MiB after a 33 s clip	941.5 MB
Ultra Q8_0 head, slice file (PR 87, not in a release yet)	same as the full file (see below)	same	same	91.8x on 1 thread, 257.8x on 8 threads, busy machine	12 MiB loaded, 241 MiB after a 33 s clip	6.0 MB
Redux packed head, full ASR GGUF	95.4	94.3	94.2	221x, 8 threads (older run); 94.9x on 1 thread and 267.1x on 8 threads in the slice run, busy machine	372 MiB loaded, 603 MiB after a 33 s clip	213.3 MB
Redux packed head, slice file (PR 87, not in a release yet)	same as the full file	same	same	95.1x on 1 thread, 266.7x on 8 threads, busy machine	16 MiB loaded, 246 MiB after a 33 s clip	9.9 MB
Silero F16, same build as the slice run	see Silero rows			736x on 1 thread, 485x on 8 threads, busy machine	9 MiB loaded, 13 MiB after a 33 s clip	1.3 MB
whisper.cpp Silero	94.1	not run	not run	in process: 686x on 1 thread	36 MB (test harness)	0.9 MB
ONNX Silero (onnxruntime)	94.0	not run	not run	in process: 449x on 1 thread, Python loop	104 MB	2.3 MB

How to read it:

• The synthetic and TED columns come from the heads against Silero benchmark. The whisper.cpp and ONNX rows come from the Silero parity benchmark, which uses a different synthetic corpus (120 clips, 3 conditions). The clean F1 of "Silero" and "whisper.cpp Silero" are therefore not from the same clips. Compare rows only inside one benchmark.
• "Matched settings" means Silero ran with the same segmentation settings as the heads (threshold 0.5, minimum speech 100 ms, minimum silence 200 ms, no padding). See Metrics.
• Head speeds on 8 threads come from two separate runs on a busy machine (see the caveats of each section): the older run (216x to 222x for four head builds, a Python wrapper, load average about 6) and the slice run (255x to 268x, one build of the C++ harness, load average up to 17 at the start of some rounds). They differ by about 20%, which is the size of the noise on this machine for 8 threads, not a difference between methods. The 1 thread rounds of the slice run were within about 5% of each other, except for two outlier rounds (69.8x and 84.7x).
• The speed columns of Silero and the heads are not a like for like comparison of the models. They show the cost of using each one as a stand alone gate.
• The heads need the ASR GGUF unless you use a slice file. A slice is a GGUF with only the head path, made by scripts/slice_vad_gguf.py (PR 87, in master, not in a tagged release yet). Its VAD output is byte for byte the same as the parent file on three clips, so the accuracy columns apply to the slice unchanged. See Slice only head. The published slices are listed in vad.md.
• All memory and load numbers for the heads are in the slice section.

Machine and build

Item	Value
CPU	AMD Ryzen 9 class (Zen 5), 16-core desktop part; the OS shows 20 logical CPUs, one thread per core
Memory	84 GB
OS and compiler	Linux 6.8, GCC 13.3, CMake 3.28
ggml	commit e705c5fe (the submodule of this repository), GGML_NATIVE=ON, Release
parakeet.cpp	the VAD code of PR 83 (merged as 6165e3d); see the per benchmark build notes below
whisper.cpp	commit 60c0be6ac8fa71b1a2ae2dd938a31a34a508e774 (version 1.9.4), CPU only
Load	the machine was shared with other jobs; load is given per benchmark

Build commits per benchmark:

Benchmark	parakeet.cpp build
Heads against Silero	a development build of the VAD branch; the commit was not recorded (unverified)
Silero in whisper.cpp and in parakeet.cpp	c1a8391102d34074f48e758c441758327d2465b7, the VAD branch before its squash into 6165e3d
Long talks, batched decode	three builds, called base, b0 and b1: master before PR 81, PR 81 (cbabaf5), PR 82 (9fdf3a1). The result files do not record the commits; the mapping is from the PR 82 description (unverified)
Timing re-run	6165e3d, see Timing re-run

Metrics

All VAD output is reduced to speech regions (start and end in seconds).

• Frame level precision, recall, F1. The regions of the detector and of the reference are turned into a speech mask on a 10 ms grid. Counts of true positive, false positive, false negative and true negative frames are summed over all clips of a group, then precision, recall and F1 are computed from the sums. Pooled, not an average of per clip scores.
• Boundary error. For each reference start (and end), the distance in ms to the nearest detected start (end). Reported as median and 90th percentile, in ms, plus the miss rate: the share of reference boundaries with no detected boundary within 1 s. "All conditions" pools start and end errors.
• Agreement. Two detectors compared with each other: F1 of the two speech masks, Cohen's kappa, and IoU (intersection over union) of the masks.
• Probability parity. Per frame, the absolute difference of the speech probability against onnxruntime on the official ONNX file. Also the share of frames where the decision at 0.5 flips.
• Matched settings. The segmentation step is not the same code in every system, and the defaults differ. For a fair model comparison, Silero in the first benchmark runs the silero-vad package (get_speech_timestamps) with threshold 0.5, min_speech_duration_ms 100, min_silence_duration_ms 200, speech_pad_ms 0. These are the head defaults. Silero's reference code also lowers the threshold by 0.15 once speech has started (hysteresis); the parakeet.cpp segmenter has no hysteresis (see vad.md). "Silero (its own defaults)" uses the package defaults (minimum speech 250 ms, minimum silence 100 ms, padding 30 ms). The heads ran with their own defaults, which are the matched values.

Parakeet heads against Silero

Data

Synthetic clips. Built from LibriSpeech test-clean (public). Every clip has five utterances in a random order. Each utterance is trimmed at the first and last 10 ms frame whose RMS is above 1% of the peak frame RMS. The utterances are joined with gaps of 0.3 to 2.0 s (uniform), with 0.5 to 1.5 s before the first and after the last. The gaps hold Gaussian noise at a standard deviation of 0.001. The reference speech region of an utterance is the whole trimmed utterance, so short pauses inside an utterance count as speech.

Noise is added to the whole clip at a signal to noise ratio (SNR) measured against the power of the speech regions: white or pink, at 20, 10, 5 and 0 dB, plus a clean condition. That gives 9 conditions of 24 clips: 216 clips, 1080 reference regions. The speech layout of clip k is the same in every condition.

How the 24 clips were chosen is partly unverified. The script that cut the LibriSpeech subset for this run was not saved. scripts/vad_bench/fetch_data.py takes every 13th utterance of test-clean, which is the scheme of the Silero parity corpus, and the numbers below were measured on a subset of that kind. A rebuild with the script will give the same kind of corpus, not the same bytes, and the numbers will differ slightly.

TED-LIUM talks. Three talks from distil-whisper/tedlium-long-form (test split), 3178.6 s in total. Which three is not recorded in the result files (unverified). The reference is built from word timestamps of the 0.6B v3 F16 ASR model run on 30 s pieces: words closer than 0.4 s are merged into one region. This reference is derived from an ASR model and is coarser than the synthetic one. It is also not independent of the Parakeet family. The reference speech fraction is 91.9%. One more talk was left out of the run by a name filter; the reason was not recorded.

Results: synthetic, threshold 0.5

Cells are precision / recall / F1, in percent. "Silero M" is Silero with matched settings, "Silero D" with its own defaults.

Condition	Silero M	Silero D	Ultra Q8_0	Redux packed
clean	99.1 / 89.7 / 94.1	98.7 / 90.5 / 94.4	98.2 / 92.0 / 95.0	98.4 / 92.7 / 95.4
white 20 dB	99.4 / 88.4 / 93.5	99.0 / 89.2 / 93.9	98.8 / 89.9 / 94.1	98.7 / 91.7 / 95.1
white 10 dB	99.3 / 88.0 / 93.3	98.9 / 89.0 / 93.7	98.9 / 88.3 / 93.3	98.2 / 91.9 / 94.9
white 5 dB	99.2 / 87.8 / 93.1	98.8 / 88.5 / 93.4	98.9 / 87.2 / 92.7	97.4 / 91.9 / 94.6
white 0 dB	99.2 / 85.8 / 92.0	98.8 / 86.7 / 92.4	98.3 / 87.9 / 92.8	96.4 / 92.3 / 94.3
pink 20 dB	99.4 / 88.4 / 93.6	99.1 / 89.3 / 93.9	98.8 / 90.8 / 94.6	98.6 / 92.0 / 95.2
pink 10 dB	99.3 / 88.0 / 93.3	99.0 / 89.0 / 93.7	99.0 / 88.8 / 93.6	98.1 / 92.2 / 95.1
pink 5 dB	99.4 / 87.6 / 93.1	99.0 / 88.5 / 93.5	99.0 / 88.4 / 93.4	97.4 / 92.6 / 94.9
pink 0 dB	99.2 / 85.2 / 91.7	98.8 / 85.9 / 91.9	96.3 / 90.8 / 93.4	95.5 / 93.2 / 94.3

What the table supports: the three detectors are within about 1.5 F1 points of each other at 5 dB and above, and up to 2.6 points apart at 0 dB. Silero has the higher precision and the lower recall. Both heads have the higher recall. In the loudest noise (0 dB) the heads keep their F1 better than Silero (pink 0 dB: Silero M 91.7, Ultra 93.4, Redux 94.3). The recall of all detectors is below 94% because the reference counts pauses inside utterances as speech, so the absolute recall values are not a measure of missed speech.

Boundary error, in ms (median / 90th percentile, miss rate), start then end:

Condition	Silero M	Silero D	Ultra Q8_0	Redux packed
clean	S 171/527 (0.0%) E 121/366 (0.0%)	S 162/499 (0.0%) E 137/343 (0.0%)	S 123/460 (0.0%) E 168/325 (0.0%)	S 152/459 (0.0%) E 151/328 (0.0%)
white 10 dB	S 249/526 (0.0%) E 116/371 (0.0%)	S 232/500 (0.0%) E 129/357 (0.0%)	S 211/470 (0.0%) E 147/472 (0.8%)	S 163/466 (0.8%) E 152/389 (0.8%)
white 0 dB	S 262/571 (0.8%) E 117/459 (2.5%)	S 237/526 (0.8%) E 142/451 (1.7%)	S 162/461 (0.0%) E 149/494 (2.5%)	S 202/460 (0.8%) E 149/404 (1.7%)
pink 10 dB	S 248/526 (0.0%) E 106/384 (0.0%)	S 232/502 (0.0%) E 117/343 (0.0%)	S 226/479 (0.0%) E 136/445 (0.0%)	S 176/445 (0.0%) E 148/340 (0.0%)
pink 0 dB	S 261/570 (0.0%) E 145/411 (3.3%)	S 248/530 (0.0%) E 150/408 (3.3%)	S 180/457 (2.5%) E 162/470 (2.5%)	S 200/457 (1.7%) E 160/428 (1.7%)

All conditions pooled:

System	Median	90th percentile	Start median	End median	Miss rate
Silero M	143	501	237	116	0.4%
Silero D	149	484	232	130	0.3%
Ultra Q8_0	154	465	181	149	0.6%
Redux packed	153	437	170	149	0.4%

Boundary errors of 100 to 250 ms are large next to the 32 and 80 ms frame sizes. Most of this comes from the reference (trimmed utterances next to short gaps), so only the ordering of start errors is informative: the heads place starts about 50 to 70 ms closer to the reference than Silero M does. The signed start error median is 237 ms for Silero M, 232 for Silero D, 162 for Ultra and 147 for Redux (the detector starts later than the reference). The signed end error median is 3, 20, -10 and 54 ms. Segments per reference region: Silero M 1.96, Ultra 1.92, Redux 1.76 (a value above 1 means the detector splits utterances at their inner pauses).

Threshold sweep, pooled over clean, white 10 dB, white 0 dB and pink 10 dB (precision / recall / F1):

System	threshold 0.3	threshold 0.5	threshold 0.7
Silero M	98.5 / 90.3 / 94.3	99.2 / 87.9 / 93.2	99.5 / 86.2 / 92.4
Ultra Q8_0	97.1 / 92.4 / 94.7	98.6 / 89.3 / 93.7	99.0 / 86.2 / 92.1
Redux packed	94.8 / 95.0 / 94.9	97.7 / 92.3 / 94.9	98.9 / 90.0 / 94.2

Agreement with Silero M on all synthetic clips: Ultra F1 96.6, kappa 0.882, IoU 93.4. Redux F1 96.3, kappa 0.865, IoU 92.8. Ultra against Redux: F1 97.2, kappa 0.896.

Results: TED-LIUM talks

Frame level on 3178.6 s, with the ASR derived reference (speech fraction 91.9%):

System	Precision	Recall	F1	Speech fraction found
Silero M	97.3	86.8	91.8	82.0%
Silero D	97.4	87.4	92.1	not recorded
Ultra Q8_0	97.1	90.7	93.8	85.8%
Redux packed	97.1	91.5	94.2	86.6%

Agreement: Ultra and Silero M F1 97.0, kappa 0.813; Redux and Silero M F1 96.9, kappa 0.801; Ultra and Redux F1 98.8, kappa 0.912. The "Silero D" speech fraction was not printed in the run output.

The ordering is the same as on the synthetic clips: equal precision, the heads have the higher recall. The gap in F1 between Silero M and the heads is 2 to 2.5 points. Because the reference comes from a Parakeet model, a bias in favor of the Parakeet heads is possible and was not measured.

Results: speed of the heads and of Silero ONNX

300 s of one talk (the first 300 s of the 1249 s talk), 4 repetitions, load average 5.8 to 6.2 during the run. Silero ran through the silero-vad Python package (ONNX, one thread). The heads ran with 8 threads.

System	Speed, x real time
Silero ONNX, 1 thread	393
Ultra Q8_0, 8 threads	216
Ultra F16, 8 threads	218
Redux packed, 8 threads	221
Redux dequantized F16, 8 threads	222

The run printed one speed per system. The result file does not say whether it is the best or the median of the 4 repetitions (unverified). The speeds that the TED script printed during the accuracy run (load average 6 to 42) are not valid and are not used. Thread pinning was not recorded for this run.

Caveats

• References are coarse. The synthetic reference counts pauses inside utterances as speech and uses an energy trim. The TED reference comes from ASR word times.
• Absolute precision and recall depend on that reference. Compare detectors with each other, on the same clips.
• Differences of less than about one F1 point between two detectors in one cell are not claimed. There are no confidence intervals; one corpus was used.
• Speed numbers: see the load notes above. Detectors with different thread counts are not a model comparison.
• The build commit of this run was not recorded.

Silero in whisper.cpp and in parakeet.cpp

This benchmark asks whether the Silero port in parakeet.cpp gives the same result and the same speed as the Silero VAD that ships in whisper.cpp.

Setup

• Models. silero_vad.onnx (version 6.2.3, SHA-256 1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3); parakeet.cpp GGUF F32 (1398e5ce...) and F16 (8160489282...) made with scripts/convert_silero_vad_to_gguf.py from that ONNX file; whisper.cpp model ggml-silero-v6.2.3-ggml.bin (abd79739...), converted from the same ONNX with the whisper.cpp conversion script, and the model of the whisper.cpp model repository ggml-silero-v6.2.0.bin (2aa269b7...). The full list of hashes and versions is in silero_versions_and_sha256.txt.
• Reference. onnxruntime on the official ONNX file, with the official 64 sample context, one thread.
• whisper.cpp access. A small test program (scripts/vad_bench/wvad.cpp) over the public whisper_vad_* functions: probabilities, and the segments of the whisper.cpp C++ code with its defaults and with matched parameters. It is not part of whisper.cpp.
• Corpus. 120 synthetic clips (40 clips of 5 utterances, in clean, white 10 dB and pink 10 dB), built with the same recipe as above from 200 LibriSpeech test-clean utterances (every 13th). Plus the first 600 s of one TED-LIUM talk.
• Segmentation. To compare only the models, the same Python code turns every probability stream into segments: threshold 0.5, negative threshold 0.35, minimum speech 100 ms, minimum silence 200 ms, padding 0 ("matched post"). The own defaults of whisper.cpp and parakeet.cpp are also shown.

Probability parity against onnxruntime

All 120 clips and the 600 s talk, one value per 32 ms chunk:

System	Max abs diff	Mean abs diff	Frames > 0.05 off	Decision agreement at 0.5	Flipped frames
whisper.cpp, converted 6.2.3	0.7373	0.00224	0.54%	99.826%	309
whisper.cpp, repository 6.2.0	0.7373	0.00224	0.54%	99.826%	309
parakeet.cpp F32	0.0001	0.00002	0.00%	99.999%	2
parakeet.cpp F16	0.0036	0.00005	0.00%	99.997%	5

whisper.cpp (converted 6.2.3) against parakeet.cpp F32: mean abs diff 0.00224, 311 flipped frames. The two whisper.cpp model files give the same result (0 flipped frames between them). Mean abs diff against onnxruntime, first chunk of each clip / the rest: whisper.cpp 0.00088 / 0.00212, parakeet.cpp F32 0.00002 / 0.00002, F16 0.00005 / 0.00005. On the 600 s talk alone, the number of flipped frames against onnxruntime is 57 of 18750 for whisper.cpp, 0 of 18750 for parakeet.cpp F32 and 1 for F16. The cause of the whisper.cpp difference was not investigated.

Segment quality on the synthetic clips

Precision / recall / F1 in percent, 40 clips per condition, same reference as above:

System (settings)	clean	white 10 dB	pink 10 dB	all
onnxruntime (matched post)	99.3 / 89.3 / 94.0	99.4 / 87.8 / 93.2	99.5 / 87.4 / 93.1	99.4 / 88.2 / 93.5
whisper.cpp (matched post)	99.3 / 89.4 / 94.1	99.4 / 87.8 / 93.3	99.5 / 87.4 / 93.0	99.4 / 88.2 / 93.5
parakeet.cpp F32 (matched post)	99.3 / 89.3 / 94.0	99.4 / 87.8 / 93.2	99.5 / 87.4 / 93.1	99.4 / 88.2 / 93.5
parakeet.cpp F16 (matched post)	99.3 / 89.3 / 94.0	99.4 / 87.8 / 93.2	99.5 / 87.4 / 93.1	99.4 / 88.2 / 93.5
whisper.cpp (its own defaults)	98.9 / 90.3 / 94.4	99.2 / 89.0 / 93.8	99.3 / 88.6 / 93.6	99.1 / 89.3 / 94.0
whisper.cpp (its C++ code, matched parameters)	99.3 / 89.4 / 94.1	99.4 / 87.8 / 93.3	99.5 / 87.4 / 93.0	99.4 / 88.2 / 93.5
parakeet.cpp F32 (its own defaults)	99.1 / 89.3 / 93.9	99.3 / 87.6 / 93.1	99.4 / 87.3 / 93.0	99.3 / 88.1 / 93.3

Boundary error against the reference, all conditions pooled (start median / 90th percentile, end median / 90th percentile, miss rate): onnxruntime, whisper.cpp, parakeet.cpp F32 and F16 with matched post all give 241/553 and 108/396, 0.2%. whisper.cpp with its own defaults gives 209/521, 124/367, 0.0%. parakeet.cpp with its own defaults gives 211/523, 116/392, 0.0%.

Agreement of the speech masks (matched post), 120 clips pooled (F1 / kappa / IoU):

Pair	F1	kappa	IoU
whisper.cpp vs parakeet.cpp F32	99.92	0.9969	99.84
whisper.cpp vs onnxruntime	99.92	0.9969	99.84
parakeet.cpp F32 vs onnxruntime	100.00	1.0000	100.00
parakeet.cpp F16 vs F32	100.00	0.9999	100.00
whisper.cpp 6.2.3 vs 6.2.0 file	100.00	1.0000	100.00

On the 600 s talk (no reference), speech fraction found: onnxruntime 79.1%, whisper.cpp 78.8%, parakeet.cpp F32 79.1%. whisper.cpp against parakeet.cpp F32: F1 99.75, kappa 0.9882.

What this supports: with the same post processing, whisper.cpp Silero and parakeet.cpp Silero give the same segment quality on these clips (all within 0.1 F1 point). The difference between whisper.cpp and parakeet.cpp in their own defaults (94.0 against 93.3 F1 on all clips) comes from their different default settings, not from the model. The F16 file is not measurably worse than F32 here.

Speed

Each job ran on one pinned core, on a 300 s clip (the first 300 s of the 600 s talk; the clip file was not kept, so this is assumed from the preparation script), serialized with a lock. A run started only when the mean number of other runnable tasks over 10 s was below 4, or the 1-minute load average was below 5. The machine has 20 logical CPUs and was never idle, so the 1-minute load average at the start of the runs was between 4.2 and 15.9, and the mean runnable count was 1.8 to 5.5. These are not quiet machine numbers. Wall time is the process wall time, including process start and model load for the command line tools.

Job	Runs	Wall time, median (min to max), s	In process time, median, s	x real time (median)	Peak RSS
whisper.cpp detect_speech only (test program)	5	0.456 (0.446 to 0.473)	0.438	686	36 MB
whisper-vad-speech-segments	5	0.452 (0.444 to 0.457)	not applicable	664	28 MB
parakeet-cli vad, F32	5	0.455 (0.425 to 0.464)	not applicable	659	60 MB
parakeet-cli vad, F16	5	0.451 (0.436 to 0.553)	not applicable	666	60 MB
onnxruntime, Python, 1 thread	3	0.806 (0.738 to 1.241)	0.668	449	104 MB

The x real time column is 300 s divided by the in process time when there is one, else by the wall time. The four Silero command line jobs are within the spread of their repeated runs (0.43 to 0.55 s), so the numbers do not show a speed difference between whisper.cpp and parakeet.cpp. The onnxruntime figure is a Python loop that calls the session once per 512 sample chunk. It is not a tuned onnxruntime number.

whisper.cpp with tiny.en, one thread, same 300 s clip, end to end: 15.14 s without VAD, 19.68 s with the Silero VAD (3 runs each, load average 4.6 to 8.3). The ASR part dominates. Why the run with VAD is slower was not investigated.

Caveats

• The load notes above apply. The runs were on a shared machine.
• One clip and one core. No claim is made about other lengths, other CPUs, or more threads.
• The onnxruntime run had 3 repetitions. The other jobs had 5.
• The measured whisper.cpp is a source build at the commit above, CPU only.

Long talks: batched decode for VAD segments

PR 82 decodes the VAD segments in groups of up to 16, using the exact batched decoder of PR 81. Each segment is still encoded alone. This measures that change, not the VAD itself.

Data: four TED-LIUM long-form talks of 18 to 25 minutes, merged to one file each. Models: Ultra Q8_0 and Redux packed. Command: parakeet-cli transcribe --vad --json --threads 8, pinned to 8 cores. Three builds, interleaved, 5 runs each: 120 runs. The build order was rotated for each repetition. Before each run the script waited until the mean number of other runnable tasks over 10 s was below 4. The mean number of other runnable tasks at the gate was 0 to 1.5, but the 1-minute load average at the start of the runs was 2.5 to 18.0 (it includes the earlier runs of the same job and other jobs), see scripts/vad_bench/results/longform_b1_loadlog.txt and the per run values in longform_b1_results.tsv. The 1-minute load average was therefore often above 5. Wall time is the time of the whole command, including model load.

Median wall time in seconds over 5 runs:

Talk (length, s)	Model	Segments	base	PR 81	PR 82	PR 82 vs PR 81	Peak RSS PR 82 vs PR 81
BillGates (1505.8)	Ultra Q8_0	59	44.2	43.7	40.8	1.072x	+1.7%
BillGates	Redux packed	62	28.1	27.7	25.9	1.071x	+2.2%
AimeeMullins (1249.0)	Ultra Q8_0	46	36.9	36.9	34.8	1.058x	+1.7%
AimeeMullins	Redux packed	46	23.4	23.7	21.9	1.079x	+2.5%
JaneMcGonigal (1167.6)	Ultra Q8_0	50	35.9	34.9	34.0	1.029x	+1.4%
JaneMcGonigal	Redux packed	47	24.1	23.1	23.0	1.006x	+2.2%
DanielKahneman (1095.5)	Ultra Q8_0	44	32.6	33.1	32.9	1.006x	+1.7%
DanielKahneman	Redux packed	42	21.4	21.5	21.2	1.013x	+2.2%

The average of the eight speed ratios is 1.042x (range 1.006x to 1.079x). Peak memory of the whole transcription is 1.4 to 2.5 percent higher with PR 82, 30 to 40 MB on a base of 1.4 to 2.0 GB. The transcript (JSON output) has the same hash for all three builds in all 8 talk and model pairs.

Caveats: the gain is small and several of the ratios are inside the spread of the 5 runs (some single runs of the same configuration differ by more than 3%); do not claim more than a small gain. Only CPU and x86 were measured. The memory numbers are for the whole transcription, not for the VAD.

Silero parity with the reference (PR 83)

This table is also in vad.md. Test clip: tests/fixtures/speech.wav padded with silence (310 chunks), against onnxruntime 1.27.0 and the ONNX file of version 6.2.3. Maximum probability difference:

File	16 kHz	8 kHz
F32	9e-7	2e-6
F16	7e-4	3e-3

One run of tests/silero_vad_probe on one CPU thread measured about 40 us per chunk for F32 and F16 at both rates. That is one run on one machine. It is not a benchmark and is not used above.

Slice only head

A "slice" is a GGUF that holds only the tensors the VAD path of an Ultra or Redux file reads. The code that makes it and loads it is PR 87 (merged to master as e53a253, not in a tagged release yet; the latest tag is v0.5.0). How to make and use a slice is in vad.md. This section compares each full ASR GGUF with its slice, and with Silero F16, built from the same source tree.

Setup

• One build of the PR 87 code, one machine (see the table above), the first 300 s of the talk used elsewhere on this page for speed (ted300), and a 33 s clean synthetic clip for memory. The harness is scripts/vad_bench/vad_slice_bench.cpp (a small program built inside the source tree of PR 87, since it uses internal headers) and the driver is scripts/vad_bench/slice_bench.py.
• Speed: one timed VAD run after a warm-up run, 5 rounds with the 9 configurations interleaved, the best of the 5 reported (the median is in the result file), on 1 thread and on 8 threads. The 8 thread runs were pinned to the 8 idlest cores at the time. Each round waited for the gate: mean number of other runnable tasks under 4 or the 1-minute load average under 5.
• Load time: the model load alone, median of 10. Warm means the file is in the page cache (the first load, which warms it, is dropped). Cold means the page cache for the file was evicted before each load.
• Peak memory: peak resident set size from /usr/bin/time -v, once per configuration: loading only, and parakeet-cli vad on the 33 s clip with 8 threads.
• Same output: parakeet-cli vad --probabilities was run with each slice and with its parent file on three clips (the 33 s clip, a noisy clip, the 600 s talk). All four slices gave byte-identical JSON on all three clips (12 of 12 pairs).

Results

Model	Full file	Slice file	Load, warm / cold, full (ms)	Load, warm / cold, slice (ms)	Peak RSS loaded, full / slice (MiB)	Peak RSS after the 33 s clip, full / slice (MiB)
Ultra Q8_0	941.5 MB	6.0 MB	276 / 424	0.2 / 2.1	905 / 12	1134 / 241
Ultra F16	1441.9 MB	9.9 MB	429 / 666	0.4 / 3.5	1382 / 15	1613 / 246
Redux packed	213.3 MB	9.9 MB	92 / 128	0.5 / 3.7	372 / 16	603 / 246
Redux dequantized F16	1441.9 MB	9.9 MB	434 / 669	0.2 / 3.8	1382 / 16	1613 / 246
Silero F16	1.3 MB	not applicable	0.4 / 1.5	not applicable	9 / not applicable	13 / not applicable

Speed on the 300 s clip, x real time, best of 5 (1 thread / 8 threads):

Model	Full file	Slice file
Ultra Q8_0	91.6 / 266.6	91.8 / 257.8
Ultra F16	94.5 / 255.5	94.6 / 265.3
Redux packed	94.9 / 267.1	95.1 / 266.7
Redux dequantized F16	95.1 / 265.3	94.8 / 268.1
Silero F16	736.3 / 485.4	not applicable

Medians of the 5 rounds (1 thread / 8 threads) are lower: Ultra Q8_0 full 91.4 / 250.5, slice 91.5 / 227.8; Redux packed full 93.1 / 252.8, slice 92.4 / 236.3; Silero F16 702.1 / 473.9. The slice files are 6,007,328 bytes (Ultra Q8_0) and 9,939,488 bytes (the other three). The SHA-256 of the four slice files is in slice_sha256.txt.

What this supports:

• A slice is 6 to 10 MB, loads in under 4 ms, and needs about 12 to 16 MiB loaded and about 240 to 250 MiB for a 33 s clip. The full file needs 372 MiB to 1.4 GiB just to load.
• Slice and full file run at the same speed. The differences between them on 1 thread (best of 5) are 0.1% to 0.3%, far inside the spread of the 5 rounds.
• Silero F16 is smaller (1.3 MB, 13 MiB after the clip), loads in about 0.4 ms, and was about 7.7 to 8 times faster than the heads on 1 thread (736x against 92x to 95x).

Caveats

• The 8 thread numbers are noisy. The machine was busy: the gate lines of the speed rounds in slice_gate.log show a 1-minute load average from 4.2 to 17.3 at the start of the rounds, and some rounds ran at 9 to 17 (the gate also accepts a low runnable count when the load average is high). The best of 5 on 8 threads is within the noise of the other values (the 8 thread rounds of one configuration ranged by up to 30%, for example Ultra Q8_0 full from 185.0x to 266.6x). Do not read a ranking from the 8 thread column. The 1 thread rounds of one configuration are within about 5% of each other, except for two outlier rounds (Redux dequantized F16 full 69.8x, slice 84.7x).
• Load times are medians of 10 and tiny for slices, so the warm slice values (0.2 to 0.5 ms) are close to the timer's resolution and the maximum was up to 2.2 ms.
• Peak memory was measured for a 33 s clip only. The peak memory for a 600 s clip was not measured, and the memory after the clip grows with the clip length.
• The slice code is in master (PR 87) but not in a tagged release yet. GPU backends and the streaming VAD were not tested with slices.
• These numbers come from one run of the harness. They were not repeated.
• The same output check shows that the probabilities are identical, so the accuracy tables of this page apply to the slices. It was run on three clips, not on the whole corpus.

Timing re-run

A re-run of the timing parts on a quiet machine was tried and did not happen. The rule was: run only when the 1-minute load average is below 5, check before each run, poll every 10 minutes for up to 3 hours, serialize with a lock and pin cores. The machine was shared with other jobs. The load average was checked 19 times, from 22:43 to 01:43, and was never below 5. The lowest value was 6.67 and the highest was 82.37. The poll log is in timing_rerun_load_log.txt. No timing was run for this page. The speed numbers of the slice section were taken separately, on a busy machine, as described there. They are not quiet-machine numbers either. The speed numbers on this page are the old ones, taken under the loads that each section states. Numbers taken on a busy machine: all speeds of the heads against Silero section, all Silero speed and memory numbers of the parity section, all slice section numbers, and the batched decode times (load average 2.5 to 18.0 at the start). The setup for the re-run (a build of 6165e3d, the 300 s clip, the models and the driver) was ready, and speed_silero.py and speed_heads.py run it in one command each when the machine is quiet.

The accuracy numbers do not depend on load and were not re-run.

The head on noise-only audio

The Parakeet VAD head gives false alarms on audio that has no speech. Silero does not. This was first measured on synthetic data: LibriSpeech test-clean utterances with added white or pink noise (see the fusion experiment for the corpus). A follow-up root-cause study (below) then looked at why, at other noise types and levels, and at what it costs in transcribe --vad. The numbers in the first two tables are frame rates at threshold 0.5 with the default segmenter settings (bridge 0.1 s, minimum speech 0.1 s, minimum pause 0.2 s, no padding).

Test	Silero	Ultra head (Q8_0)	Redux head (packed)
Speech-free 30 s clips (96 clips: white and pink noise at 20, 10, 5 and 0 dB), share of frames called speech	0.00%	99.44%	97.83%
A 30 s noise stretch inside a file that also has speech (96 files), share of frames in the stretch called speech	0.0%	17.7%	55.2%

For the stretch inside a speech file, the rate depends on the noise level:

Noise level (SNR)	Ultra head	Redux head	Silero
20 dB	0.0%	35.2%	0.0%
10 dB	0.6%	65.3%	0.0%
5 dB	14.3%	60.8%	0.0%
0 dB	55.8%	59.4%	0.0%

Each cell pools 12 white noise and 12 pink noise files. The numbers are in scripts/vad_bench/fusion/results.md (noise-only clips) and gap_results.md (the stretch inside a file).

Root-cause study

Verdict: this is not a bug in parakeet.cpp. It is a property of the model. Scripts and result files: scripts/vad_bench/noise_dive/.

Our implementation is faithful. An independent reference was built from Hugging Face transformers (feature extractor and subsampler) and the head structure documented in ternary.md. On 8 signals it matches parakeet-cli vad --probabilities to a maximum probability difference of 1.1e-4 to 2.6e-4 on F16 files (Redux and Ultra F16), and up to 1.1e-2 for the Ultra Q8_0 file (quantisation), and the speech fractions agree (res_A.md). On the 12 stretch files checked, the CLI and the reference agree on the false-alarm rate to within 0.5 points (res_cli_check.txt). The one real difference is digital silence (see Proposed behaviour).

Why

The head judges each 80 ms frame by its level and texture relative to the average of the file it is in. Steady loud noise, or a file that is only noise, therefore sits at a logit of about +1.5 to +2 (probability 0.8 to 0.9). Speech sits at +5 to +12 and pauses at -3 to -8:

Logit (median, 5th to 95th percentile)	Redux	Ultra
Speech in clean clips	+8.0 (-3.1 to +12.7)	+7.0 (-2.7 to +13.2)
Pauses in clean clips (digital -60 dBFS dither)	-8.1 (-11.6 to -5.5)	-3.2 (-5.0 to -1.8)
Noise-only file at -23 dBFS	+1.5 (+0.2 to +3.0)	+1.7 (+0.5 to +3.1)
Noise-only file at -50 dBFS	+0.5 (-0.6 to +1.8)	+1.3 (+0.1 to +2.6)

What the study found about the cause:

• The per-file normalisation of the features is one input, not the whole cause. With a fixed normalisation (mean and standard deviation taken from the speech clips), noise-only files at -60 and -50 dBFS stop firing (Redux 0.0 percent, Ultra 0.0 to 49 percent depending on the statistics), but at -30 and -23 dBFS they still do (Redux 78 and 97 percent with one set of statistics). A noise stretch inside a speech file does not go away either (res_B3.txt).
• Level alone does not explain the logit on noise. A held-out regression of the logit on six frame features (level, absolute log level, flux, flatness, 2 to 8 Hz modulation, tilt) gives R2 of 0.01 to 0.03 on white, pink and speech-shaped noise (about 0.2 on brown noise) and 0.74 to 0.76 on speech (res_B4.txt). Inside a speech file the logit follows the mean level of the stretch relative to the file (correlation 0.80 for Redux and 0.84 for Ultra with the median logit, res_B8.txt) and its temporal variation (res_B10.txt).
• The share of noise in the file is not monotonic. For white noise at -15 dB re the talk, Ultra calls 100 percent of a 5 s stretch speech, 23 percent of a 60 s stretch and 3 percent of a 90 s stretch (res_B7.md).
• Amplitude invariance does not hold. Above about -40 dBFS RMS the output is flat. Below it the output falls off (white noise alone in a file: Redux 94 percent speech frames at -40 dBFS, 22 at -60, 0 at -80; Ultra 99, 85, 7; res_B1.txt), because of the 2^-24 guard inside the log of the mel features. That attribution is from the study notes; no result file isolates the guard.
• Moondream's cards say the head exists so that recordings can be cut at pauses into segments of at most 30 s. What it was trained on is not stated. Nothing in the cards claims robustness to noise or music. Moondream's own runtime was not available, so the head wiring is inferred from the documentation and is consistent with the behaviour (a residual variant of the context block breaks Ultra: clean speech F1 81.0 instead of 95.0, res_A2_variants.txt).

What the head does with different noise

Each signal is the only content of a 30 s file (375 frames), share of frames called speech at threshold 0.5 (res_B2.md):

Signal	Redux	Ultra
Digital silence	98.7%	99.7%
Constant tone (1 kHz, -20 dBFS)	0.0%	98.9%
Sine sweep 50 to 7500 Hz	5.9%	45.1%
Clicks, 3 per second	3.5%	46.9%
Clicks, 10 per second	33.1%	94.4%
Hum 50 Hz and harmonics, -25 dBFS	0.0%	0.0%
Music-like chords, -20 dBFS	23.5%	54.9%
Brown noise, -25 dBFS	28.3%	19.2%
White noise, -25 dBFS	98.7%	99.2%
Pink noise, -25 dBFS	97.3%	98.4%
Speech (LibriSpeech, for comparison)	91.5%	88.5%

Redux and Ultra differ by stimulus, and neither is simply worse. Redux reacts to the texture of flat-spectrum noise (white, pink, speech-shaped) and reacts much less to tones, hum, sparse clicks and music. Ultra reacts to level, and also to clicks, sweeps and music, but it stops firing on white and pink noise sooner once the noise is below the speech. A pure tone fires Ultra only when it is alone in a file (98.9 percent above); inside a speech file it gave 0 percent (next table).

How far below the speech the noise must be

In a speech file, with a 30 s stretch of a given noise type, share of stretch frames called speech by the level of the stretch below the speech (6 clips per cell, res_B6.txt):

Stretch type	Redux 10 dB	Redux 20 dB	Redux 30 dB	Ultra 10 dB	Ultra 20 dB	Ultra 30 dB
White	79	53	0	0	0	0
Pink	63	33	0	0	0	0
Speech-shaped	48	43	1	46	0	0
Clicks, 10 per second	16	9	0	64	22	1
Clicks, 3 per second	1	1	0	26	9	0
Music-like	16	10	1	44	8	0
Brown noise, 50 Hz hum, 1 kHz sine	0	0	0	0	0	0

• Brown noise, hum and a pure tone never trigger the head inside a speech file, for either model.
• Redux needs the white or pink stretch to be 30 dB below the speech before it stops (it is still 33 to 53 percent at 20 dB). Ultra stops at 10 dB for white and pink, 20 dB for speech-shaped noise, and 30 dB for clicks and music.
• Alone in a file, white noise stops being called speech below about -55 dBFS for Redux and about -70 dBFS for Ultra (res_B1.txt; the grid has 10 dB steps: Redux 22 percent at -60 and 8 at -70, Ultra 42 at -70 and 7 at -80).

This corrects the earlier statement that false alarms grow with the noise level. That is true for Ultra (stretch in a speech file, white and pink pooled: 0 percent at 20 dB SNR, 14 at 5 dB, 56 at 0 dB). It is not true for Redux, which is already at 35 percent at 20 dB and stays at 59 to 65 percent from 10 dB to 0 dB. And for both models the false-alarm rate is not monotonic in the share of the file that is noise (res_B7.md).

The user-visible cost in transcribe --vad

A 60 s block of inserted audio (white, pink, clicks, music; at -35, -20 and -5 dB relative to the RMS of the talk) was put at 111.7 to 171.7 s of a 300 s excerpt of a real talk. Words that start inside the block are hallucinations, because the block has no speech (res_D.md, 24 runs per model):

Case	Hallucinated words in the block
Without VAD, all 24 runs (12 per model)	0
--vad, Redux	0 in 11 files, 2 for white noise at -20 dB
--vad, Ultra	0 in 9 files, 15, 12 and 3 for music at -35, -20 and -5 dB
--vad, Silero (Ultra decoder, music at -35 dB, the only Silero file transcribed)	20

Example text: "I don't know, I don't know, I don't know" for Ultra on music at -35 dB. The Silero case kept an 18 s segment of music because its 30 s segment ended 18 s into the block (res_D_silero.txt).

Per-frame head calls inside the inserted block (res_D_probs.md): Redux at most 12 percent in any file; Ultra 68 percent for white at -5 dB and 38 percent for music and clicks at -5 dB. At -35 dB the head called 0 percent of the block speech in 5 of the 8 files. Even so, the segments the head produced covered 24 to 60 s of the 60 s block (Redux 24 to 60 s, Ultra 54 to 60 s) and Silero about 18 s.

The cause is the segmenter, not the head. With a 60 s block the head's segments still covered 54 to 60 s of it in nearly every file, even where the head called 0 percent of the block speech (Ultra white and music at -35 dB). A hard 30 s cut can land just before speech resumes, so a piece with a little speech at its edge is kept and decoded. On the two Silero files checked (white at -20 dB, music at -35 dB) Silero dropped about 42 s of the block. This segmenter behaviour deserves its own look. It is not fixed.

So the user-visible cost is small: wasted decoding of segments that are noise, and rarely a hallucinated phrase. With --vad the talk words outside the block differed from the run without VAD by at most 6 words in about 814 (res_D.md). The head does well on real speech: on a 1168 s talk it called speech 97.0 percent (Redux) and 96.3 percent (Ultra) of the time, where Silero called 93.4 percent, and the head and Silero agree with an F1 of 98.0 to 98.3 (res_E2_talk.txt). On synthetic clips with pauses its frame F1 is 95.4 (Redux) and 95.0 (Ultra), and 94.7 and 93.6 with added white and pink noise (res_E.md).

Mitigations we tested

Offline, on a Python reimplementation of the segmenter on a 10 ms grid, with the speech clips and noise of the fusion corpus (res_E.md). F1 is the frame F1 on speech; the noisy set is white and pink noise at 20 to 0 dB. False alarms are the share of frames called speech in noise-only files and in a 30 s stretch embedded in speech. Lower is better for false alarms.

Rule	Redux F1 clean / noisy	Redux false alarms, noise-only / stretch	Ultra F1 clean / noisy	Ultra false alarms, noise-only / stretch
Head, threshold 0.5 (default)	95.4 / 94.7	97.8 / 55.2	95.0 / 93.6	99.4 / 17.3
Head, threshold 0.7	94.8 / 93.9	76.5 / 10.1	94.2 / 91.8	94.2 / 3.6
Head, threshold 0.9	93.9 / 92.0	16.3 / 0.1	92.4 / 87.5	25.8 / 0.0
Head, threshold 0.97	92.5 / 89.4	1.0 / 0.0	90.0 / 80.6	0.5 / 0.0
Head 0.5, minimum speech 1.0 s	94.6 / 94.1	96.6 / 29.0	93.8 / 91.9	99.4 / 6.9
Head 0.5, energy gate (frame at least 25 dB below the file P95)	92.1 / 94.3	97.8 / 53.2	92.0 / 93.3	99.4 / 17.3
Head 0.5, keep a run only if its median logit is at least 2.5 (p about 0.92)	95.4 / 94.7	0.1 / 0.3	94.5 / 90.3	0.0 / 0.0
Silero 0.5 AND head 0.5	93.4 / 91.8	0.0 / 0.0	93.3 / 91.2	0.0 / 0.0
Two-stage (Silero decides, head moves the edges)	95.4 / 94.8	0.0 / 0.0	95.1 / 94.2	0.0 / 0.0
Silero 0.5 alone, for reference	93.9 / 92.4	0.0 / 0.0	93.9 / 92.4	0.0 / 0.0

In plain language:

• Raising the threshold trades recall for fewer false alarms and still leaves noise-only files firing until about 0.97, where speech F1 has fallen by 3 to 5 points (clean) and up to 13 points (Ultra, noisy). At 0.9 most false alarms in a speech file go away, at a cost of 1.5 to 2.6 F1 points on clean speech.
• The energy gate and a longer minimum speech do not help: the gate removes speech (clean F1 86.1 to 94.6, depending on the gate) and leaves noise-only files at 97.8 and 99.4 percent. A minimum speech of 1.0 s helps only for the embedded stretch (Ultra 17.3 to 6.9) and costs about 0.8 to 1.2 clean F1 points.
• The median-logit run gate is the best cheap option. It drops almost all false alarms and costs nothing for Redux on clean speech (95.4) and 0.5 F1 points for Ultra (3.3 on the noisy set). With the run threshold at 3.5 (not in the table) Redux clean F1 is 94.9 and Ultra 93.6. It is now implemented as the opt-in run gate (a median probability of 0.92 is a median logit of 2.5) and was measured on real recordings, see Real recordings: it removes most noise false alarms of the Redux head, not music. On the 1168 s talk it removed 0.2 to 0.3 percent of the head's speech frames at 2.5 (res_E2_talk.txt). In the inserted block test at most 18 percent of the block frames remained, and 0 in 21 of 24 files (res_D_mit.md).
• The two-stage rule needs Silero. It removes the false alarms and keeps the F1 of the head. See the fusion experiment.

Proposed behaviour

The defaults stay as they are. One candidate fix, not in this PR: a guard for digital silence or a constant tone. Our CLI calls 98.7 percent (Redux) and 99.7 percent (Ultra) of the frames of a silent file speech, because it accumulates the normalisation in double and gets exactly zero features. The reference with the same double accumulation gives a speech probability of 0.76 to 0.86 per frame on the same file (res_B2.md column "mean p"). The Hugging Face float32 normalisation leaves rounding noise instead and gives a probability of about 0; that float32 run is not among the committed result files.

Limits of this study

• Moondream's runtime was not available. The head wiring is inferred from the documentation and is consistent with the behaviour, but not checked against it.
• The noise is synthetic: LibriSpeech with generated white, pink, brown, speech-shaped, click, hum, tone and music-like signals, plus one real talk.
• The mitigation numbers come from a Python reimplementation of the segmenter on a 10 ms grid, not from the C++ code.
• The hallucination counts come from one talk and one insertion point. Only one Silero file was transcribed.
• The black-box check of the head against the released runtime was skipped (it needs CUDA or Metal).

Fusing Silero and the head (offline experiment)

The head has the higher recall and Silero has the cleaner decisions. This experiment asked whether combining them beats each alone. Fusion is not implemented in parakeet.cpp. It is an offline experiment: the rules run in a Python port of the segmenter (scripts/vad_bench/fusion/), on probabilities saved from Silero and from the heads. No C++ code, API or option changed.

Update: the gain did not hold on real recordings. The +0.75 (Ultra) and +1.13 (Redux) F1 points below come from synthetic clips. On 59 real recordings, against the best single detector tuned the same way, the tuned fusion gained +0.10 points (Ultra) and +0.46 points (Redux), and it gave no word error rate gain. Fusion was therefore not built. Keep this section as an experiment note and read Real recordings for the current conclusion.

Setup

• Corpus: rebuilt from LibriSpeech test-clean. 342 speech clips (38 sets of 5 utterances, each set in 9 conditions: clean, and white and pink noise at 20, 10, 5 and 0 dB SNR) and 96 noise-only 30 s clips (12 per noise type and level).
• Detectors: Silero ONNX 6.2.3 (run with onnxruntime), the Ultra Q8_0 head and the Redux packed head, through parakeet-cli vad --probabilities.
• All probabilities go to a 10 ms grid by sample-and-hold. All rules use the same post-processing: bridge 0.1 s, minimum speech 0.1 s, minimum pause 0.2 s, no padding.
• Tuned numbers use speaker-disjoint 4-fold cross-validation: the parameters are chosen on three folds and scored on the fourth. All intervals are 95 percent bootstrap intervals over clips.
• Score: frame level precision (P), recall (R) and F1 against the reference, pooled over the 342 speech clips. The reference is the span of each utterance in the synthetic clip, with an energy trim.

The two-stage rule

Silero decides what is speech. The head only changes the edges: it moves the start of a Silero region earlier by up to 160 to 320 ms and its end later by up to 160 ms, but only where the head also says speech, and it fills a gap between two regions if the gap is shorter than 300 to 600 ms and the head calls at least half of it speech. The default setting is 160 ms before, 160 ms after, 300 ms fill; the best setting found on all the clips was 320 ms before, 160 ms after, 600 ms fill. Where Silero finds no speech, the head can never add any. This is why the rule has no false alarms on noise.

Results at the defaults (threshold 0.5)

Pooled over 342 clips, percent.

System	P	R	F1
Silero	99.6	85.6	92.1
Ultra head	98.6	89.4	93.8
Redux head	97.6	92.0	94.7
OR of both (Redux)	97.6	92.1	94.8
AND of both (Redux)	99.6	85.5	92.0
Two-stage (Ultra)	99.0	90.0	94.3
Two-stage (Redux)	98.9	91.2	94.9

Results after tuning (cross-validated)

System	P	R	F1	F1 gain over the best single detector
Silero, tuned threshold	98.3	91.1	94.5
Ultra head, tuned threshold	95.8	93.0	94.4
Redux head, tuned threshold	96.9	92.8	94.8
Two-stage (Ultra)	98.5	92.3	95.3	+0.75 [+0.65, +0.86]
Two-stage (Redux)	98.0	93.9	95.9	+1.13 [+1.00, +1.28]

The gain is measured against the better of the two single detectors with their own tuned threshold (Silero for Ultra, the Redux head for Redux).

Recall when precision must stay at 99 percent or more (the threshold is chosen on the training folds):

System	Recall at P of 99 percent or more
Silero	88.6
Ultra head	87.4
Redux head	89.4
Two-stage (Ultra)	90.6
Two-stage (Redux)	91.5

In plain language

At the defaults on this corpus, for every 100 s of real speech Silero misses 14.4 s, the Redux head 8.0 s and the two-stage rule 8.8 s (the Ultra head 10.6 s). For every 100 s that a detector calls speech, Silero is wrong for 0.4 s, the Redux head for 2.4 s and the two-stage rule for 1.1 s (the Ultra head 1.4 s). The two-stage rule keeps most of the head's extra recall and about two thirds of Silero's precision advantage over the Redux head, and it never fires on noise-only audio. These are the default rows of the table above, expressed in seconds; they are frame counts, not a statement about words.

Rules that did not help

• OR, max and mean of the two probabilities. They inherit the head's false alarms on noise. OR and max call 99.4 percent of the frames of the noise-only clips speech with Ultra (97.8 percent with Redux), the same as the head alone. The mean at the default weights is clean on noise-only clips (0.0 to 0.1 percent) but gained less on speech (F1 93.4 for Redux against 94.9 for the two-stage rule).
• AND and min. No false alarms, but they are no better than Silero: recall 85.5 against 85.6.
• Switching between the two on an estimated noise level. At the defaults it gave the same false alarms on noise-only clips as the head alone (99.4 percent of the frames with Ultra, 97.8 percent with Redux), so it did not remove the problem.
• Logistic regression and gradient boosting on both probabilities and their neighbours. They scored the highest F1 on the synthetic clips (95.1 to 96.3 at threshold 0.5), but they depend on the data they were fitted on. Fitted without noise-only clips, boosting called 95 percent of the frames of noise-only clips speech with Ultra and 60 percent with Redux, and the logistic regression 20 to 25 percent (Ultra) and 5 to 10 percent (Redux). Adding noise-only clips to the training set brought those false alarms down to under 0.3 percent, but the version fitted without them, applied to a real 600 s talk, called about 90 percent of it speech, against 77.5 percent for Silero and 83 to 84 percent for the heads (there was no reference for that talk, so this is a warning sign and not a measured error). The two-stage rule called 84 percent.

Cost

The two-stage rule needs both detectors to run. The cost is about the sum of the two. From the speeds above (Silero about 660x real time and a head about 93x on one thread, shared machine) both together run at about 80x real time. The rule itself is cheap compared with the model runs, but it was not timed separately.

Caveats

• Synthetic read speech with synthetic white and pink noise. Babble, music, reverberation and non-English speech were not tested.
• Part of every tuned gain comes from the reference convention (it counts the pauses inside an utterance as speech and uses an energy trim). A detector that bridges short pauses gains from that. A gain of one F1 point is not a claim about real use.
• The word-time reference of the TED talks was lost, so the rules were not scored on real talks. Only the share of speech per rule on one 600 s talk is reported.
• One corpus, one random seed for the bootstrap, one machine for the probabilities.

Scripts, the small result files and how to reproduce: scripts/vad_bench/fusion/.

Trimming segments and the word filter

Two changes to what transcribe --vad hands to the decoder. Scripts and the full output are in scripts/vad_bench/decoder_guards (results/tables.txt). The options are described in vad.md.

The run was not made on a quiet machine: the load average was between 5 and 50 (see the first line of results/tables.txt). Word error rates and word counts do not depend on load; no timing is claimed.

Setup

• Old: parakeet-cli before the change (the cuts of the previous segmenter). New: the default, each segment trimmed to its speech plus 0.3 s. --vad-trim 0 gave the old output byte for byte on all 45 files that were run both ways (15 per detector).
• Detectors: the Ultra head, the Redux head (dequantized F16), and TDT 0.6B v3 with Silero. The model was Ultra F16 for the head runs.
• Talks: three whole TED-LIUM long-form talks that were not used to choose any setting (5627 reference words). Words are lower-cased, punctuation removed.
• Speech in noise: 4 sets of 6 LibriSpeech test-clean utterances with gaps (about 50 s each, so the segmenter cuts them), clean, with white noise at 5 dB SNR, or with pink noise at 0 dB SNR. 429 reference words per condition.
• Noise block: 90 s of a talk with 60 s of synthetic noise (white, pink, clicks, music-like tones) put in at a quiet point, 30 files. Any word the decoder returns inside the block is an invented word. "Seconds decoded" is the overlap of the cuts with the block, from vad --mode segments.
• Noise alone: 63 files of 30 s (seven noise types, three levels), decoded whole.

Trimming: word error rate, enlarged measurement

The first run (below) had small sets and read a small loss for the Redux head as real. It was repeated on a larger set, with the same trim of 0.3 s against the old cuts (--vad-trim 0):

• 9 whole TED-LIUM talks (21,540 reference words), 4 used to tune and 5 held out.
• About 375 noisy LibriSpeech files (6 utterances with gaps each, white and pink noise at 20, 10, 5 and 0 dB SNR, 3 noise seeds, split by speaker into tune and held out). The rows below use white 5 dB, pink 5 dB and pink 0 dB (15,642 reference words for Ultra and v3, 17,946 for Redux).
• Models: Redux packed, Ultra Q8_0, and TDT 0.6B v3 Q8_0 with Silero F16. The first run used F16 files, so the two runs differ in the models as well as in the size.
• The delta is trim 0.3 minus the old cuts, in WER points (negative means trim is better), with a 95 percent interval from a paired bootstrap over talks and utterances.

Set	Redux head	Ultra head	v3 + Silero
9 talks	+0.08 (-0.03..+0.20)	+0.01 (-0.06..+0.08)	+0.01 (-0.06..+0.10)
5 held-out talks	+0.16 (-0.01..+0.36)	+0.02 (-0.06..+0.11)	+0.03 (-0.08..+0.20)
4 tune talks	-0.01 (-0.14..+0.13)	+0.00 (-0.10..+0.12)	-0.02 (-0.09..+0.04)
noisy speech (white 5, pink 5, pink 0)	-0.12 (-0.57..+0.45)	-0.27 (-0.57..+0.01)	-0.26 (-0.57..+0.02)
pink 0 dB alone (5,982 words)	-0.15 (-0.90..+0.71)

No interval for talks or for noisy speech excludes zero. The Redux loss of the first run (4.39 to 4.51 on talks, 12.35 to 13.29 on pink 0 dB) does not hold up: on 9 talks it is +0.08, and on pink 0 dB it has the other sign. The first run looked worse for two reasons. It was small: its four noisy sets had 429 words each, so one word is 0.23 points and the +0.93 was four words. And Redux is the most volatile of the three: trim 0.3 changes the text of 15.5 percent of its talk segments, against 10.0 percent for Ultra and 7.6 percent for v3, and moving the pad from 0.30 to 0.32 s (a change that cannot matter) changes the text of 13.5 percent of them and the talk WER by -0.05 (0.35 s: -0.07, 16.6 percent). That is the noise floor of this comparison. Of the +18 net errors that Redux gains on talks, none are at the cut edges (0 net) and 18 are in the interior of the segments, where the decoder sees a slightly different input; only 1 of 21,459 words is cut away.

Redux with other trims, against the old cuts (talks / noisy speech with white and pink noise at four levels, 47,856 words), is not monotonic in the trim: 0.1 gives +0.03 / -0.01, 0.2 gives +0.02 / -0.11, 0.3 gives +0.08 / -0.12, 0.5 gives -0.02 / +0.06, and 1.0 gives -0.00 / -0.01. Three other rules (0.5 s before and after, 0.5 s before and 0.3 s after, and trimming only the edges of at least 0.5 s) are within the intervals of 0.3 on every set, and none is better than the default. Padding is nearly free in noise: with trim 1.0 the decoder still gets only 5.7 s (Redux), 13.3 s (Ultra) and 0.6 s (v3) of a 60 s noise block, against 4.6, 12.1 and 0.0 s with 0.3 (old cuts: 33.2, 37.2, 27.0). The 0.3 pad cuts away few words: on noisy speech 14 correct words of 47.8k (Redux), 14 of 16.1k (Ultra) and 19 of 15.5k (v3), counted as words whose time lies outside the kept audio (a lower bound). The median lateness of the detected speech start against the true start is -5 ms for Redux, 79 ms for Ultra and 237 ms for Silero.

Trim also has a real benefit that the first run could not see. With Silero the old cuts drop whole sentences on clean speech when a long segment starts with silence: on the held-out clean files the v3 WER goes from 6.37 to 3.19 with trim 0.3 (-3.19, interval -7.76..-0.14). That interval and the Redux clean interval (-0.37, -0.80..-0.07 on held-out clean) are the only ones at 0.3 that exclude zero, and both favour the trim.

Decision: the default stays 0.3 s. No code or option changed with this measurement; a minimum edge length for trimming (trim_min_sec) was considered and not added.

Limits:

• The noise is synthetic and added to read speech. Real room noise and overlapping speech were not tested.
• The models are not the ones of the first run (see above).
• The truth for where speech starts and ends is loose: it is the span of each utterance in the synthetic file, so a pause inside an utterance counts as speech.
• Only 5 talks are held out, and the clean sets are small (1,350 words held out).
• There is no correction for the many comparisons in the tables. Read a single interval as a range, not as a test.

Tables, the scripts and a note on how to regenerate them: scripts/vad_bench/trim_regression and results/trim_regression_followup.md.

Trimming: word error rate, first run (small sets, superseded)

Set	Detector	Old	Trim 0.3	Change
3 talks	Ultra head	3.68	3.68	+0.00
3 talks	Redux head	4.39	4.51	+0.12
3 talks	v3 + Silero	3.45	3.45	+0.00
clean	Ultra / Redux / v3 + Silero	2.10 / 3.26 / 1.86	1.86 / 3.03 / 1.86	-0.23 / -0.23 / +0.00
white noise 5 dB	Ultra / Redux / v3 + Silero	4.90 / 7.23 / 5.59	4.43 / 6.76 / 5.13	-0.47 / -0.47 / -0.47
pink noise 0 dB	Ultra / Redux / v3 + Silero	6.53 / 12.35 / 7.93	5.59 / 13.29 / 7.69	-0.93 / +0.93 / -0.23

These are the numbers of the first run, kept for the record. The text that went with them said the Redux head loses a little on talks and on pink noise. The enlarged measurement above shows that this was noise: the sets are too small to tell (one word is 0.23 points), so read them as "neutral".

Trimming: the noise block

Detector	Seconds of the 60 s block decoded, old	New	Invented words, old	New
Ultra head	33.8	12.0	0	0
Redux head	29.7	0.3	0	0
v3 + Silero	27.7	0.1	11 (5 files)	0

The seconds are means over the 30 files. The head still passes loud white and pink noise (-5 dB against the speech) as speech, and then the trim cannot remove it: Ultra decodes 52 to 58 s of those blocks. The Redux head also fires on digital silence. The words the decoder invented were rare in this run (Ultra and Redux gave none, in the block or in 63 noise-only files), so the trim's benefit here is mostly seconds saved, and for v3 the 11 invented words.

Word filter at 0.5

Check	Ultra head	v3 + Silero
WER on the 3 talks, trim 0.3 / plus filter	3.68 / 3.68	3.45 / 3.45
WER on speech in noise (12 files, 1287 words): off / 0.5 / 0.7 / 0.9	3.96 / 3.96 / 3.96 / 6.06	4.90 / 4.90 / 4.90 / 16.08
Words dropped at 0.9 on those files	31	164
Words dropped at 0.5 on all files with real speech	0 of 15375	3 of 15406
Invented words in the 5 blocks that had them (old cuts, filter alone)	no events	11 -> 1, 0 of 1484 other words lost
Invented words in the 63 noise-only files	0	1 -> 0 (v3)

At 0.5 the filter is free on these sets and it removes most of the few invented words there were. At 0.9 it costs real words, most on v3. Hence 0.5 is the suggested value.

Limits

• The noise is synthetic. The event counts are small: 11 invented words in 5 files, all from v3. A filter that removes 10 of 11 is a weak estimate of a rate.
• The filter was not run against invented words on Ultra, Redux or an RNN-T model (none occurred), and not at all on a CTC model of 0.6B (a unit test runs it on one fixture). The drop_punct_only option for CTC rests on the 110M CTC head, not measured here.
• A single lone invented word between real speech was not tested.
• Silero followed by the head (the two-stage rule), GPU backends and the quantized files other than the F16 ones were not tested.
• The trim changes transcripts of long audio through the VAD paths slightly; the numbers above are from three talks and 12 clips per condition.

Real recordings

The earlier sections use synthetic clips and a few talks. This one repeats the main questions on real recordings with human labels: is a head better than Silero, is fusing them worth it, what do the false alarms cost, and does the choice of detector change the word error rate. It also measures the run gate (see vad.md), the one change that came out of it.

Scripts, the recording list and the result tables are in scripts/vad_bench/real_recordings/.

In plain language:

• On speech-dominant audio the choice of detector does not change the word error rate. All systems are within 0.17 points on clean TED talks.
• The Redux head has the best default frame F1 (92.8) and loses the least speech. Silero is the most precise and the only one that stays quiet on music, noise and environmental sounds.
• The heads call most of an hour of music or noise speech. The run gate removes most of the noise false alarms of the Redux head and recovers about 80 percent of the word error rate lost on recordings with inserted music and noise. It does not remove music.
• Fusing Silero and a head, the best idea of the synthetic study, gained 0.10 (Ultra) and 0.46 (Redux) F1 points over the best single detector and nothing in word error rate. It was not built.

Setup

Part	Data	Size
Speech	VoxConverse dev and test (36 recordings), AMI far-field sdm (9 meetings), AVA-Speech film clips with human labels (14 clips)	12.1 h of audio, 9.5 h of labelled speech
No speech	MUSAN music (16 files), MUSAN noise (6), ESC-50 environmental sounds (5)	2.6 h
Word error rate	Five TED-LIUM long-form talks, and three of them with non-speech clips inserted at pauses (the composite)	0.8 h

• Splits are by recording. VoxConverse dev and AMI validation tune. VoxConverse test and AMI test are held out. AVA and the non-speech files split by the parity of an MD5 of the id. Held out: 20 speech recordings, 3 of them AMI meetings.
• Detectors: Silero F16, the Ultra head and the Redux head, from parakeet-cli vad --probabilities. Systems are scored on a 10 ms grid. Frame precision, recall and F1 are pooled over the speech recordings. Intervals are 95 percent bootstrap intervals over recordings. A Python replica of the segmenter was checked against the CLI.
• "Untuned" means no parameter was fitted: each detector with its own defaults. "Tuned" means chosen on the tuning split and scored on the held-out split.
• The non-speech files have no speech by construction, so every frame called speech there is a false alarm.

Frame F1, untuned

All recordings, percent.

System	VoxConverse	AMI	AVA	Pooled F1	P	R
Silero, own defaults	96.4	86.7	84.7	90.6 [88.4, 92.3]	97.8	84.4
Ultra head	96.4	89.7	82.7	90.8 [89.3, 92.0]	91.3	90.4
Redux head	97.3	92.2	85.4	92.8 [91.6, 93.7]	92.8	92.7
Fusion (two-stage rule), Ultra, defaults	97.4	89.2	87.2	92.4 [90.5, 93.8]	96.9	88.2
Fusion (two-stage rule), Redux, defaults	97.6	90.4	87.5	92.9 [91.1, 94.3]	96.8	89.3
Ultra head, run gate 0.92	96.4	85.4	85.4	90.2 [88.4, 91.7]	96.5	84.7
Redux head, run gate 0.92	96.9	90.8	86.2	92.5 [91.0, 93.7]	97.0	88.3

Reading it: Silero is precise (97.8) and misses speech (recall 84.4). The Redux head has the best F1 at its defaults. The Ultra head is not clearly different from Silero (pooled difference 0.26 points, interval [-0.72, 1.36]). The heads have the higher recall and the lower precision. The gate moves a head toward Silero's precision. Its pooled cost at 0.92 is 0.6 points for Ultra and 0.3 for Redux (paired differences -0.61 [-1.55, 0.39] and -0.32 [-0.98, 0.31], so neither is clearly different from zero). Per domain the gate costs 4.3 (Ultra) and 1.4 (Redux) points on AMI, nothing on VoxConverse for Ultra and 0.4 for Redux, and gains 2.7 and 0.8 points on AVA.

Tuned, held out

Parameters (threshold, min_speech, min_pause, padding, and for the other rules their own) were chosen on the tuning recordings and scored on the held-out ones. Pooled held-out F1, with the parameters chosen:

System	Held-out F1 [95% CI]	Chosen
Silero, tuned	93.4 [91.2, 94.9]	threshold 0.1, min_speech 0.25, min_pause 0.5, pad 0.1
Ultra head, tuned	91.5 [89.4, 93.2]	threshold 0.8, min_speech 0.25, min_pause 0.5, pad 0.2
Redux head, tuned	93.3 [91.5, 94.7]	threshold 0.6, min_speech 0.5, min_pause 0.5, pad 0.1
Fusion, Ultra, tuned	93.5 [91.4, 95.0]
Fusion, Redux, tuned	93.8 [91.8, 95.3]
Redux head with run gate 0.8, tuned	93.9 [92.2, 95.2]	threshold 0.5 (gate 0.8), min_speech 0.25, min_pause 0.5, pad 0.03

Untuned on the same held-out recordings the scores are 89.1 (Silero), 89.3 (Ultra) and 91.4 (Redux), so tuning adds 4.3, 2.2 and 1.9 points. Paired against the best single detector tuned the same way (Silero, for both), the tuned fusion gains 0.10 points [-0.03, 0.26] with Ultra and 0.46 points [0.30, 0.66] with Redux. That is less than the +0.75 and +1.13 of the synthetic experiment. By domain (VoxConverse / AMI / AVA) the gain is -0.07 / -0.29 / +0.58 with Ultra and +0.15 / +0.65 / +0.71 with Redux.

Fusion: not worth a flag

The fusion rule needs both detectors to run. It gains 0.10 points (Ultra) and 0.46 points (Redux) over the best tuned single detector, and it gave no word error rate gain (see below). Fusion is not implemented and there is no flag for it. The synthetic section above stays as an experiment note.

The reference is noisy

Part of what tuning gains comes from how the references are written, not from the detectors:

• In VoxConverse and AMI the frames Silero "misses" are mostly pauses inside annotated turns. The annotators mark a whole turn and the metric counts the pauses in it as speech. Tuning therefore pushes min_pause to 0.5 s (bridge the pauses) and the padding to 0.1 s. That says something about the labelling convention and little about the detector.
• With a collar of 0.1 s around every reference boundary (cells within 0.1 s of a change are ignored), every F1 rises by about 0.7 to 0.9 points and the ranking does not change:

System	F1, no collar	F1, collar 0.1 s	F1, collar 0.25 s
Silero, own defaults	90.6	91.3	91.7
Ultra head	90.8	91.5	91.9
Redux head	92.8	93.5	93.9
Fusion, Ultra, defaults	92.4	93.1	93.6
Fusion, Redux, defaults	92.9	93.7	94.2
Redux head, run gate 0.92	92.5	93.2	93.8

Read an F1 difference of a point or less as within the reference noise.

False alarms on audio without speech

Seconds called speech per hour of audio, with 95 percent intervals over files where given. Music by MUSAN source is in results/extras.md.

System	Music	Noise	ESC-50	Pooled s/h	Pooled regions/h
Silero, own defaults	135	1	1	48 [0, 136]	27
Ultra head	2823	1979	1605	2174 [1882, 2524]	794
Ultra head, run gate 0.92	1625	398	395	826 [523, 1229]	158
Redux head	2123	1599	1236	1685 [1435, 1987]	1013
Redux head, run gate 0.92	644	23	131	269 [122, 482]	85
Fusion, Ultra, defaults	165	2	5	60	24
Fusion, Redux, defaults	164	2	5	59	23

The Silero music figure comes from one MUSAN source (Jamendo, 294 s/h); the other four sources give 0 to 5.

The run gate (keep a run of p >= 0.5 only when the median of its frames is at least the gate) trades speech for false alarms along one curve. F1 is the pooled F1 of the head at threshold 0.5 with bridge 0.1 s, min_speech 0.1 s, min_pause 0.2 s and no padding:

Head	Gate	Pooled F1	Music s/h	Noise s/h	ESC-50 s/h
Redux	off	92.8	2123	1599	1236
Redux	0.8	93.5	1305	639	377
Redux	0.9	92.8	762	79	167
Redux	0.92	92.5	644	23	131
Redux	0.96	91.0	265	4	34
Redux	0.98	88.6	107	1	15
Ultra	off	90.8	2823	1979	1605
Ultra	0.8	91.2	2389	1226	959
Ultra	0.9	90.7	1831	576	488
Ultra	0.92	90.2	1625	398	395
Ultra	0.96	87.7	761	31	178
Ultra	0.98	83.4	339	9	80

Redux at 0.98 reaches 107 / 1 / 15 s/h but costs 4.2 F1 points. Ultra at 0.98 costs 7.4 points and music is still at 339 s/h. The gate is not a music rejector: music runs have a high median like speech. A lower gate costs less speech and removes less: Redux at 0.8 gains 0.7 F1 points and removes 39 to 70 percent of the false alarms, Ultra at 0.8 removes 15 to 40 percent.

What the decoder receives

transcribe --vad hands the decoder the segments from segment_by_vad with the trim at 0.3 s (the default). Seconds of reference non-speech inside the decoded segments, per hour of audio:

System	VoxConverse	AMI	AVA
Silero	273	355	885
Ultra head	317	598	1348
Redux head	296	620	1328
Ultra head, run gate 0.92	258	412	927
Redux head, run gate 0.92	238	426	832
Oracle (reference speech mask)	230	375	833

Speech lost (labelled speech outside the segments, percent, VoxConverse / AMI / AVA): Silero 0.2 / 5.3 / 6.3, Ultra head 0.0 / 0.6 / 0.3, Redux head 0.1 / 0.3 / 0.2, Ultra head with the gate 0.9 / 3.9 / 5.1, Redux head with the gate 0.9 / 2.1 / 7.8. The heads lose the least speech.

Of one hour of non-speech audio, the share the decoder receives (trim 0.3), in percent:

System	Music	Noise	ESC-50
Silero	6	0.03	0.03
Ultra head	95	84	94
Redux head	91	82	94
Ultra head, run gate 0.92	53	14	27
Redux head, run gate 0.92	27	1.1	5.6

A head hands the decoder most of an hour of music or noise. The gate cuts that share by half or more for Redux, and it stays far above Silero.

Word error rate

The ASR models are the ones of the heads (Ultra Q8_0 and the packed Redux), decoding the segments each system gives.

Clean TED talks (5 talks, about 2880 s decoded). The detector does not matter on speech-dominant audio. WER in percent:

ASR model	Head	Silero	Fusion, defaults	Fusion, tuned	Run gate 0.92
Ultra	4.07	4.18	4.18	4.21	4.24
Redux	5.02	5.02	5.02	4.97	4.97

All systems are within 0.17 points (Ultra) and 0.05 points (Redux) of each other, and every paired interval includes zero.

Composite recordings. Three of the talks with 40 s of music, 30 s of noise and 40 s of vocal music inserted at pauses. The reference is the talk, so every word decoded over the inserted audio is an insertion. It is synthetic by construction and has three talks. WER in percent:

ASR model	Head	Silero	Fusion, defaults	Run gate 0.92
Ultra	5.91 [3.66, 8.75]	3.87 [2.91, 4.90]	4.16	4.24 [2.97, 5.79]
Redux	6.94 [4.76, 9.55]	4.76 [3.85, 5.71]	5.02	5.22 [4.25, 6.24]

The gate recovers about 80 percent of the head's penalty against Silero (Ultra: 1.67 of 2.04 points; Redux: 1.72 of 2.18). Silero is still the best on this audio. The decoded seconds are 1137 (Silero), 1377 and 1367 (Ultra and Redux head) and 1160 and 1143 (gate).

The 30 s hard cut

The segmenter cuts at the last pause of a 30 s window, and when there is none it cuts at 30 s inside speech. The heads hard-cut more often than Silero on VoxConverse: 9.7 hard cuts per hour for Silero, 32.0 for Ultra and 35.1 for Redux (0.5 to 7.7 per hour on AMI and AVA).

The measured cost of a forced cut is small. On the five TED talks, cutting Silero's speech mask every 20 s regardless of pauses (142 hard cuts in 0.8 h, against 2 for cutting at pauses) adds 26 word errors with the Ultra model and 28 with the Redux model, which is about 0.18 to 0.2 word errors per forced cut. At 35 cuts per hour that is about 7 word errors per hour, or 0.06 points of WER on talks of about 11 000 words per hour. After the trim, the hard cut is not the main cost.

What the decoder still gets that is not speech is inside the pieces. Edge margins are 44 to 75 s/h. The rest are pauses that stay inside a segment: for the Redux head, pauses under 1 s, from 1 to 5 s and of 5 s or more add up to 140 / 86 / 35 s/h on VoxConverse, 297 / 358 / 102 on AMI and 370 / 535 / 172 on AVA. Cutting at every pause of 2 s or more (instead of only at the last pause of a window) lowers the decoded non-speech by 22 to 59 percent but loses more speech: with the Redux head, speech lost goes from 0.1 / 0.3 / 0.2 to 0.1 / 0.8 / 1.3 percent, and with Silero from 0.2 / 5.3 / 6.3 to 0.5 / 9.0 / 11.1 (VoxConverse / AMI / AVA). Lost speech is a worse error than a decoded pause, so the cut rule is unchanged.

Limits

• DIHARD, CallHome and the MUSAN speech files were not used: their speech labels were not usable for frame scoring.
• WER was measured on TED talks only. The meetings and film clips have no word-level reference here.
• The composite recordings are synthetic (three talks, clips inserted at pauses). They show the effect of long non-speech stretches, not an average over real audio.
• The cut policies were measured by non-speech seconds and lost speech, not by WER.
• The Silero ONNX model was not run in this study. The Silero numbers are from the parakeet.cpp GGUF.
• Held out: 20 recordings, and only 3 AMI meetings, so the held-out intervals are wide.
• The reference noise described above applies to VoxConverse and AMI. Differences of about a point between systems are within it.

When to use which

This is limited to what the numbers above support.

• Stand alone gate, any ASR model, small files. Silero. Its GGUF is 1.3 to 2.2 MB, the command line tool needs about 60 MB, and it ran at about 660x real time on one thread on a shared machine. It gave frame level F1 of 91.7 to 94.4 on the synthetic data and 91.8 to 92.1 on the TED talks.
• Always-on gate, or audio with long stretches without speech. Silero. The heads give false alarms on noise-only audio, and the effect depends on the noise type and level (see the head on noise-only audio); Silero gave none. On real recordings Silero called 48 s of an hour of music, noise and environmental sounds speech, the Ultra head 2174 s and the Redux head 1685 s. Use Silero with its own defaults. Lowering its threshold to 0.2 or 0.3 raises the pooled F1 on real speech from 90.6 to 92.4 or 91.8 and keeps music at or below 199 s/h (noise and ESC-50 at 5 s/h or less); see Real recordings.
• The Redux head, for long recordings that are mostly speech (talks, meetings, interviews). On 59 real recordings it had the best default F1 (92.8 against 90.6 for Silero and 90.8 for the Ultra head), lost the least speech (0.1, 0.3 and 0.2 percent on VoxConverse, AMI and AVA, against 0.2, 5.3 and 6.3 for Silero), and gave the same word error rate as Silero on clean TED talks.
• Recordings with long stretches of music or noise. Silero. Behind a head the decoder receives most of an hour of music or noise, and the word error rate on the composite recordings was 6.94 percent (Redux head) against 4.76 (Silero).
• The run gate, opt-in, for a head. It drops a speech run whose median probability is below the gate. At 0.92 it cut the Redux head's false alarms from 1685 to 269 s/h and recovered about 80 percent of the word error rate lost on the composite recordings, at a pooled cost of 0.3 F1 points (0.6 for Ultra, which gains less). It does not remove music and it is not a noise rejector. See vad.md.
• The Parakeet head when Ultra or Redux is already loaded, on audio that is mostly speech (for example recorded talks before transcription). The head costs no extra model and its scores are close to Silero in the same tests: within about 1.5 F1 points on the synthetic clips at 5 dB and above (Ultra was 0.4 lower in white noise at 5 dB), 2 to 2.5 points higher on the TED talks, and 1.7 to 2.6 points higher at 0 dB pink noise. It has the higher recall and the lower precision. On these clips its start times are closer to the reference.
• The Parakeet head as a stand alone detector. With a slice file (PR 87, in master, not in a tagged release) the head is a 6 to 10 MB file that loads in under 4 ms and needs about 240 to 250 MiB for a 33 s clip, with the same output as the full file. Without it, the head needs the full ASR GGUF (213 MB to 1.4 GB). Silero is still smaller (1.3 MB), uses less memory (13 MiB after the clip) and was about 8 times faster per core (736x against 92x to 95x on 1 thread, on a busy machine). Choose the head for its accuracy on the tests above, Silero for size and speed.
• Silero from whisper.cpp or from parakeet.cpp. On the 120 clips and the 600 s talk the two give the same segments (mask F1 99.92) and the same speed within the noise of a shared machine. Choose by the rest of your stack, not by quality or speed. The probabilities of parakeet.cpp match onnxruntime more closely (mean abs diff 0.00002, 2 flipped frames) than those of whisper.cpp do (0.00224, 309 flipped frames), but this did not change the segment scores.
• Both together. In an offline experiment on synthetic clips, Silero deciding and the head only moving the edges (the two-stage rule) gave 0.75 to 1.13 more F1 points than the best single detector and no false alarms on noise. On real recordings the gain was 0.10 (Ultra) and 0.46 (Redux) points and there was no word error rate gain, so it is not implemented in parakeet.cpp. See the fusion experiment and Real recordings.
• Not supported by these numbers: any claim about GPUs, ARM, other languages, streaming latency, or the effect of the VAD on WER beyond the TED talks and the composite recordings of Real recordings.

How to reproduce

The scripts are in scripts/vad_bench/, with a README that lists the inputs, the commands and the Python packages. No audio and no model files are committed. In short:

1. Fetch the public audio: fetch_data.py, fetch_ted_talks.py.
2. Slice section: vad_slice_bench.cpp (build it inside the PR 87 tree) and slice_bench.py.
3. Heads against Silero: compare_synthetic.py then analyze_synthetic.py; ted_reference.py then compare_ted.py; speed_heads.py.
4. Silero parity: make_clips.py, build wvad.cpp against whisper.cpp, silero_collect.py, silero_analyze.py; speed with speed_silero.py.
5. Long talks: longform_b1.sh.
6. Noise root-cause study: noise_dive/.
7. Segment trim and word filter: decoder_guards/.
8. Real recordings and the run gate: real_recordings/.

Small result files of the runs on this page (tables, per run timings and load logs) are in scripts/vad_bench/results/. The raw per clip predictions are not committed; a re-run regenerates them.