Minimize hotkey-to-final-text latency with provider-aware Rust codecs and parallel execution

[MyButtermilk]

Objective

Minimize Scriber's complete user-visible latency:

hotkey/button activation
    -> microphone ready
    -> authoritative audio captured
    -> provider request/stream completed
    -> final transcript returned
    -> final transcript visibly inserted into the intended target

The primary KPI is no longer encoder time or even stop -> requestStarted. It is:

activation_received_to_final_text_observed_ms

For toggle recording, also retain the critical secondary KPI:

stop_requested_to_final_text_observed_ms

The end marker must prove that the exact expected final transcript is visible in the guarded target application. clipboard_set, paste_requested, an API callback, or first_paste alone is not sufficient.

This issue should pursue the lowest total latency for each exact provider, route, model, workload, recording duration, and supported audio format. It should use the fastest verified Rust implementation for the winning format, including pinned Nightly/SIMD builds where they materially improve the complete path.

The detailed provider-format matrix in the following comment is normative input to this issue:

#18 (comment)

This work depends on the reusable HTTP transport from #16 and should include the OpenRouter STT route from #17 when available.

Product-level optimization rule

Do not choose one universal codec for every provider.

The production objective is:

For every frozen provider + endpoint + model + workload:
    choose the fastest verified representation and implementation
    that preserves transcript quality, reliability, privacy, and compatibility.

Examples:

Azure MAI / mai-transcribe-1.5
    -> benchmark zero-copy WAV, Nightly Rust FLAC, Rust MP3, current FFmpeg MP3
    -> no Opus unless the exact endpoint contract changes

Smallest batch / exact Pulse route
    -> benchmark passthrough, WAV, MP3, FLAC, Ogg/Opus, WebM/Opus

Deepgram async / exact pre-recorded model
    -> benchmark passthrough and documented Opus/MP3/FLAC/WAV choices

OpenAI Transcriptions
    -> benchmark accepted MP3/WAV/WebM paths; do not select unsupported Opus/FLAC

Provider-native realtime routes
    -> use their exact streaming encoding contract instead of forcing a batch file codec

Unknown models, custom endpoints, and unresolved OpenRouter upstream providers must fail closed to a conservative capability set.

Primary and diagnostic timing contract

Extend the existing HotPathTracer and installed text-target benchmark rather than creating an unrelated timer.

Required start markers

activation_received
hotkey_received                 # native Tauri QPC marker for global hotkey
button_received                 # bound UI/native marker for button activation
start_request_dispatched
recording_state_visible
mic_ready
first_audio_frame
first_audible_audio_frame

Required stop/finalization markers

stop_requested
last_audio_frame_captured
last_chunk_sent_to_pipeline
capture_stopped
encoder_tail_started
encoder_tail_completed
request_started
first_request_chunk_sent
last_request_chunk_sent
provider_final_received
transcript_parsed
post_processing_completed       # only when enabled
injection_target_validated
clipboard_set
paste_requested
final_text_observed

Required KPIs

activation_received_to_final_text_observed_ms
hotkey_received_to_final_text_observed_ms
button_received_to_final_text_observed_ms
stop_requested_to_final_text_observed_ms
stop_requested_to_provider_final_received_ms
provider_final_received_to_final_text_observed_ms

Because the primary measurement includes the time during which the user is speaking, benchmark comparisons must use identical authoritative audio duration and content. Also report:

non_speech_overhead_ms = activation_to_final_text - authoritative_fixture_duration

For real user sessions, store only privacy-safe aggregate phase timings; never infer or persist spoken content from timing data.

The existing markers such as hotkey_received_to_mic_ready_ms, stop_requested_to_first_paste_ms, stop_requested_to_provider_final_received_ms, and provider_final_received_to_clipboard_set_ms remain useful diagnostics, but the release decision must use the final observed-text KPI.

Critical-path model

Treat the workflow as a directed acyclic graph rather than a sequential list.

A simplified batch/final-only route should become:

activation received
├─ start/lease microphone immediately
├─ freeze provider + endpoint + model + capability entry
├─ acquire/reuse provider HTTP connection
├─ initialize selected encoder/resampler
├─ capture injection-target guard
└─ update overlay/UI
       ↓
first PCM frame
├─ authoritative PCM spool
├─ VAD/speech diagnostics
└─ selected provider-compatible encoder running during capture
       ↓
stop requested
├─ stop/drain physical capture
├─ finalize only the small encoder tail
├─ clean audio routing / resume prewarm
├─ validate target and prewarm injection path
└─ prepare multipart headers/request metadata
       ↓
encoded artifact or stream ready
└─ stream multipart upload over the already warm connection
       ↓
provider final response
├─ parse/validate final text
├─ finish optional post-processing
└─ revalidate target in parallel where safe
       ↓
insert text
       ↓
observe exact final text in target

For provider-native realtime routes, the network stream runs during capture and there is no batch audio-preparation stage after Stop. The relevant optimization is connection/session readiness, provider finalization, and final text insertion.

Parallel execution requirements

1. Idle and application-start prewarm

Perform non-sensitive, non-billable preparation before the first dictation:

• keep the existing Rust audio sidecar alive;
• prewarm microphone/VAD/SmartTurn resources where already supported;
• create the shared provider HTTP sessions and connection pools from Reuse provider HTTP sessions and add connection-level STT diagnostics #16;
• resolve and cache provider capabilities and selected model metadata;
• preallocate bounded PCM and encoder buffers;
• initialize reusable resampler plans and codec tables where thread-safe;
• warm the native text-injection path without importing legacy fallbacks unnecessarily;
• compile/runtime-dispatch SIMD kernels once rather than on the Stop path.

A provider connection may be proactively warmed only through a documented idempotent/non-billable operation. Never issue a transcription POST merely to warm a connection.

2. Work started at activation

At activation_received, launch independent work concurrently:

• microphone acquisition and first-frame delivery;
• provider route/model freeze;
• HTTP pool/connection acquisition;
• selected encoder initialization;
• resampler initialization if required;
• injection-target identity capture;
• overlay/UI update.

None of these tasks may delay microphone capture unless it is required for correctness.

3. Encode during capture for buffered/final-only routes

Choose the format before the first audio frame from the frozen capability and benchmark policy. Tee PCM into exactly one production encoder while preserving the authoritative PCM path:

capture callback
    -> authoritative PCM/frame-pipe
    -> non-blocking bounded SPSC/ring buffer
    -> dedicated Rust encoder worker
    -> encoded spool/pages/frames

Requirements:

• capture callback never waits on the encoder;
• no authoritative PCM is dropped;
• encoder overload fails the candidate locally and preserves a fallback before any provider POST;
• use a dedicated CPU pool/thread, not the async/network event loop;
• avoid oversubscribing the CPU with VAD, resampling, and many codec workers;
• production encodes one selected format, not every candidate simultaneously;
• benchmark variants run in separate/interleaved trials so redundant codecs do not distort results.

4. Parallel Stop boundary

On Stop, run safe independent tails concurrently:

• capture stop/drain;
• final encoder flush/container close;
• audio-routing cleanup and microphone-prewarm handoff;
• injection-target revalidation;
• request metadata/header construction;
• UI transition to Transcribing.

Use a structured-concurrency owner so failure or cancellation joins every child task and cleans every handle/artifact.

5. Overlap finalization and upload where the format permits

The selected transport should begin consuming bytes as early as the provider contract safely permits:

• WAV: once total PCM bytes are known at Stop, emit a virtual RIFF header followed directly by the authoritative PCM spool; avoid a complete copied WAV file;
• MP3: stream already-produced frames and append the final flush frames; do not require nonessential ID3/Xing metadata for STT;
• Ogg/Opus or WebM/Opus: stream completed pages/clusters and append the terminal page/cluster;
• FLAC: benchmark a streamable/final-header strategy against a seekable encoded spool. Do not delay upload for checksums/metadata that the provider does not require.

Do not start a billable batch transcription POST before the user stops unless that exact provider documents a cancellable streaming-ingest contract and Scriber deliberately implements it. Provider-native realtime sessions are the exception because streaming is their normal contract.

6. Parallel response tail and injection preparation

While awaiting the provider final response, safely prepare:

• guarded target revalidation;
• clipboard snapshot strategy;
• native SendInput/clipboard route readiness;
• post-processing connection reuse when post-processing is enabled.

Insert immediately after the final accepted text is ready. Measure actual target observation, not only injection API completion.

Fastest implementation candidates by format

The phrase “fastest Rust implementation” means the fastest implementation on Scriber's reference Windows systems and fixtures, not the fastest upstream marketing number.

WAV / PCM

Implement a zero-copy or near-zero-copy Rust virtual WAV stream:

44-byte RIFF header + existing PCM spool

This is the no-compression lower bound and likely winner for short recordings on fast uploads.

FLAC

Required candidates:

flacenc stable
flacenc pinned-nightly SIMD/default
rezin-flac parallel long-form
FFmpeg FLAC fast control

The current upstream flacenc reports approximately 575x realtime on stable and 1,309x realtime for its Nightly default benchmark, with a compression ratio close to the reference FLAC level-5 result. These figures use a small music corpus and are hypotheses only; Scriber must remeasure 16-kHz mono speech.

Nightly flacenc is the preferred maximum-speed FLAC candidate if it wins the installed Windows benchmark and passes toolchain, packaging, and reproducibility gates. rezin-flac is a long-form parallel challenger; do not assume many workers help short dictation.

Opus

Required candidates for exact provider/model routes that explicitly accept Opus:

ruopus pure Rust SIMD
libopusenc reference control
optional crime/moosicbox façade controls

ruopus reports encode performance at approximately parity with SIMD libopus at matched complexity, between roughly 560x and 1,088x realtime in its published modes. Benchmark speech-oriented bitrate/complexity settings and full Ogg/WebM container finalization.

Prefer ruopus when it is the fastest qualifying implementation and passes conformance, accuracy, packaging, cancellation, and long-session tests. Retain libopusenc as the reference and safe fallback.

MP3

Required candidates:

mp3lame-encoder in-process control
shine-rs / Shine-derived maximum-speed challenger
current FFmpeg MP3 control

Rust Nightly cannot by itself make the native LAME backend a Rust-native codec, but the in-process wrapper can remove FFmpeg process and pipe overhead. The Shine family reports substantially faster encoding than LAME in its own benchmark, but its simple psychoacoustic model can produce lower-quality/larger MP3 output. It is eligible only if actual STT WER/CER and exact-fixture results remain within the quality guardrail.

The production MP3 implementation is whichever candidate minimizes complete latency while passing quality and licensing gates—not automatically the pure-Rust option.

Original-file passthrough

For file, YouTube, and meeting artifacts, unchanged passthrough is the first candidate whenever the exact provider/model accepts the original format. No encoder can outperform zero re-encoding.

Nightly, SIMD, PGO, and build strategy

Maximum-speed experiments must include a pinned and reproducible Nightly toolchain without making the entire application accidentally depend on unpinned Nightly behavior.

Required build variants:

stable release baseline
pinned Nightly release
pinned Nightly + crate SIMD feature
pinned Nightly + PGO
pinned Nightly + LTO/codegen tuning

Requirements:

• pin the exact Nightly date/version in repository configuration;
• record compiler, crate, feature, and target fingerprints in benchmark artifacts;
• use release builds only;
• evaluate fat/Thin LTO, codegen-units=1, panic strategy, and PGO on Scriber speech fixtures;
• do not ship one target-cpu=native binary that fails on other users' CPUs;
• prefer runtime SIMD dispatch or separately compiled baseline/AVX2 kernels;
• validate SSE2/baseline, AVX2, and any optional higher feature path on real supported machines;
• include the selected codec/toolchain capabilities in sidecar self-test and runtime attestation;
• retain a stable implementation fallback when the Nightly artifact cannot run or fails validation.

Nightly is selected for production only when the complete installed-app path is materially faster and the build is reproducible, supportable, and secure.

Provider-aware selection policy

Use the verified matrix from the linked issue comment and freeze the result per job.

Suggested policy order:

1. Native realtime route:
       use the exact provider streaming contract;
       avoid batch re-encoding.

2. Direct file/YouTube/meeting route:
       pass through the original accepted format unchanged.

3. Buffered/final-only Live Mic route:
       choose the measured fastest accepted format for the provider/model,
       duration bucket, machine capability, and validated network profile.

4. If the chosen encoder is unavailable or fails locally:
       select a pre-approved local fallback before sending any provider request.

5. If no verified representation exists:
       fail locally with an explicit unsupported-format error.

Possible generated choices:

wav_pcm16_virtual
mp3_shine_nightly
mp3_lame_in_process
flac_flacenc_nightly_simd
flac_rezin_parallel
ogg_opus_ruopus
ogg_opus_libopusenc
webm_opus_verified_backend
current_ffmpeg_mp3_fallback

Do not use guessed live bandwidth as a routing signal. Network-specific routing may be added only if a bounded, private, stable signal demonstrably improves p95 without oscillation. Initially derive conservative duration/size crossovers from controlled network profiles.

Benchmark design

Workloads

Run each eligible provider/format/implementation against identical authoritative fixtures:

5 s
15 s
30 s
60 s
180 s
600 s where practical

Include German and English clean speech, moderately noisy speech, silence/near-silence lifecycle cases, and the actual 16-kHz/48-kHz capture paths.

Route classes

Report separately:

provider-native realtime
buffered/final-only Live Mic
file direct upload
YouTube direct upload
meeting finalization
local ONNX

Do not mix these routes into one average.

Network profiles

Where reproducible shaping is available:

fast upload / low RTT
normal broadband
constrained upload
higher RTT / packet-loss tail
IPv4-only
IPv6-only
dual-stack warm reuse

Repetitions

For every production candidate:

• at least one cold call;
• at least 20 warm calls for provider measurements where cost permits;
• interleaved/randomized candidate order;
• connection-created and connection-reused samples reported separately;
• failures/cancellations excluded from successful latency percentiles and reported separately;
• exact same model, endpoint, language, vocabulary, post-processing, and injection target.

Metrics

Primary:

activation_received_to_final_text_observed_ms
stop_requested_to_final_text_observed_ms
non_speech_overhead_ms

Supporting:

hotkey -> mic ready
hotkey -> first audible frame
capture stop/drain
time to first encoded byte
encoder tail/finalization
output bytes and compression ratio
request start -> first byte
upload duration
provider processing
provider final -> target validation
provider final -> final text observed
CPU time during capture and after Stop
peak memory
thread/process count
installer size and startup impact

Quality and correctness:

• exact transcript fixtures where deterministic;
• WER/CER and punctuation comparison where nondeterministic;
• no truncation, clipped tail, leading delay, invalid duration, sample-rate mismatch, or channel error;
• WAV/FLAC exact PCM round-trip;
• MP3/Opus decode through independent decoders;
• exact final text observed in the intended guarded target.

Baselines and hypotheses

Preserve the current production baseline:

PCM spool
    -> post-stop FFmpeg MP3 64 kbit/s
    -> new/current HTTP path
    -> provider final
    -> injection

Initial hypotheses to test, not promises:

• in-process MP3 may remove approximately 100–300 ms from common dictations;
• encoding during capture may reduce local Stop-to-request work by 80–98%;
• complete warm end-to-end gains may commonly be 10–20%;
• Opus-compatible routes on constrained uploads may gain 20–35% or several hundred milliseconds;
• direct WAV may win short recordings and lose longer recordings on constrained uploads;
• Nightly FLAC may make compression cost nearly negligible, but payload size and provider behavior still determine the global winner.

The selected policy must follow measured total latency, not these estimates.

Safety and replay rules

A local encoder or format fallback is allowed only before any billable STT request may have been accepted.

Do not automatically replay audio in another format after:

• connect/read/total timeout with ambiguous request state;
• cancellation after upload begins;
• connection reset after bytes were sent;
• provider 5xx or unknown response;
• provider/model failure where acceptance is uncertain.

Do not race two paid providers or two billable formats to take the fastest response. Provider-native safe transport recovery before request commitment remains the HTTP stack's responsibility.

Parallelism must not weaken privacy:

• no API keys, audio, transcript text, encoded payloads, or personal paths in logs;
• bounded request/flow IDs only;
• temporary artifacts under allowlisted roots;
• deterministic cleanup on success, failure, cancellation, and shutdown.

Implementation ownership

Prefer extending the existing long-lived Rust scriber-audio-sidecar rather than spawning another per-dictation codec process.

Suggested modules:

Frontend/src-tauri/src/audio_codec.rs
Frontend/src-tauri/src/audio_prepare.rs
Frontend/src-tauri/src/provider_audio_capabilities.rs
Frontend/src-tauri/src/latency_markers.rs
src/runtime/provider_http.py              # #16
src/core/provider_audio_formats.py

Suggested Rust boundary:

#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum EncodedAudioFormat {
    WavPcm16,
    Mp3,
    Flac,
    OggOpus,
    WebmOpus,
}

trait StreamingAudioEncoder: Send {
    fn format(&self) -> EncodedAudioFormat;
    fn write_pcm_i16(&mut self, interleaved: &[i16]) -> Result<(), AudioEncodeError>;
    fn finish(self: Box<Self>) -> Result<AudioPreparationArtifact, AudioEncodeError>;
}

Suggested backend feature gates:

codec-mp3-lame
codec-mp3-shine-nightly
codec-opus-ruopus
codec-opus-libopusenc
codec-flac-flacenc
codec-flac-flacenc-nightly
codec-flac-rezin-parallel

No broad default feature may silently add every codec to the installer.

Delivery plan

Stage 0 — end-to-end instrumentation

• extend the native hotkey/button marker contract;
• add final_text_observed with exact target verification;
• preserve current hot-path metrics;
• establish cold/warm current-route baselines.

Stage 1 — capability registry

• encode the documented provider/endpoint/model matrix from the linked comment;
• separate batch formats from realtime encodings;
• add evidence and verification dates;
• fail closed for unknown routes;
• add missing gemini_stt capability coverage.

Stage 2 — maximum-speed local codec lab

• implement zero-copy WAV;
• implement Nightly flacenc and stable control;
• implement ruopus and libopusenc control;
• implement in-process LAME and Shine challenger;
• evaluate PGO/LTO/SIMD and long-form parallel FLAC;
• publish local latency, quality, resource, and packaging results.

Stage 3 — parallel capture pipeline

• initialize selected codec at activation;
• tee PCM through a non-blocking bounded Rust queue during capture;
• overlap Stop cleanup, encoder tail, target validation, and request preparation;
• stream multipart output as early as safely possible.

Stage 4 — provider end-to-end matrix

• direct Azure and every eligible current STT provider;
• OpenRouter separately after Add OpenRouter as a native speech-to-text provider #17;
• cold/warm and shaped-network runs;
• exact observed-text endpoint;
• derive provider/model/duration format policies.

Stage 5 — production rollout

• enable only the winning implementation per provider/route/model;
• retain safe stable/local fallbacks;
• remove or isolate benchmark-only variants;
• complete installed-app, cancellation, failure-tail, packaging, signing, privacy, and support-bundle gates.

Acceptance criteria

Primary outcome

• The primary benchmark begins at the authoritative hotkey/button activation marker.
• It ends only when the exact final transcript is observed in the intended target.
• Identical audio fixtures make full activation-to-final comparisons valid.
• Results report p50, p90, p95, maximum, variance, and failure rate.
• Production chooses the lowest-latency verified route per provider + endpoint + model + workload.

Parallel pipeline

• Provider/session preparation, encoder initialization, target capture, and mic startup overlap where safe.
• Buffered routes encode during capture, not entirely after Stop.
• Capture never blocks on compression and authoritative PCM is never lost.
• Stop cleanup, encoder tail, request preparation, and target validation overlap where safe.
• Multipart upload consumes already-produced bytes without unnecessary whole-file copies.
• Every concurrent child task/thread is owned, cancelled, joined, and cleaned deterministically.

Nightly and codec selection

• Stable and pinned-Nightly implementations are directly compared.
• The fastest qualifying implementation is selected per format.
• Nightly/SIMD/PGO configuration is reproducible and attested.
• Runtime CPU dispatch prevents incompatible binaries.
• MP3, Opus, FLAC, WAV, and passthrough candidates are limited to exact supported provider/model routes.
• Lossy candidates pass STT quality gates; lossless candidates round-trip exactly.

Reliability and safety

• No new capture stalls, dropped frames, malformed files, deadlocks, orphaned processes/threads, or unclosed transports.
• No second billable request is caused by codec fallback after an ambiguous failure.
• No sensitive audio, transcript, credential, filename, or path data is logged.
• Existing provider, replay, installed workflow, release, and packaging tests continue to pass.

Non-goals

• A universal codec for all providers.
• Selecting Nightly because it is newer rather than because total latency is lower.
• Optimizing only encoder microseconds while worsening upload/provider/injection time.
• Encoding several production formats simultaneously and racing them.
• Starting a billable batch request before Stop without an explicit provider streaming contract.
• Guessing Opus support from generic OGG/WebM support.
• Choosing thresholds from intuition rather than measured crossover evidence.
• Replacing provider-native realtime transports with batch-file uploads.

References

1. Minimize hotkey-to-final-text latency with provider-aware Rust codecs and parallel execution #18 (comment)
2. Reuse provider HTTP sessions and add connection-level STT diagnostics #16
3. Add OpenRouter as a native speech-to-text provider #17
4. https://github.com/MyButtermilk/Scriber/blob/main/src/core/hot_path_tracer.py
5. https://github.com/MyButtermilk/Scriber/blob/main/scripts/measure_recording_hot_path_baseline.py
6. https://github.com/MyButtermilk/Scriber/blob/main/src/azure_mai_stt.py
7. https://github.com/MyButtermilk/Scriber/blob/main/Frontend/src-tauri/src/audio_sidecar.rs
8. https://github.com/yotarok/flacenc-rs/blob/main/report/report.stable.md
9. https://github.com/yotarok/flacenc-rs/blob/main/report/report.nightly.md
10. https://docs.rs/crate/ruopus/latest
11. https://docs.rs/libopusenc/latest/libopusenc/
12. https://docs.rs/mp3lame-encoder/latest/mp3lame_encoder/
13. https://github.com/toots/shine
14. https://docs.rs/crate/rezin-flac/latest
15. https://docs.rs/rayon/latest/rayon/
16. https://doc.rust-lang.org/rustc/profile-guided-optimization.html
17. https://doc.rust-lang.org/rustc/codegen-options/index.html
18. Speed up OpenRouter dictation by 22% without forcing IPv4 DevEmperor/DictateKeyboard#210

[MyButtermilk]

Provider audio-format matrix — Context7 and official documentation verification

Verification date: 2026-07-20

This section is normative input for the codec-selection and benchmark work in this issue. It documents the audio formats accepted by every STT provider currently exposed by Scriber, plus the planned OpenRouter STT route from #17.

Context7 was used to cross-check the current official SDK surfaces for Mistral, Deepgram, and ElevenLabs [18–20]. The provider's own endpoint documentation remains authoritative for format acceptance. SDK support, a generic vendor format list, or support on a different endpoint must never be treated as proof that the exact Scriber provider/model route accepts that format.

Interpretation rules

• Batch/direct-file means an endpoint that receives a complete recording, uploaded bytes, or an audio URL.
• Realtime means a persistent stream of raw or containerized audio frames. A realtime encoding is not automatically a valid direct-file format, and vice versa.
• OGG alone does not mean Ogg/Opus. It may mean Ogg/Vorbis or merely an auto-detected Ogg container.
• WebM alone does not mean WebM/Opus unless the provider explicitly identifies the codec or Scriber has route-specific test evidence.
• Opus is marked eligible only where the exact endpoint documentation explicitly names opus, ogg_opus, OGG Opus, or WebM Opus.
• Provider-native support and Scriber implementation support are separate. A provider may accept a format that the current Scriber route does not yet expose.
• Unknown models and custom base URLs must fail closed to a conservative format set until verified.

Legend:

✅ Explicitly documented for the exact endpoint or route
⚠️ Container is documented, but codec subtype is not; probe or route evidence required
❌ Not documented for the route and must not be selected
— Not applicable to this Scriber route

1. Batch and direct-file formats

Scriber provider key(s)	Provider-native batch/direct-file formats	Opus status	Current Scriber state and required policy
soniox, soniox_async	AAC, AIFF, AMR, ASF, FLAC, MP3, OGG, WAV, WebM, M4A, MP4 [1]	⚠️ Official async list names OGG and WebM but not their codecs. Scriber's existing WebM/Opus path is implementation evidence, not permission to map every OGG/WebM file to Opus.	Direct upload exists. Keep the existing verified WebM/Opus route, WAV fallback, and exact route tests. Pass through other formats only after upload tests.
mistral, mistral_async	WAV, MP3, FLAC, OGG, WebM [2]	⚠️ OGG/WebM codec is not specified.	Both current Scriber keys ultimately use buffered/REST transcription; the mistral label is segmented, not provider-native realtime. Do not select Opus without a model-specific probe.
smallest, smallest_async	WAV with PCM/Linear PCM; MP3; FLAC; OGG with Vorbis or Opus; M4A with AAC or ALAC; WebM with Opus or Vorbis [3]	✅ Ogg/Opus and WebM/Opus are explicit.	Direct upload exists. Opus may enter the benchmark and capability registry for the exact Pulse route. Preserve mono requirements and provider limits.
assemblyai	Audio: 3GA, 8SVX, AAC, AC3, AIF, AIFF, ALAC, AMR, APE, AU, DSS, FLAC, FLV, M4A, M4B, M4P, M4R, MP3, MPGA, OGG, OGA, MOGG, OPUS, QCP, TTA, VOC, WAV, WMA, WV. Video: WebM, MTS, M2TS, TS, MOV, MP2, MP4, M4P, M4V, MXF [4]	✅ .opus is explicit.	Async direct upload exists. Prefer original-file pass-through; AssemblyAI explicitly recommends avoiding unnecessary transcoding.
assemblyai_realtime	Provider has a separate pre-recorded API with the formats above, but this Scriber key is streaming-only.	— for file preparation	Do not expose file codec selection through the realtime key. Current streaming path should remain raw PCM16/μ-law according to its own session contract.
deepgram_async	More than 100 formats are accepted. Common documented formats include MP3, MP4, MP2, AAC, WAV, FLAC, PCM, M4A, Ogg, Opus, and WebM [5].	✅ Opus is explicit.	Direct upload exists. Keep the exact MIME type and original bytes where practical. Test uncommon formats before adding them to the static registry.
deepgram	The same provider has pre-recorded support, but the current Scriber key is native streaming.	— for current file route	Do not use the provider's broad pre-recorded list to alter the current Nova streaming transport. Realtime encodings are tracked separately below.
openai_async	MP3, MP4, MPEG, MPGA, M4A, WAV, WebM [6]	❌ No OGG, Opus, or FLAC in the current Transcriptions API file list.	Direct multipart upload exists. MP3 or WAV are the relevant generated candidates; preserve accepted source MP4/MPEG/MPGA/M4A/WebM files.
openai	This Scriber key uses Realtime transcription and has no direct-file route.	— for file preparation	Use only the realtime session input formats documented below.
groq	FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, WebM [7]	⚠️ OGG is listed without an explicit codec.	The provider accepts files, but Scriber currently marks groq as not supporting direct file upload. Keep the current segmented route until a direct adapter and tests are deliberately added. WAV is recommended by Groq for latency; FLAC for lossless size reduction.
azure_mai / mai-transcribe-1.5	WAV, MP3, FLAC; file must be below 300 MB [8]	❌ Opus is not documented.	Direct route exists. The current adapter converts non-MP3 input to MP3; this issue must benchmark WAV, MP3, and FLAC before changing that behavior.
planned openrouter_stt / microsoft/mai-transcribe-1.5	OpenRouter's generic STT list includes WAV, MP3, FLAC, M4A, OGG, WebM, and AAC, but support varies by routed provider [9]. For the MAI model, use the conservative Azure set: WAV, MP3, FLAC.	❌ for the MAI route until the routed provider changes or an exact supported-contract test exists.	Do not infer Opus from OpenRouter's generic list. Store capabilities by OpenRouter model and resolved provider where available.
gemini_stt on the current Gemini audio-understanding/generate-content route	WAV, MP3, AIFF, AAC, OGG Vorbis, FLAC [10]	❌ OGG is explicitly Vorbis on the current route.	Direct file/Files API handling exists in pipeline.py, but gemini_stt is currently missing from src/core/provider_capabilities.py. Add and test the missing capability entry. Do not import broader format enums from a different Gemini API surface into this route.
google	Google Cloud Speech supports route/version-dependent encodings including LINEAR16/WAV, FLAC, MULAW, ALAW, AMR, AMR-WB, OGG_OPUS, WEBM_OPUS, MP3, SPEEX_WITH_HEADER_BYTE, MP4_AAC, and M4A_AAC [11].	✅ OGG_OPUS and WEBM_OPUS are explicit in supported API versions.	Scriber currently exposes Google Cloud as streaming-only. Freeze the actual API version/model and keep current PCM streaming until a separately tested file route is implemented.
gladia, gladia_async	Audio: AAC, AC3, EAC3, FLAC, M4A, MP2, MP3, OGG, OPUS, WAV. Video: 3G2, 3GP, AVI, FLV, M4V, Matroska, MOV, MP4, WMV [12]	✅ standalone Opus is explicit for pre-recorded STT.	Direct upload exists. Opus may be selected for the pre-recorded route, but not for Gladia V2 realtime.
speechmatics_async	The last publicly indexed official SaaS/legacy documentation lists AAC, AMR, FLAC, M4A, MP3, MP4, MPEG, OGG, and WAV; some container documentation also lists MOV [13].	⚠️ OGG codec is not identified.	Direct batch upload exists. Treat this row as provisional and revalidate against the live SaaS API schema during implementation. Until then, advertise only formats exercised by Scriber tests.
speechmatics	Provider batch formats exist, but the current Scriber key is realtime-only.	— for current file route	Keep file capability disabled unless a direct batch route is selected. Realtime input is documented separately below.
modulate, modulate_async	Velma multilingual batch: AAC, AIFF, FLAC, MP3, MP4, MOV, OGG, Opus, WAV, WebM. The English fast endpoint accepts Opus only [14].	✅ Opus is explicit.	Both realtime and batch adapters exist. Store the endpoint/model in the capability key because the English-fast and multilingual endpoints have materially different format contracts.
elevenlabs	Scribe v2 accepts AAC, AIFF, OGG, MP3, OPUS, WAV, WebM, FLAC, MP4/M4A and the documented MIME aliases; supported video includes MP4, AVI, Matroska/MKV, MOV, WMV, FLV, WebM, MPEG, and 3GPP [15].	✅ audio/opus is explicit for batch Scribe.	Scriber currently exposes ElevenLabs Live only. Do not claim direct-file support until a Scribe v2 batch adapter is implemented, but retain this inventory for future pass-through planning.
onnx_local	No provider upload. The ONNX model receives decoded PCM from Scriber's local media-preparation path.	—	Do not run MP3/Opus compression for local inference. Source-container support is determined by the bundled local media decoder/FFmpeg profile, not by the ONNX ASR model.

2. Realtime and streaming input formats

These capabilities must be stored separately from direct-file formats.

Scriber provider key	Documented realtime input	Opus eligibility	Scriber policy
soniox	Auto-detected AAC, AIFF, AMR, ASF, WAV, MP3, FLAC, OGG, WebM; raw signed/unsigned PCM at 8/16/24/32 bits in LE/BE forms; float PCM 32/64 LE/BE; μ-law; A-law [1].	⚠️ OGG/WebM codec not explicit in the format type.	Current Soniox native stream should remain on its tested transport. Do not substitute a generated Opus container without a realtime-specific test.
mistral when a future native realtime route is used	The realtime API accepts an audio byte stream with an audio_format contract [2].	Model/SDK enum must be frozen and tested before enabling Opus.	Current Scriber mistral is segmented REST and must not be relabeled as native realtime.
smallest	linear16, linear32, alaw, mulaw, opus, ogg_opus; sample rates 8, 16, 22.05, 24, 44.1, and 48 kHz [3].	✅ raw Opus and Ogg/Opus are explicit.	Opus is eligible for a dedicated realtime benchmark, independent of the batch decision.
assemblyai_realtime	Raw mono PCM16 with sample rate supplied in the session; μ-law is also documented for telephony paths [4].	Do not enable Opus in Scriber without an exact current session-contract test.	Current PCM16 path remains the conservative default. A file such as MP3 must be decoded before streaming.
deepgram	Raw encodings include LINEAR16, LINEAR32, FLAC, A-law, μ-law, AMR-NB, AMR-WB, Opus, Ogg-Opus, Speex, and G729. Flux explicitly accepts WAV/Linear16, Ogg/Opus, and WebM/Opus containers [5].	✅ for models/endpoints that explicitly support it; Flux is explicit.	Scriber currently uses Nova-3 streaming. Do not copy Flux capabilities to Nova without exact model evidence.
openai	PCM 16-bit, 24 kHz, mono by default; G.711 μ-law and G.711 A-law are supported session formats [6].	❌ Opus is not a current Realtime transcription session input.	Keep PCM16/G.711 handling separate from batch file preparation.
google	The API supports raw/container encodings including LINEAR16/WAV, FLAC, μ-law, A-law, AMR/AMR-WB, OGG_OPUS, WEBM_OPUS, MP3 and additional version-specific AAC formats [11].	✅ Ogg/Opus and WebM/Opus are explicit.	Current Pipecat route should continue using its configured tested encoding; any Opus change needs exact API-version and SDK tests.
gladia	V2 realtime accepts only wav/pcm, wav/alaw, and wav/ulaw, with PCM bit depths 8/16/24/32 [12]. V2 explicitly removed MP3, FLAC, OGG/Vorbis, and Opus.	❌ for Gladia V2 realtime.	Opus is batch-only for the current capability model.
speechmatics	Publicly indexed official legacy realtime material documents raw PCM f32 LE, PCM s16 LE, and μ-law; container support in legacy appliances is broader [13].	⚠️ No current public SaaS Opus contract was verified.	Revalidate the current SaaS SDK schema. Keep the existing tested PCM route until then.
modulate	Self-describing WAV, MP3, OGG, FLAC, WebM, AAC, and AIFF are auto-detected; raw PCM is configured with format, sample rate, and channel count [14].	⚠️ Batch explicitly accepts Opus, while the streaming list names OGG/WebM containers without guaranteeing their codec subtype.	Keep the current tested stream representation. Add Ogg/Opus or WebM/Opus to realtime selection only after an exact endpoint test.
elevenlabs	Scribe v2 Realtime consumes streamed audio; the official SDK surface and examples use PCM audio with an explicit sample rate [15][20].	Do not equate batch audio/opus support with realtime input support.	Current Scriber live path stays on tested PCM. Batch Opus becomes relevant only if a separate batch adapter is added.
groq, azure_mai, openrouter_stt, gemini_stt, onnx_local	No provider-native realtime file/container route is used by the corresponding Scriber keys.	—	Their buffered/final-only behavior must use the direct-file matrix or local PCM path.

3. Canonical Scriber capability representation

Avoid storing only file extensions. The capability registry should distinguish container, codec, sample representation, and route.

Suggested canonical values:

class AudioInputFormat(StrEnum):
    WAV_PCM16 = "wav_pcm16"
    WAV_PCM24 = "wav_pcm24"
    WAV_PCM32 = "wav_pcm32"
    RAW_PCM16 = "raw_pcm16"
    RAW_PCM32_FLOAT = "raw_pcm32_float"
    MP3 = "mp3"
    FLAC = "flac"
    AAC = "aac"
    AIFF_PCM = "aiff_pcm"
    M4A_AAC = "m4a_aac"
    M4A_ALAC = "m4a_alac"
    OGG_VORBIS = "ogg_vorbis"
    OGG_OPUS = "ogg_opus"
    WEBM_OPUS = "webm_opus"
    WEBM_VORBIS = "webm_vorbis"
    MP4_AUDIO = "mp4_audio"
    MULAW = "mulaw"
    ALAW = "alaw"
    AMR_NB = "amr_nb"
    AMR_WB = "amr_wb"
    SPEEX = "speex"
    G729 = "g729"

Suggested route-scoped capability object:

@dataclass(frozen=True)
class ProviderAudioInputCapabilities:
    provider: str
    route: str
    model_family: str
    batch_formats: frozenset[AudioInputFormat]
    realtime_formats: frozenset[AudioInputFormat]
    direct_passthrough_formats: frozenset[AudioInputFormat]
    preferred_lossy_format: AudioInputFormat | None
    preferred_lossless_format: AudioInputFormat | None
    max_upload_bytes: int | None
    evidence_kind: Literal[
        "official_endpoint_docs",
        "official_sdk_schema",
        "scriber_integration_test",
        "bounded_live_probe",
    ]
    evidence_reference: str
    verified_at: date

The key must include at least:

(provider, route, model_family)

Examples:

("azure_mai", "llm_speech_batch", "mai-transcribe-1.5")
("openrouter_stt", "audio_transcriptions", "microsoft/mai-transcribe-1.5")
("gladia", "v2_live", "solaria-1")
("gladia_async", "v2_pre_recorded", "default")
("modulate", "velma_2_streaming", "multilingual")
("modulate_async", "velma_2_batch", "multilingual")
("modulate_async", "velma_2_batch_english_vfast", "english-fast")

4. Provider-aware selection implications

The production selector should follow this order:

1. Exact original-file pass-through when accepted by the frozen route/model.
2. Direct WAV when accepted and below the measured WAV/compression crossover.
3. Ogg/Opus or WebM/Opus only when explicitly supported by that exact route/model.
4. MP3 as the broad lossy compatibility path where accepted.
5. FLAC as the lossless compressed path where accepted and benchmarked.
6. Local failure before upload when no verified representation exists.

Initial Opus-eligible batch candidates

Explicit provider documentation currently makes Opus a valid benchmark candidate for:

smallest / smallest_async
assemblyai
deepgram_async
gladia / gladia_async pre-recorded
modulate / modulate_async batch
elevenlabs Scribe batch, if a future Scriber adapter is added
Google Cloud routes supporting OGG_OPUS or WEBM_OPUS

Route-specific or additional Scriber evidence is still required for:

Soniox WebM/Opus — existing Scriber route evidence, vendor list is container-generic
Mistral OGG/WebM — codec subtype not documented
Groq OGG — codec subtype not documented
Speechmatics OGG — codec subtype not documented and current SaaS docs need revalidation
Modulate realtime OGG/WebM — stream list is container-generic
OpenRouter non-MAI models — verify each model and routed provider

Opus must remain disabled for:

Azure MAI / MAI Transcribe 1.5
OpenRouter MAI while Azure is the routed provider
OpenAI Transcriptions API
Gemini current audio-understanding/generate-content route
Gladia V2 realtime

5. Required implementation and test changes

• Extend ProviderCapabilities or add a dedicated route-scoped audio-format registry.
• Add the currently missing gemini_stt capability entry.
• Keep direct-file formats separate from realtime encodings.
• Keep generic containers separate from exact codec/container pairs.
• Add a test that rejects ogg as proof of ogg_opus unless explicit evidence exists.
• Add a test that rejects webm as proof of webm_opus unless explicit evidence exists.
• Freeze provider, route, model, and chosen format with every persisted transcription job.
• Make original-file pass-through the first preparation decision.
• Reject unknown custom models conservatively instead of inheriting another model's formats.
• Record evidence source and verification date with every capability entry.
• Add request-shape tests for every format Scriber claims to support.
• Add one bounded, non-sensitive live provider probe before promoting a previously untested format.
• Never perform an automatic second billable request in another format after an ambiguous failure.
• Add a documentation test ensuring every Settings STT provider has either a batch format entry, a realtime entry, or an explicit local/no-upload classification.

6. Documentation caveats

1. Provider format contracts change. This matrix should be regenerated or reverified when provider SDKs, API versions, default models, or Scriber route implementations change.
2. Context7 is valuable for checking current SDK types and method surfaces, but the endpoint documentation governs server acceptance.
3. Speechmatics is the least certain row because the exhaustive publicly indexed source is legacy documentation. It must be revalidated against the current SaaS OpenAPI/SDK before broadening Scriber pass-through.
4. Gemini exposes broader audio MIME types on newer API surfaces, including an audio/opus enum in the Interactions API. Scriber's current gemini_stt implementation uses the audio-understanding/generate-content path, whose documented list is narrower. Capabilities must follow the implemented endpoint, not the broadest vendor endpoint.
5. Deepgram support varies by streaming model. Flux explicitly supports Opus variants; Scriber's current Nova route must not inherit Flux-only capabilities.
6. OpenRouter routes can change upstream providers. Cache or persist enough route evidence to explain the selected format, but do not log credentials, audio, transcripts, or sensitive filenames.

Sources

1. https://soniox.com/docs/stt/async/async-transcription and https://soniox.com/docs/sdk/node-SDK/reference/types
2. https://docs.mistral.ai/resources/known-limitations and https://docs.mistral.ai/studio-api/audio/speech_to_text/realtime_transcription
3. https://docs.smallest.ai/waves/v-4-0-0/documentation/speech-to-text-pulse/pre-recorded/audio-formats and https://docs.smallest.ai/waves/documentation/speech-to-text-pulse/realtime-web-socket/audio-formats
4. https://www.assemblyai.com/docs/faq/what-audio-and-video-file-types-are-supported-by-your-api and https://www.assemblyai.com/docs/streaming/guides/stream_prerecorded_file_realtime
5. https://developers.deepgram.com/docs/supported-audio-formats and https://developers.deepgram.com/docs/encoding
6. https://platform.openai.com/docs/guides/speech-to-text and https://platform.openai.com/docs/api-reference/realtime
7. https://console.groq.com/docs/speech-to-text
8. https://learn.microsoft.com/en-us/azure/ai-services/speech-service/mai-transcribe
9. https://openrouter.ai/docs/guides/overview/multimodal/stt and https://openrouter.ai/microsoft/mai-transcribe-1.5/providers
10. https://ai.google.dev/gemini-api/docs/audio
11. https://cloud.google.com/speech-to-text/docs/reference/rest/v1/RecognitionConfig and https://cloud.google.com/speech-to-text/docs/reference/rpc/google.cloud.speech.v2
12. https://docs.gladia.io/chapters/limits-and-specifications/supported-formats and https://docs.gladia.io/api-reference/v2/live/init
13. https://legacy.docs.speechmatics.com/en/cloud/introduction and https://legacy.docs.speechmatics.com/en/real-time-appliance/api-v2/
14. https://docs.modulate.ai/quickstart
15. https://elevenlabs.io/docs/overview/capabilities/speech-to-text
16. https://github.com/MyButtermilk/Scriber/blob/main/src/core/provider_capabilities.py
17. https://github.com/MyButtermilk/Scriber/blob/main/Frontend/client/src/pages/Settings.tsx
18. https://context7.com/mistralai/client-ts
19. https://context7.com/deepgram/deepgram-js-sdk
20. https://context7.com/elevenlabs/elevenlabs-js