A realtime streaming speech server (Orpheus 3B on vLLM, SNAC codec, FastAPI websocket) and an audio
tokenization pipeline (parquet → decode → VAD → SNAC encode → shards), built and measured on a single
NVIDIA L4. Every number in this README is read from a file in results/, which records the GPU,
model, vLLM version and date of each run (vLLM 0.32.0, torch 2.13.0+cu130).
- Single-stream decode is memory-bandwidth-bound: W4A16 plus a 4-bit lm_head took Orpheus 3B from 36.5 to 123.3 tok/s (realtime needs 82.0) with WER within noise of bf16.
- Serving: 6 realtime streams per L4 (from 2) and 137 ms to first audio (from 247); the remaining limit is vLLM and SNAC time-slicing the GPU across processes.
- Time to first audio is not time to first speech: about 0.6 s of lead-in silence (finding and proposed fix below).
- Pipeline: 53.4 h of LibriTTS-R tokenized at 6.68 h of audio/min on one L4, bound by the GPU encoder; fp16 rejected because it changes tokens.
| metric | result | source (results/) |
|---|---|---|
| Single-stream decode speed | 123.3 tok/s (8.11 ms/token, RTF 0.67) vs 36.5 tok/s for bf16 | step1d_overhead.json, step1b_precision.json |
| Time to first audio, 1 stream | 247 → 137 ms p50 (138 ms p95 after) | baseline_sweep_levels.csv, step6c_final_sweep.json |
| Time to first audio, 6 streams | 187 ms p50 (214 ms p95) | step6c_final_sweep.json |
| Max realtime streams per L4 | 2 → 6 (p95 RTF 0.996, no gaps); 5 with one level of margin (p95 RTF 0.979); 7 fails (p95 RTF 1.038) | optimizations.csv, step6c_final_sweep.json |
| WER vs bf16 (4-bit lm_head, step 5) | 2.9% vs 2.5%: +0.40 pts [-0.62, +1.73], within noise | quality_baseline.json |
| Tokenization pipeline throughput | 6.68 hours of audio / min on one L4 (GPU stage alone: 6.9) | pipeline.json |
| Tokens per audio hour (SNAC 24 kHz, 7 tokens per frame) | 297,483 | pipeline.json |
Decode speed per precision (batch size 1, one stream, no server; realtime needs 82.0 tokens/s):
| precision | ms / token | tokens / s | RTF (generation only) | ≥ 82.0 tok/s needed | status |
|---|---|---|---|---|---|
| bf16 | 27.4 | 36.5 | 2.25 | no | baseline |
| fp8, W8A8 (CUTLASS) | 18.0 | 55.5 | 1.48 | no | rejected: not realtime |
| fp8, weight-only (Marlin) | 16.2 | 61.7 | 1.33 | no | rejected: not realtime |
| W4A16 (GPTQ, group 128) | 11.0 | 91.0 | 0.90 | yes | first realtime config |
| W4A16 + 4-bit lm_head | 8.1 | 123.3 | 0.67 | yes | served default |
Samples are in results/samples/ (GitHub offers a download or preview for each WAV).
step6c_first_chunk.wav: the final server configuration (W4A16 + 4-bit lm_head, batched SNAC, early first chunk) streaming a paragraph: "Streaming speech synthesis is mostly a race against the list…"quality/w4a16_lmhead_s04_seed0.wav: the 4-bit lm_head model reading a date and time from the quality set: "The meeting is on March 3rd, 2025, at 4:30 in the afternoon."quality/bf16_s04_seed0.wav: the same sentence and seed from the bf16 model, for comparison.
Serving path. The websocket server and the SNAC decoder share one process; vLLM's engine core runs in its own process (Orpheus 3B, W4A16 + 4-bit lm_head). All concurrent streams share one vLLM engine (continuous batching, 7 audio tokens per frame) and one batched SNAC decoder (one call for all streams, PCM16 at 24 kHz).
flowchart TB
C[Client] -->|text| WS[Websocket server]
WS --> V[vLLM engine]
V -->|token ids| FA[Chunk planner]
FA -->|frame windows| SN[Batched SNAC decoder]
SN -->|PCM16 audio| WS
WS -->|audio chunks| C
Tokenization pipeline. CPU worker processes decode (WAV bytes) and trim (webrtcvad); one GPU process encodes in length-bucketed batches; token ids are written as zstd parquet shards.
flowchart LR
P[Parquet] --> D[Decode] --> VAD[VAD trim] --> E[SNAC encode] --> T[Token ids] --> S[Shards]
Single-stream decode speed, including the rejected fp8 variants, is in the precision table above; the ladder
lists the serving-side changes, in order, including the ones that were rejected. "Max realtime streams" is the highest load level with p95 RTF below 1.0, no gaps
and no errors, tested on that row's levels (screens test 4, 6 and 8 streams only). The full data is in
results/optimizations.csv.
| step | change | outcome | max realtime streams | TTFA p50 at 1 stream (ms) | why |
|---|---|---|---|---|---|
| 4 | w4a16_lmhead + per-stream SNAC | baseline | 2 | 247 | One SNAC call per window on one worker thread; the worker saturates and throughput flattens. |
| 6a | BatchedChunkDecoder: one SNAC call for all streams' pending windows | adopted | 4 | 245 | One SNAC call for every stream's pending window. SNAC no longer saturates; vLLM and SNAC now contend for the GPU. |
| 6b | growing later chunks, cap 4 (buffer-aware; first chunk unchanged) | rejected | 4 | — | At the limit there is no playback-buffer slack, so chunks stay small; mixed window lengths split SNAC batches; small gaps at 6 streams. |
| 6b | growing later chunks, cap 8 (buffer-aware; first chunk unchanged) | rejected | 6 | — | No capacity gain; savings only appear at loads that already pass. |
| 6b | SNAC weights/activations in fp16 (eager) | rejected | 4 | — | Each new (batch, window length) shape costs seconds of cuDNN setup; unwarmed shapes hit it live and TTFA p95 spikes. Audio was fine. |
| 6b | SNAC CUDA graphs | rejected | 6 | — | 2-4× faster alone, but in the server SNAC time is mostly waiting for a GPU time slice shared with vLLM's process. |
| 6b | fp16+CUDA graphs / torch.compile (fp32 or fp16) | not screened | — | — | Not screened: it shortens SNAC kernel time, which is not what limits the server. |
| 6b | engine core in the server process | not possible | — | — | vLLM's async engine always runs its core in a child process; MPS did not start in the test container. |
| 6b | --snac-max-wait-ms 1 | rejected | 4 | — | Batches did not grow (windows already arrive together); the wait only adds latency and gaps. |
| 6b | --snac-max-wait-ms 2 | rejected | 4 | — | Same as 1 ms: no larger batches, more gaps. |
| 6b | final 6b config = 6a batched SNAC | adopted | 6 | 247 | Fine sweep of the 6a configuration: 6a's coarse sweep had skipped levels 5-7. |
| 6c | first chunk waits for 3 frames: frame 0 emitted with 2 right frames | superseded | 6 | 190† | Works, but superseded by the 2-frame first chunk. |
| 6c | first chunk waits for 2 frames: frame 0 with 1 right frame; next frames ramp from 1 to 2 right frames as the client buffer allows | adopted | 6 | 135† | Smallest first chunk with clean audio; time to first audio drops at every load level. |
| 6c | first chunk waits for 1 frame (frame 0 with no right context) | rejected | — | — | Audio not clean: the first frames have no right context. |
| 6c | SNAC decode on the CPU (fp32, 12 cores) | rejected | — | — | The CPU cannot keep up with the windows per second that 8 streams need. |
| 6c | final 6c config = 6b final + first chunk waits for 2 frames | adopted | 6 | 137 | Server default. Lower time to first audio at every level; capacity unchanged, margin at 6 streams thinner. |
† Time to first audio from the 10-request, 1-stream screen recorded in the notes column of results/optimizations.csv.
Decoding is memory-bandwidth-bound; W4A16 plus a 4-bit lm_head crossed realtime. At batch size 1 every generated token reads all the weights once, and speed tracks bytes read: bf16 (6.2 GiB of weights) decodes at 36.5 tok/s, fp8 (3.73 GiB) at 55.5, W4A16 (2.31 GiB) at 91.0. Realtime needs 82.0 tok/s per stream (12.2 ms per token), so only the 4-bit configurations qualify. A profile of the W4A16 decode step (10.8 ms, GPU 99% busy, no idle gaps) showed the bf16 lm_head, a single matrix-vector product over the 156,940-entry vocabulary, taking 3.8 ms of it (35%). Quantizing it too (GPTQ, calibrated on the model's own generated tokens, lm_head untied from the embedding) brought the step to 8.11 ms: 123.3 tok/s, 3.4× bf16. Word error rate stayed within noise of bf16 (see Quality).
A measurement artefact was found and corrected (step 1d). Earlier runs reported W4A16 at 56.1 tok/s
(17.8 ms/token) and pointed to unexplained overhead. The profile showed the GPU was busy 99% of the time, and
the cause was the measuring script itself: it redirected the engine's stdout/stderr to read the startup log,
which added CPU work to every engine step. That work is hidden when a GPU step is long (bf16 was unaffected) and
exposed when it is short, so all the quantized speeds were too low. The script was fixed and every affected
number re-measured (91.0 tok/s for W4A16). The "fp8 is inefficient at batch 1" hypothesis that rested on the
inflated numbers was withdrawn. Details: results/step1d_profile.md.
SNAC decoding was the first serving bottleneck, and batching fixed it. With one SNAC call per stream window on one worker thread, the baseline load test passed only 2 concurrent streams: the SNAC worker was 100% busy from 8 streams on and total throughput flattened near 6.7 audio-seconds per second. Batching all streams' windows into one call lifted the throughput ceiling to 16.6 (2.5×) and the max to 4 streams in the coarse sweep; the fine sweep of the same configuration later showed 6.
The current limit is cross-process GPU time-slicing. After batching, neither vLLM nor SNAC is saturated alone: at 8 streams SNAC is only 0.51 busy, and vLLM alone needs 8.97 ms per step (step 6a benchmark). Inside the server the same step takes 12.21 ms, because vLLM's engine core and the SNAC decoder are separate processes that take turns on the GPU. That lands on the 12.2 ms realtime budget. Making SNAC cheaper (fp16, CUDA graphs, larger chunks, waiting to form bigger batches) left vLLM's per-token time unchanged, since the time lost depends on how often SNAC runs rather than on how much each call does. Running both in one process or under MPS would address it; neither was possible in the test environment (see the ladder).
Time to first audio is not time to first speech. Orpheus begins each utterance with a lead-in of near-silence;
in the 60 step 5 clips for the served config audible speech starts a median of about 0.6 s into the audio (docs/DECISIONS.md, step 6c).
Cutting time to first audio from 247 ms to 137 ms at one stream (smaller first chunk) therefore does not
bring speech forward by the same amount. Proposed fix (not built): hold back generated frames until the first
frame above a loudness threshold and start streaming from there, which docs/DECISIONS.md estimates would bring
first speech forward by about 0.3 s at one stream. The audio would be unchanged apart from the dropped silence.
The tokenization pipeline is bound by the GPU encoder, and fp16 was rejected because it changes tokens. The
GPU stage runs at 6.9 hours of audio per minute and the slowest CPU stage (VAD) has about 10× that, so adding CPU
cores does not help. The SNAC encoder runs at the full 24 kHz sample rate and is mostly elementwise work
(61% of GPU time versus 17% in matrix-multiply convolutions), so it is memory-bound. torch.compile on the
encoder fuses those operations: 1.24–1.26× faster with 99.4–99.9% of codes identical, adopted together with
a padded-sample batch budget. fp16 autocast was +13% faster but changes 3–8% of the codes; because the codes are the
training data, a faster encoder that silently alters the dataset is not an acceptable trade. Prefetching on a
second thread made things slower (Python's lock starves the GPU launch loop) and was turned off.
| run | audio (h) | end-to-end (h / min) | GPU stage capacity (h / min) | GPU busy | peak GPU memory (GiB) |
|---|---|---|---|---|---|
| baseline (16 clips / batch) | 12.3 | 5.48 | 5.76 | 94% | 22.1 |
| tuned: 4M-sample batches, pinned, prefetch, compile | 12.3 | 5.27 | 5.39 | 77% | 6.6 |
| tuned without prefetch | 12.3 | 6.14 | 6.42 | 86% | 6.5 |
| tuned, 8M-sample batches | 12.3 | 6.1 | 6.47 | 86% | 22.2 |
| full split, tuned without prefetch | 53.4 | 6.68 | 6.9 | 94% | 8.0 |
Method. src/speechserve/quality_sentences.txt holds 30 fixed sentences (short, numbers and dates, names,
punctuation and questions, long, technical, two tongue twisters), none in the GPTQ calibration set. Each config
synthesizes every sentence with voice tara for 2 seeds (60 clips per config) through the server's own decode
path. Clips are transcribed with Whisper large-v3 and scored with jiwer (WER, CER) after Whisper's English text
normaliser. Because that normaliser treats numbers inconsistently (for example "4:30" against "4.30"), a
format-insensitive WER that also ignores number and ordinal formatting is reported next to it. Differences from bf16
use a paired bootstrap over sentences (10000 resamples, 95% interval).
| config | clips | WER mean | WER mean, format-insensitive | CER mean | Δ WER vs bf16 [95% CI] | Δ WER vs bf16, format-insensitive [95% CI] |
|---|---|---|---|---|---|---|
| bf16 | 60 | 2.5% | 1.6% | 0.9% | reference | reference |
| W4A16 | 60 | 4.2% | 1.4% | 1.0% | +1.75 pts [-0.14, +4.30] | -0.24 pts [-2.49, +1.60] |
| W4A16 + 4-bit lm_head (step 5) | 60 | 2.9% | 0.8% | 0.7% | +0.40 pts [-0.62, +1.73] | -0.75 pts [-2.28, +0.30] |
| W4A16 + 4-bit lm_head, final server config (step 6c) | 60 | 3.1% | 0.9% | 0.9% | not run against bf16 (see below) | — |
Result. The 4-bit lm_head config scores 2.9% against 2.5% for bf16: +0.40 pts [-0.62, +1.73], and the interval includes zero. On the format-insensitive metric the figures are 0.8% and 1.6%. The final server configuration (early first chunk, batched decoder) was re-scored against the step 5 run of the same model: +0.24 pts [-0.53, +1.01], within noise. The seed-to-seed change in WER for the same sentence is 1.0 to 2.5 points, larger than the config differences.
Caveats. This harness can only detect differences of roughly ±2 WER points (docs/DECISIONS.md, step 5). WER measures intelligibility, not
naturalness, prosody or codec artefacts; no listening test or MOS model was run (example clips are in
results/samples/). One voice, English only; Whisper itself also makes transcription errors that count against the TTS.
Most of the clips above 20% WER are formatting disagreements, not mispronunciations, and every config mangles the
same tongue twister, so that is a model property, not a quantization one.
docs/SCALING.md (generated by scripts/scaling_math.py) translates the single-L4 measurements to a larger
scale, with every assumption labelled. At the measured 6 streams per L4, 100 concurrent streams need
17 GPUs and 1,000 need 167 (20 and 200 with one level of margin). Tokenizing 2,000,000 hours
of audio is about 4,990 L4-hours at the measured pipeline rate. One epoch over that data (5.95e+11 tokens) is about
2,005 GPU-hours for an 800M model and 12,032 for 4.8B at an assumed 40% MFU. Memory per GPU under DDP, FSDP, HSDP
and tensor parallelism is worked out in the document; no multi-GPU training was run.
All results were produced on Colab with an L4 runtime. Large files (models, clips, shards) live on Drive under
/content/drive/MyDrive/speech-infra and never in git.
- Launcher. Open
notebooks/00_colab_launcher.ipynbon an L4 runtime. It checks the GPU, mounts Drive, logs in to Hugging Face and clones the repo. Colab secrets:HF_TOKEN(the Orpheus models are gated; accept their terms on Hugging Face first) andGH_TOKEN(only needed for a private clone or to push results). Tokens are never written to files. - Environment.
bash scripts/setup_colab.shinstalls the pinned stack fromrequirements.txt(vLLM 0.32.0, torch 2.13.0+cu130), applies the cuDNN fix described indocs/DECISIONS.md, and checks that SNAC decodes on the GPU. - Precision and decode-speed measurements (steps 1 to 1d):
python scripts/step1_generate.py --precision bf16 # also fp8, fp8_kv, fp8_marlin, w4a16 python scripts/step1d_overhead.py lmhead_w4a16 # one variant per process - Quantization.
python scripts/make_calib.py # calibration set from bf16 generations (Drive) bash scripts/setup_quant_venv.sh # isolated venv for llm-compressor /content/venvs/quant/bin/python scripts/quantize_w4a16.py # W4A16 /content/venvs/quant/bin/python scripts/quantize_w4a16.py --lm-head # W4A16 + 4-bit lm_head (default preset) - Server (own tmux window):
python scripts/run_server.py. Defaults are the final configuration: presetw4a16_lmhead, batched SNAC decoder, first chunk after 2 frames. - Load test (final sweep behind the headline table):
The step 4 baseline used
python scripts/run_loadtest.py --levels 1 4 5 6 7 8 10 12 16 --duration 60 --out results/step6c_final_sweeppython scripts/run_server.py --decoder per_stream --first-wait-frames 4and the default levels. - Quality.
python scripts/run_quality.py --decoder per_stream --first-wait-frames 4 # step 5, all three configs python scripts/run_quality.py --presets w4a16_lmhead --run step6c_first_wait2 --decoder batched \ --first-wait-frames 2 --out results/step6c_quality --baseline-csv results/quality_baseline_clips.csv - Tokenization pipeline.
python -c "from speechserve.pipeline.download import download; download()" # LibriTTS-R train-clean-100, local disk python scripts/run_pipeline.py --label full --workers 4 --budget-batches --max-batch-samples 4000000 \ --pin --compile --copy-to-drive python scripts/validate_shards.py - Derived files (CPU only):
python scripts/scaling_math.py(docs/SCALING.md and results/scaling.json),python scripts/make_figures.py(results/figures),python scripts/make_readme.py(this file). - Tests (CPU only):
pip install -e . numpy pytest && pytest. Tests that need a GPU or vLLM are skipped with a reason.
Docker. A Dockerfile for the server is included (CUDA 13 base image, versions pinned from requirements.txt,
the cuDNN fix, model directory and preset from environment variables). It has not been built or run: the
environment this project was developed in has no Docker daemon. Treat it as untested; the base image tag in
particular should be confirmed before building.
docker build -t speechserve .
docker run --gpus all -p 8000:8000 -e HF_TOKEN -v /path/to/models:/models:ro speechserve
/path/to/models must contain orpheus-3b-w4a16-lmhead/ from step 4.
- One GPU, one machine. Every measurement is a single NVIDIA L4 on Colab, with the load generator on the same machine. Capacity at 6 streams is at the edge (p95 RTF 0.996); 5 streams leaves margin.
- Quality evidence is narrow. One voice (
tara), English only, 30 sentences, WER only (no naturalness measure), quantization calibrated on a small set of the model's own outputs. - Time to first speech is dominated by the lead-in silence; the proposed fix is not implemented.
- Cross-process GPU time-slicing (the serving limit) is not resolved: in-process serving and MPS could not be tried.
- Pipeline: one dataset (53.4 hours, 33,113 clips) on one L4, read from local disk; remote storage and other GPUs not measured. Encoded tokens differ slightly from an uncompiled encoder (see findings).
- No multi-GPU or multi-node training was run;
docs/SCALING.mdis arithmetic on stated assumptions, not measurement, and says nothing about any particular organisation's systems. - Docker image not built. The GitHub Actions workflow runs only the CPU tests; GPU and vLLM paths are verified
by the Colab runs recorded in
results/, not by CI. - Model weights, audio datasets and shards are not in the repository.
MIT, see LICENSE. The Orpheus and SNAC models carry their own licenses (check them before reuse); LibriTTS-R is CC BY 4.0.