MOSS-TTS lab: German evaluation and latency benchmarks

An independent evaluation of the OpenMOSS/MOSS-TTS model family for German text-to-speech, zero-shot voice cloning, and multi-turn latency.

Tested on an NVIDIA GeForce RTX 5070 Ti (16 GB VRAM) with PyTorch 2.9.1 (CUDA 12.8) on Windows 11.

---

Executive summary and key findings

Model	Architecture	Size	VRAM footprint	Avg. RTF (warm)	German viability verdict
MOSS-TTS-Realtime	MossTTSRealtime	1.7B backbone + codec	~11.1 GB	~2.97 (Raw PyTorch)	Limited suitability for German: Functional baseline synthesis, but prone to Index-0 homograph phonetic shifts, EOS repetition loops, and reference-opening echoes.
MOSS-TTS-Local-Transformer	MossTTSLocal	1.7B backbone + codec	~12.5 GB	~6.5 (Non-realtime)	Not suitable for German voice agents: High generation latency (21 to 24 s for 3.5 s audio) and phonetic misinterpretations on hyphenated compound subwords.

---

Empirical findings for German

Official documentation highlights multi-language support and sub-second latency targets. Testing the model on German text reveals several specific behaviors:

1. Index-0 homograph ambiguity and context priming

German words spelled identically to English words (dies, gift, rat, wand, hell, fast, kind) are mispronounced in English (/daɪz/ instead of /diːs/) when they appear at the very start of a prompt. At index 0, the text attention window has no preceding context and defaults to English token probabilities from pretraining. Preceding the homograph with a full German sentence (Guten Tag, hat der Test funktioniert? Dies ist...) fixes the pronunciation because the attention heads lock into German phonetics.

2. System prompt instructions are ignored

Adding explicit German directives to the system prompt (Always speak in German with German pronunciation) produces no change in pronunciation. MossTTSRealtime is a direct text-to-audio token predictor, not an instruction-tuned model. It does not follow prompt directives.

3. Repetition loops from missed EOS tokens

During voice cloning with reference audio ending in silence, the model occasionally repeats the target sentence multiple times in succession. The autoregressive decoder misses the <|eos_audio|> stop token and cycles through the KV-cache.

4. Reference opening echo

When target text shares an identical opening phrase with reference audio (such as Guten Tag), the model often reproduces the continuation of the reference recording rather than synthesizing the provided text.

5. Hyphenated compound glitches in Local-Transformer

In MOSS-TTS-Local-Transformer, hyphenated compound words like MOSS-Labor split into ["MOSS", "-", "Labor"]. The model then reads Labor with English pronunciation (/ˈleɪbər/) instead of German /laˈboːɐ̯/.

---

Benchmark results

Tested using benchmarks/benchmark_latency_rtf.py with deterministic seeds across 3 warm repetitions per prompt length.

Hardware and runtime environment

• GPU: NVIDIA GeForce RTX 5070 Ti (15.9 GB VRAM, Blackwell architecture)
• CUDA / PyTorch: PyTorch 2.9.1+cu128 (CUDA 12.8, SDPA attention enabled)
• Runtime: HuggingFace transformers (v5.0.0) with PyTorch eager mode (uncompiled, unquantized baseline without vLLM-Omni or TensorRT-LLM)
• Operating System: Windows 11 (audio I/O routed via soundfile / libsndfile 1.2.2)

Benchmark table

Aggregate mean across 3 test runs (9 warm iterations):

Benchmark Category	Words	Audio Duration	Generation Time	Decoding Time	Total Latency	Real-Time Factor (RTF)	Peak VRAM
Model & Codec Init	-	-	-	-	9.02 s	-	11.11 GB
Cold Start (Short)	10	3.8 s	11.81 s	0.33 s	12.14 s	3.21	11.11 GB
Warm: Short (1 Sentence)	10	3.8 s	11.06 s	0.12 s	11.18 s	2.97	11.11 GB
Warm: Medium (3 Sentences)	25	14.0 s	40.41 s	0.45 s	40.86 s	2.93	11.29 GB
Warm: Long (Paragraph)	58	26.7 s	79.42 s	0.79 s	80.20 s	3.00	11.29 GB

Context on RTF and decision on further optimization: Across all 3 benchmark runs with raw HuggingFace transformers and PyTorch eager execution, MOSS-TTS-Realtime averaged an RTF of ~2.97 (about 3x slower than real-time playback). While optimized runtimes like vLLM-Omni (using FlashAttention and C++ kernel fusion) can lower latency to real-time levels (RTF < 0.5), we did not run further optimization benchmarks here. Because the synthesis has fundamental phonetic and stability issues in German (Index-0 homographs, repetition loops, and echo patterns), faster inference does not resolve the underlying quality limits for German voice agents.

---

Multi-turn dialogue session state

To keep voice timbre, speaking rhythm, and German phonetic context consistent across multi-turn sessions, MossTTSRealtimeStreamingSession preserves the GPU KV-cache between turns:

from mossttsrealtime.streaming_mossttsrealtime import MossTTSRealtimeStreamingSession

# 1. Initialize session once on the GPU
session = MossTTSRealtimeStreamingSession(model, tokenizer, codec, ...)

# 2. Sequential multi-turn dialogue loop
for turn_idx, turn in enumerate(dialogue):
    session.reset_turn(
        user_text=turn["user_text"],
        assistant_text=turn["assistant_text"],
        include_system_prompt=(turn_idx == 0),  # System and voice prompt only on turn 0
        reset_cache=(turn_idx == 0),             # Keep GPU KV-cache across turns
    )
    
    # 3. Stream text deltas into audio chunks
    session.push_text(turn["assistant_text"])

---

Repository structure

moss_tts_lab/
├── benchmarks/
│   ├── benchmark_latency_rtf.py  # Latency, RTF, throughput and VRAM benchmark runner
│   └── README.md                 # Benchmark suite documentation
├── data/
│   ├── audio_data/
│   │   └── README.md             # Guidelines for speaker reference recordings
│   └── output/
│       ├── benchmarks_01/        # Benchmark run 1 audio and report
│       ├── benchmarks_02/        # Benchmark run 2 audio and report
│       └── benchmarks_03/        # Benchmark run 3 audio and report
├── legacy_scripts/
│   ├── common.py                 # Shared helpers, soundfile audio I/O and ENV path resolvers
│   ├── run_smoke_test.py         # Minimal GPU viability and WAV generation test
│   ├── voice_clone.py            # Zero-shot voice cloning benchmark (Berlin vs. Austria)
│   ├── homograph_test.py         # Homograph pronunciation and system prompt diagnostic
│   ├── language_tag_test.py      # MOSS-TTS-Local-Transformer language tag evaluation
│   └── README.md                 # Detailed documentation for all test suites
├── tests/
│   └── test_legacy_common.py     # Automated unit tests for audio loading and path logic
├── pyproject.toml                # Pinned dependencies and PyTorch cu128 repository index
├── .env.example                  # Template for local model weight storage paths
└── .gitignore                    # Protects private voice recordings and secrets

---

Quickstart

1. Installation

This project uses uv for dependency management:

# Clone the repository
git clone https://github.com/your-username/moss-tts-lab.git
cd moss-tts-lab

# Clone upstream MOSS-TTS (required by local pyproject.toml editable source)
git clone https://github.com/OpenMOSS/MOSS-TTS.git

# Synchronize virtual environment with PyTorch CUDA 12.8 and dev tools
uv sync

# Configure your local model paths
cp .env.example .env

2. Download model weights and configure paths (.env)

Download model weights from Hugging Face:

# Download MOSS-TTS-Realtime (1.7B) and MOSS-Audio-Tokenizer
hf download OpenMOSS-Team/MOSS-TTS-Realtime --local-dir /path/to/OpenMOSS-Team--MOSS-TTS-Realtime
hf download OpenMOSS-Team/MOSS-Audio-Tokenizer --local-dir /path/to/OpenMOSS-Team--MOSS-Audio-Tokenizer

# Optional: MOSS-TTS-Local-Transformer
hf download OpenMOSS-Team/MOSS-TTS-Local-Transformer --local-dir /path/to/OpenMOSS-Team--MOSS-TTS-Local-Transformer

Set the paths to your local model directories in .env:

MOSS_REALTIME_MODEL_PATH=/path/to/OpenMOSS-Team--MOSS-TTS-Realtime
MOSS_CODEC_PATH=/path/to/OpenMOSS-Team--MOSS-Audio-Tokenizer
MOSS_LOCAL_MODEL_PATH=/path/to/OpenMOSS-Team--MOSS-TTS-Local-Transformer

3. Run tests and benchmarks

# 1. Run unit tests (no GPU required):
uv run pytest

# 2. Run GPU smoke test:
uv run python legacy_scripts/run_smoke_test.py

# 3. Run latency and RTF benchmark:
uv run python benchmarks/benchmark_latency_rtf.py

# 4. Run homograph and voice cloning diagnostics:
uv run python legacy_scripts/homograph_test.py
uv run python legacy_scripts/voice_clone.py

# 5. Run Local-Transformer language tag evaluation:
uv run python legacy_scripts/language_tag_test.py

---

Voice privacy

To prevent voice cloning misuse, private reference audio recordings and cloned outputs are excluded from this repository via .gitignore. Add your own 24 kHz mono reference .wav files into data/audio_data/ to test voice cloning locally (see data/audio_data/README.md).

---

Models and licenses

Component	Hugging Face repository	License	Description
MOSS-TTS-Realtime	OpenMOSS-Team/MOSS-TTS-Realtime	Apache 2.0	1.7B Transformer backbone for streaming dialogue
MOSS-Audio-Tokenizer	OpenMOSS-Team/MOSS-Audio-Tokenizer	Apache 2.0	24 kHz RVQ neural audio codec (12.5 fps)
MOSS-TTS-Local-Transformer	OpenMOSS-Team/MOSS-TTS-Local-Transformer	Apache 2.0	Non-streaming local model with language tag support
Lab Codebase	This repository	Apache 2.0	Diagnostic test suites, benchmarks, and utilities

Both the upstream code and model checkpoints are published under the Apache 2.0 License.