Self-hostable STT evaluation toolkit for comparing speech-to-text providers on quality (WER/chrF), streaming latency, cost, and a dictation-oriented vibe score.
This is not a Whisper demo. It is a reproducible harness for builders choosing STT for dictation and voice-agent systems — CLI + local Docker service + PR-friendly reports.
PyPI package name: stt-eval-bench (the name stt-bench was already taken on PyPI). CLI command remains stt-bench.
pip install stt-eval-bench
stt-bench benchmark run --provider mock --force
stt-bench report --format markdown
stt-bench results leaderboardNo API keys required for mock runs.
From source / editable install:
git clone https://github.com/BrickerP/stt-bench.git
cd stt-bench
pip install -e ".[dev]"| Surface | Description |
|---|---|
| CLI | compare, listen, benchmark, report, results |
| Local service | stt-bench serve or docker compose up — FastAPI + SQLite |
| Providers | mock, whisper, deepgram, assemblyai (pluggable) |
| Reports | Markdown suitable for PR comments |
docker compose up --build -d
curl -s http://127.0.0.1:8766/healthz
curl -s http://127.0.0.1:8766/versionDetails: docs/service.md
cp .env.example .env # OPENAI_API_KEY / DEEPGRAM_API_KEY / ASSEMBLYAI_API_KEY
stt-bench listen 1089-134686-0000_noisy
stt-bench compare --all31 samples in data/manifest.json (LibriSpeech subset + one noisy clip).
stt-bench samples
stt-bench dataset validateWER · chrF · punctuation F1 · TTFT · RTF · cost · vibe score — see stt-bench math.
See CONTRIBUTING.md and docs/providers.md.
Maturity / Codex-for-OSS execution gate: docs/OSS_MATURITY.md.
Architecture: DESIGN.md.