Writeup: Scoring spoken English offline
Offline pronunciation and fluency scoring for read-aloud English. Give it a WAV and the text the speaker was reading; it returns a JSON report — per-phoneme goodness-of-pronunciation (GOP), fluency metrics (pace, pauses, hesitations, choppiness), and a pitch/energy plot.
Everything runs locally. The default path pulls in only numpy, soundfile, and matplotlib, and finishes in a few seconds on the bundled sample. No model download, no network.
This is a portfolio prototype. It reimplements the technique behind a private pronunciation coach — not its data or product — with generic framing and a synthetic sample so it runs anywhere.
Judging how something was said is harder than judging what was said. speechgauge produces a separate, inspectable signal for three things:
- pronunciation — which sounds were off, per phone
- fluency — pace, pauses, hesitations, choppiness
- prosody — pitch movement and range
With micromamba:
micromamba env create -f environment.yml
micromamba activate speechgauge
python -m speechgauge score samples/sample.wav \
--script "The quick brown fox jumps over the lazy dog. She sells seashells by the seashore." \
--json-out report.json --plot-out prosody.pngOr with pip:
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m speechgauge score samples/sample.wav --script "..." --json-out report.jsonOutput:
speechgauge report
====================================================
duration : 5.0s
transcript : reference-script (estimated word timing)
GOP backend : heuristic
pronunciation : mean word GOP 66.0 | flagged 4 ['fox', 'lazy', 'dog', 'she']
fluency : 82.3/100 (pace 219.7 wpm fast, long stops 1, hesitations none, choppiness no)
prosody : mean F0 200.09 Hz, pitch variation 3.82 st, voiced 66%
--json-out writes the full report (schema below); --plot-out writes the
pitch/energy PNG:
flowchart TD
WAV[WAV file] --> IO["audio_io — numpy DSP:<br/>RMS · STFT centroid · ZCR · autocorr F0"]
SCRIPT[reference script] --> G2P["g2p — CMU dict + LTS fallback"]
IO --> VAD["vad — energy-gate speech/silence timeline"]
IO --> PROS["prosody — F0 to semitone stylization"]
IO --> ALIGN["alignment — word + phone time spans"]
G2P --> ALIGN
STT["optional local STT"] -. real word timings .-> ALIGN
VAD --> FLU["fluency — pace · pauses · hesitations · choppiness"]
ALIGN --> GOP["gop — heuristic or neural, per-phone"]
IO --> GOP
GOP --> REP["report + plot — JSON · prosody PNG"]
FLU --> REP
PROS --> REP
The script is the pronunciation target. The default path does not transcribe — word and phone timings are estimated by spreading the script across the detected speech span, and the report marks them as estimated. Point it at a local STT service (below) to replace the estimate with real word timings.
Canonical phones for each script word come from the CMU Pronouncing Dictionary, with a letter-to-sound fallback for out-of-vocabulary words. Each phone is placed on the timeline and scored by one of two backends.
heuristic (default): for each phone, read its frames' voicing, relative energy,
spectral centroid, and zero-crossing rate, and compare them to an expected profile
for the phone's manner class (vowel, stop, sibilant, fricative, nasal,
approximant, affricate). The fit is z-scored within the utterance, so a phone is
flagged when it matches its target worse than the rest of the same speaker's read.
No trained model and no dependencies beyond numpy — and correspondingly limited
(see below).
neural (--gop-backend neural): a wav2vec2 phone-CTC model gives frame
posteriors; the canonical phone sequence is force-aligned with torchaudio, and
each phone is scored with the Hu-2015 log-ratio
mean_t(logP(canonical) − logP(best)). A word takes its worst phone. Needs the
[neural] extras and downloads an Apache-2.0 model on first use. Falls back to the
heuristic if the extras are missing.
F0 is tracked by autocorrelation and converted to semitones (the Hz reference cancels out for range and variation). The contour is median-filtered to drop tracker spikes, octave jumps are clamped, and it is lightly smoothed. That stylized contour is what the plot draws and what the pitch-variation number is computed from.
Pauses come from the VAD speech/silence timeline rather than word gaps, so the metrics work without a transcript. Leading and trailing silence is trimmed. Silences are tiered (pause ≥250 ms, long stop ≥400 ms, freeze ≥700 ms). From the timeline it derives pace (WPM and band), pausing, hesitations (silent freezes always; filler words when a transcript is available), and choppiness. The four combine into one score.
The bundled sample is a short synthetic clip so the demo needs nothing else. For
real evaluation use
speechocean762
(mispeech/speechocean762) — English L2 speech with expert phoneme-level
pronunciation scores, the standard benchmark for GOP-style systems. Pass a clip
and its transcript to score, and use the accuracy labels to calibrate the flag
thresholds. It's a download, not bundled.
Endpoints are read from environment variables (see .env.example); nothing is
hard-coded.
SPEECHGAUGE_STT_URL— an OpenAI-compatible transcription server (for example faster-whisper / speaches). Adds real word timings and filler detection. On the sample it raises mean word GOP and drops a false flag.SPEECHGAUGE_STT_MODEL— the transcription model id.SPEECHGAUGE_GOP_MODEL— the wav2vec2 model id for the neural backend.
If the service is unreachable, the run falls back to the offline path.
- The default GOP is a heuristic proxy, not a trained acoustic model. It flags
phones whose broad acoustics deviate from the rest of the read, so it produces
some false flags on clean speech and misses subtle substitutions. Use
--gop-backend neuralfor research-grade scoring. - Estimated alignment is coarse. Without real word timestamps, phones are spread proportionally across the speech span, so per-phone timing (and GOP) is approximate. A local STT service tightens it.
- Pace uses speaking time with silence excluded, so WPM reads higher than a wall-clock rate. The bands account for that but are not calibrated to any population here.
- Pitch tracking is a small autocorrelation tracker, not probabilistic YIN. Octave errors are smoothed, not eliminated.
- Thresholds are literature-informed priors, not tuned on labelled data. Calibrate on speechocean762 before trusting absolute numbers.
- Prototype, not production.
speechgauge/ the package (audio_io, vad, g2p, alignment, prosody,
fluency, gop, gop_neural, stt, report, plot, cli)
samples/ bundled sample.wav + committed sample_report.json / prosody.png
environment.yml micromamba environment (offline default deps)
requirements.txt pip equivalent
.env.example optional local-service configuration
Batın Örene — PhD researcher in AI (speech, evaluation, responsible AI). Related work: pronunciation assessment (PPG–DTW + GOP fusion). GitHub: batinium
{ "schema": "speechgauge/report@1", "input": { "duration_s": 5.0, "script": "..." }, "transcript": { "source": "reference-script (estimated word timing)", "timestamps_estimated": true }, "pronunciation": { "summary": { "backend": "heuristic", "mean_word_gop": 66.0, "flagged_word_count": 4, "flagged_words": ["fox", "..."] }, "words": [ { "word": "fox", "gop": 61.2, "flagged": true, "phones": [ { "phone": "F", "gop": 41.0, "z": -1.6, "flagged": true, "start_ms": 600, "end_ms": 720 } ] } ] }, "prosody": { "mean_f0_hz": 200, "f0_std_semitones": 3.8, "voiced_ratio": 0.66 }, "fluency": { "overall_score": 82.3, "pace": { "wpm": 220, "band": "fast" }, "pausing": { "long_stop_count": 1, "silence_ratio": 0.10 }, "hesitations": { "severity": "none", "filler_count": 0 }, "choppiness": { "detected": false } } }