serbian-dub

Russian → Serbian (Latin script) dubbing pipeline for films and long videos.

It transcribes the original audio, attributes lines to characters, translates with attention to register and timing budgets, synthesizes per-character voice clones, aligns speech to mouth movement, mixes to broadcast loudness, and runs automated QA that re-renders only the chunks that fail.

Two paid APIs do the heavy lifting: ElevenLabs (Scribe v2 ASR + IVC eleven_v3 voice cloning) and Anthropic Opus (attribution, translation, QA judging). Everything else is local Python: Demucs separation, MediaPipe FaceMesh lipsync analysis, loudness processing.

How it works

prep → transcribe → [full_cast] → [screenplay] → dialog_attribution
    → [srt_auto_fetch] → [srt_supplement] → asr_recheck → context
    → music_detect → audio_profile → translate → [vocab_judge]
    → length_audit → video_lipsync → synth → post_audio → mux
    → [coverage_fill] → qa_loop

[brackets] = optional, gated on a flag or API key presence. Every stage is idempotent: completed work is cached and skipped on re-run.

Under the hood:

1. Demucs (htdemucs) splits vocals from instrumental; detected music zones keep the original audio
2. Scribe v2 transcribes Russian audio with word timestamps and diarization (~30 s for a 90-min film)
3. An Opus pass assigns each segment a CHAR_* speaker and cuts clean per-character voice refs
4. A second Opus pass writes a cultural/register briefing for the film, then translates slot-aware, within characters-per-second budgets
5. MediaPipe FaceMesh finds mouth-open intervals; synthesized chunks are placed on mouth movement
6. ElevenLabs IVC renders each chunk with its character's cloned voice and emotion audio tags
7. The mix is normalized to LUFS-18 with de-essing, sidechain ducking, and true-peak limiting
8. QA scores every chunk (CER against re-transcription, SNR, speaker similarity); failures go through an automatic patch-render loop

Requirements

• macOS on Apple Silicon (developed and tested there; Linux untested)
• Python 3.14
• ffmpeg and rubberband on PATH
• ElevenLabs API key, Anthropic API key

Quick start

git clone https://github.com/liqlos/serbian-dub
cd serbian-dub
python3.14 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

cp .env.example .env        # fill in your keys
set -a && source .env && set +a   # re-run in every new shell; scripts don't auto-load .env

cp config.example.json config.json   # describe your film + cast

python scripts/run_pipeline.py \
  --source /path/to/film.mp4 \
  --start 0:00 --end 1:31:50 \
  --lang sr --tts el --variant final \
  --to qa_loop    # default stops at mux; this adds the QA + patch loop

The pipeline halts before each Opus stage (exit code 2) and writes a task file. Send that task to any capable LLM (or solve it yourself), save the response where the script expects it, and re-run the same command. Cached responses make re-runs cheap. See CLAUDE.md for the full prepare/apply protocol.

Environment variables

Variable	Required for
ELEVENLABS_API_KEY	transcription (Scribe v2), synthesis (IVC)
ANTHROPIC_API_KEY	all Opus prepare/apply stages
OPENSUBTITLES_API_KEY	optional SRT auto-fetch
OPENSUBTITLES_USERNAME	optional SRT auto-fetch
OPENSUBTITLES_PASSWORD	optional SRT auto-fetch

Common commands

# Resume from a stage
python scripts/run_pipeline.py --from translate --variant final --lang sr --tts el

# Re-run one stage only
python scripts/run_pipeline.py --from post_audio --to mux --variant final --lang sr --tts el --force-post-audio

# QA report + manual patch render
python scripts/qa_audit.py --variant final --lang sr --checks core,coverage,snr
python scripts/patch_render.py --variant final --lang sr

# Generate subtitles from the dubbed track
python scripts/generate_srt.py --variant final --lang sr

# Sanity check after edits (no API calls)
python scripts/test_smoke.py --module-import-only

Key output files

All artifacts land in out/test/, suffixed by --variant:

File	Description
sample_<variant>_final.mp4	final dubbed video
voice_<variant>.wav	assembled voice track
mix_<variant>_pro.wav	post-processed LUFS-18 mix
qa_report_<variant>.json	per-chunk QA metrics
bad_chunks_<variant>.json	chunks flagged for re-render

Documentation

• PIPELINE.md — stage-by-stage reference, chunk rules, QA issue routing, gotchas
• CLAUDE.md — guide for AI agents working on this repo: architecture, subagent protocols, caching rules (AGENTS.md is the entry pointer for non-Claude agents)
• CHANGELOG.md — development log
• docs/archive/ — early engine comparisons (XTTS/CosyVoice/F5/OmniVoice era)

Author

Aleksei Bukhalov github.com/liqlos · bukhalov.com

Licensed under MIT.