feat: studio verify: automated QA of the finished master

.claude/skills/make-explainer-video/SKILL.md

@@ -35,7 +35,7 @@ On a fresh checkout or install, run `studio setup --check`. If it exits non-zero
    - Edit `brand/` for house-style changes, never `episodes/<slug>/assets/brand/` (a synced copy).
 5. **Review**: `studio preview <slug>` opens HyperFrames Studio with narration. For a look without a browser, run `npx hyperframes snapshot episodes/<slug> --at 12.5,40` and view the frames.
 6. **Check**: `studio check <slug>` must pass with 0 errors (lint, layout, contrast) before any render.
-7. **Render**: `studio render <slug> --draft` for a review copy, then `studio render <slug>` for `out/<slug>.mp4`, `.srt` and `.chapters.txt`. Only changed scenes re-render, and `--force [ids]` forces them. A full render takes about real time. Optionally run `studio thumbnail <slug>` after editing `thumbnail/thumbnail.html`.
+7. **Render**: `studio render <slug> --draft` for a review copy, then `studio render <slug>` for `out/<slug>.mp4`, `.srt` and `.chapters.txt`. `render` then runs `studio verify` on the master (loudness, black frames, silences, speech vs script, captions, chapters) and exits 1 on errors; `--no-verify` skips it. Only changed scenes re-render, and `--force [ids]` forces them. A full render takes about real time. Optionally run `studio thumbnail <slug>` after editing `thumbnail/thumbnail.html`.
 
 ## Voice
 

AGENTS.md

@@ -32,7 +32,8 @@ uv run studio status <slug>          # word counts, est vs actual length, cues p
 uv run studio preview <slug>         # HyperFrames Studio in the browser (with narration)
 uv run studio check <slug>           # lint + layout + contrast audits; must pass
 uv run studio render <slug> --draft  # fast review copy -> out/<slug>-draft.mp4
-uv run studio render <slug>          # final -> out/<slug>.mp4 + .srt + .chapters.txt
+uv run studio render <slug>          # final -> out/<slug>.mp4 + .srt + .chapters.txt, then verifies it (--no-verify)
+uv run studio verify <slug>          # QA the master: loudness, black frames, silences, speech vs script, captions
 uv run studio music <slug> --prompt "..."  # ElevenLabs Music bed sized to the synced episode (after sync)
 uv run studio thumbnail <slug>       # out/<slug>-thumbnail.jpg (edit thumbnail/thumbnail.html)
 uv run studio archive <slug>         # copy the master, captions and thumbnail to $STUDIO_ARCHIVE (rclone)

docs/commands.md

@@ -74,7 +74,7 @@ Runs `sync`, then `hyperframes check`: lint, layout and contrast audits. Exits n
 ## `render`
 
 ```bash
-uv run studio render <slug> [--draft] [--force [ID ...]]
+uv run studio render <slug> [--draft] [--force [ID ...]] [--no-verify]
 ```
 
 Runs `sync`, renders changed scenes, assembles and masters the episode into `out/`.
@@ -84,9 +84,30 @@ Runs `sync`, renders changed scenes, assembles and masters the episode into `out
 | `--draft` | Fast, lower-quality render to `out/<slug>-draft.mp4` |
 | `--force` | Re-render every scene |
 | `--force ID …` | Re-render only the named scenes |
+| `--no-verify` | Skip the [`verify`](#verify) pass that otherwise runs on the finished master |
 
 Requires ffmpeg. See [Rendering and publishing](rendering.md).
 
+## `verify`
+
+```bash
+uv run studio verify <slug> [--draft] [--no-speech]
+```
+
+QA of the file you would upload (`out/<slug>.mp4`), also run at the end of `render`. It checks the
+resolution, frame rate and length against `timing.json`; integrated loudness and true peak against
+`loudness` and `true_peak`; black frames (error) and long frozen video (warning); silences longer than the
+script's own pauses; that a transcript of the mastered audio matches the script (so the music bed isn't
+burying the voice, and it lists the words it didn't hear); and that the `.srt` and `.chapters.txt` exist
+with chapters that meet YouTube's rules. Exits 1 if anything is an error; warnings don't fail it.
+
+| Flag | Meaning |
+| --- | --- |
+| `--draft` | Check `out/<slug>-draft.mp4`, without the captions and chapters |
+| `--no-speech` | Skip the transcription check, the slowest step |
+
+It reads `timing.json` and the existing render, so it doesn't re-sync: after editing the script, `render` first.
+
 ## `thumbnail`
 
 ```bash

docs/rendering.md

@@ -36,6 +36,14 @@ The cache lives in `episodes/<slug>/build/` and is safe to delete.
 - `studio render` warns if the chapters don't meet YouTube's rules: at least 3 chapters, each at
   least 10 seconds.
 
+## Verification
+
+After mastering, `render` runs `studio verify` on the file it just wrote and exits 1 if anything is
+an error: wrong resolution, frame rate or length, loudness or true peak off target, black frames, a
+silence longer than the script plans, a transcript that no longer matches the script (the music bed is
+too loud), or missing captions and chapters. Skip it with `--no-verify`, or run it alone later with
+`uv run studio verify <slug>`.
+
 ## Outputs
 
 | File | Use |

tests/test_verify.py

@@ -0,0 +1,216 @@
+"""verify: ffmpeg-backed checks on synthetic masters (a black segment, a silent gap, a too-quiet file)."""
+
+import shutil
+import subprocess
+import sys
+import textwrap
+
+import pytest
+
+from studio import cli, verify
+from studio.script import parse
+
+needs_ffmpeg = pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="needs ffmpeg")
+
+W, H, FPS = 320, 180, 30
+
+
+def make_master(path, secs=12.0, black=None, silent=None, level=-14.0, size=(W, H)):
+    """A noisy-video, tone-audio mp4. black/silent are (start, end) spans; level is the tone's dB gain."""
+    video = f"testsrc2=s={size[0]}x{size[1]}:r={FPS}:d={secs}"
+    vf = "null"
+    if black:
+        vf = f"drawbox=x=0:y=0:w=iw:h=ih:color=black:t=fill:enable='between(t,{black[0]},{black[1]})'"
+    af = f"volume={level}dB"
+    if silent:
+        af += f",volume=enable='between(t,{silent[0]},{silent[1]})':volume=0"
+    subprocess.run(["ffmpeg", "-v", "error", "-y", "-f", "lavfi", "-i", video, "-f", "lavfi",
+                    "-i", f"sine=f=300:r=48000:d={secs}", "-vf", vf, "-af", af, "-c:v", "libx264", "-pix_fmt", "yuv420p",
+                    "-c:a", "aac", str(path)], check=True)
+    return path
+
+
+def timing(secs=12.0, scenes=None):
+    scenes = scenes or [{"id": "a", "start": 0.0, "duration": secs, "voStart": 0.5, "voEnd": secs - 1}]
+    return {"scenes": scenes, "width": W, "height": H, "fps": FPS, "duration": secs}
+
+
+@pytest.fixture
+def script(write_script):
+    return write_script("## a: A\nHello there.\n")
+
+
+@needs_ffmpeg
+def test_loudness_ok_and_too_quiet(tmp_path):
+    loud = tmp_path / "loud.mp4"
+    subprocess.run(["ffmpeg", "-v", "error", "-y", "-f", "lavfi", "-i", "sine=f=300:r=48000:d=8", "-af",
+                    "loudnorm=I=-14:TP=-1.5", "-ar", "48000", str(loud)], check=True)
+    assert verify.check_loudness(loud, -14, -1.5).level == "ok"
+    quiet = make_master(tmp_path / "quiet.mp4", secs=8, level=-40)
+    found = verify.check_loudness(quiet, -14, -1.5)
+    assert found.level == "error" and "LUFS" in found.message
+
+
+@needs_ffmpeg
+def test_black_segment_is_an_error_but_a_scene_fade_is_not(tmp_path):
+    t = timing(12.0, [{"id": "a", "start": 0.0, "duration": 6.0, "voStart": 0, "voEnd": 5},
+                      {"id": "b", "start": 6.0, "duration": 6.0, "voStart": 0, "voEnd": 5}])
+    mid = verify.check_video(make_master(tmp_path / "mid.mp4", black=(2.0, 4.0)), t)
+    assert [f.level for f in mid if f.name == "black frames"] == ["error"]
+    boundary = verify.check_video(make_master(tmp_path / "edge.mp4", black=(5.7, 6.3)), t)
+    assert [f.level for f in boundary if f.name == "black frames"] == ["ok"]
+
+
+@needs_ffmpeg
+def test_silent_gap_is_an_error_but_a_short_pause_is_not(tmp_path, script):
+    t = timing()
+    gap = verify.check_silence(make_master(tmp_path / "gap.mp4", silent=(3.0, 8.0)), script, t)
+    assert gap.level == "error" and "0:03" in gap.message
+    short = verify.check_silence(make_master(tmp_path / "short.mp4", silent=(3.0, 4.0)), script, t)
+    assert short.level == "ok"
+
+
+@needs_ffmpeg
+def test_silence_is_allowed_in_a_scene_without_narration(tmp_path, script):
+    t = timing(12.0, [{"id": "a", "start": 0.0, "duration": 4.0, "voStart": 0.5, "voEnd": 3.0},
+                      {"id": "b", "start": 4.0, "duration": 6.0, "voStart": 0, "voEnd": 0},
+                      {"id": "c", "start": 10.0, "duration": 2.0, "voStart": 0.5, "voEnd": 1.5}])
+    assert verify.check_silence(make_master(tmp_path / "m.mp4", silent=(4.0, 10.0)), script, t).level == "ok"
+
+
+@needs_ffmpeg
+def test_format_checks_size_and_duration(tmp_path):
+    good = verify.check_format(make_master(tmp_path / "good.mp4"), timing())
+    assert [f.level for f in good] == ["ok", "ok"]
+    wrong = verify.check_format(make_master(tmp_path / "wrong.mp4", size=(160, 90)), timing(9.0))
+    assert [f.level for f in wrong] == ["error", "error"]
+
+
+def test_speech_compares_the_transcript_with_the_script(tmp_path, monkeypatch, script):
+    monkeypatch.setattr(verify, "run", lambda *a, **k: "")
+    heard = lambda *words: [{"text": w, "start": 0, "end": 1} for w in words]  # noqa: E731
+    monkeypatch.setattr(verify.voice, "transcribe", lambda wav: heard("Hello", "there"))
+    assert verify.check_speech(tmp_path / "m.mp4", script).level == "ok"
+    monkeypatch.setattr(verify.voice, "transcribe", lambda wav: heard("Hello"))
+    found = verify.check_speech(tmp_path / "m.mp4", script)
+    assert found.level == "error" and "not heard: there" in found.message
+    monkeypatch.setattr(verify.voice, "transcribe", lambda wav: [])
+    assert verify.check_speech(tmp_path / "m.mp4", script).level == "error"
+
+
+def test_speech_without_narration_is_ok(tmp_path, write_script):
+    silent = write_script("## a: A\n> duration: 3\n")
+    assert verify.check_speech(tmp_path / "m.mp4", silent).level == "ok"
+
+
+def test_silence_limit_follows_the_script(write_script):
+    s = write_script("## a: A\nOne. {pause 3} Two.\n")
+    assert verify.silence_limit(s, {}) == pytest.approx(3.5)
+
+
+def test_subtract_removes_allowed_spans():
+    assert verify._subtract(0, 10, [(2, 4), (8, 12)]) == [(0, 2), (4, 8)]
+    assert verify._subtract(3, 4, [(0, 10)]) == []
+
+
+@pytest.fixture
+def out_dir(tmp_path, monkeypatch):
+    monkeypatch.setattr(verify, "OUT", tmp_path)
+    return tmp_path
+
+
+def chapter_timing(*lengths):
+    t, scenes = 0.0, []
+    for i, n in enumerate(lengths):
+        scenes.append({"id": f"s{i}", "title": f"T{i}", "start": t, "duration": n})
+        t += n
+    return {"scenes": scenes, "duration": t}
+
+
+def test_deliverables(out_dir):
+    t = chapter_timing(20, 20, 20)
+    assert [f.level for f in verify.check_deliverables("ep", t)] == ["error", "error"]
+    (out_dir / "ep.srt").write_text("1\n", encoding="utf-8")
+    (out_dir / "ep.chapters.txt").write_text("0:00 T0\n0:20 T1\n0:40 T2\n", encoding="utf-8")
+    assert [f.level for f in verify.check_deliverables("ep", t)] == ["ok", "ok"]
+    assert verify.check_deliverables("ep", chapter_timing(20, 5))[1].level == "warn"
+    (out_dir / "ep.chapters.txt").write_text("0:05 T0\n", encoding="utf-8")
+    assert verify.check_deliverables("ep", t)[1].message == "the first chapter must start at 0:00"
+    assert verify.check_deliverables("ep", {**t, "format": "reel"})[1].level == "error"
+
+
+def test_report_counts_errors(capsys):
+    n = verify.report([verify.Finding("ok", "a", "fine"), verify.Finding("warn", "b", "hmm"),
+                       verify.Finding("error", "c", "bad")])
+    assert n == 1
+    out = capsys.readouterr().out
+    assert "ERROR  c" in out and "1 error(s), 1 warning(s)" in out
+
+
+def test_verify_master_needs_a_render(tmp_path, script, capsys):
+    with pytest.raises(SystemExit):
+        verify.verify_master(tmp_path / "ep", script, timing(), tmp_path / "missing.mp4")
+    assert "studio render" in capsys.readouterr().err
+
+
+# ---------------------------------------------------------------- CLI
+
+@pytest.fixture
+def episode(tmp_path, monkeypatch):
+    ep = tmp_path / "episodes" / "ep"
+    ep.mkdir(parents=True)
+    (ep / "script.md").write_text(textwrap.dedent("## a: A\nHello there.\n"), encoding="utf-8")
+    (tmp_path / "out").mkdir()
+    monkeypatch.setattr(cli, "EPISODES", tmp_path / "episodes")
+    monkeypatch.setattr("studio.common.EPISODES", tmp_path / "episodes")
+    monkeypatch.setattr(cli, "ROOT", tmp_path)
+    monkeypatch.setattr(cli, "OUT", tmp_path / "out")
+    monkeypatch.setattr(cli, "require_tools", lambda: None)
+    return ep
+
+
+def studio(monkeypatch, *argv):
+    monkeypatch.setattr(sys, "argv", ["studio", *argv])
+    cli.main()
+
+
+def stub_verify(monkeypatch, *levels):
+    calls = []
+    monkeypatch.setattr(cli.verify, "verify_master",
+                        lambda *a, **k: calls.append(k) or [verify.Finding(lv, "x", "y") for lv in levels])
+    return calls
+
+
+def test_verify_command_exit_status(episode, monkeypatch):
+    (episode / "timing.json").write_text('{"scenes": []}', encoding="utf-8")
+    calls = stub_verify(monkeypatch, "ok", "warn")
+    studio(monkeypatch, "verify", "ep", "--draft", "--no-speech")  # warnings don't fail
+    assert calls == [{"speech": False, "deliverables": False}]
+    stub_verify(monkeypatch, "ok", "error")
+    with pytest.raises(SystemExit) as exit:
+        studio(monkeypatch, "verify", "ep")
+    assert exit.value.code == 1
+
+
+def test_verify_command_needs_timing(episode, monkeypatch, capsys):
+    with pytest.raises(SystemExit):
+        studio(monkeypatch, "verify", "ep")
+    assert "studio sync ep" in capsys.readouterr().err
+
+
+@pytest.mark.parametrize("flags,verified", [([], True), (["--no-verify"], False)])
+def test_render_verifies_unless_skipped(episode, monkeypatch, flags, verified):
+    monkeypatch.setattr(cli, "_sync", lambda slug: (episode, parse(episode / "script.md"), {}))
+    monkeypatch.setattr(cli.renderer, "render_episode", lambda *a, **k: episode / "master.mp4")
+    calls = stub_verify(monkeypatch, "ok")
+    studio(monkeypatch, "render", "ep", *flags)
+    assert bool(calls) is verified
+
+
+def test_render_fails_when_verification_finds_errors(episode, monkeypatch):
+    monkeypatch.setattr(cli, "_sync", lambda slug: (episode, parse(episode / "script.md"), {}))
+    monkeypatch.setattr(cli.renderer, "render_episode", lambda *a, **k: episode / "master.mp4")
+    stub_verify(monkeypatch, "error")
+    with pytest.raises(SystemExit) as exit:
+        studio(monkeypatch, "render", "ep")
+    assert exit.value.code == 1

tools/studio/cli.py

@@ -7,6 +7,7 @@
     uv run studio preview <slug>          open HyperFrames Studio on the episode
     uv run studio check <slug>            HyperFrames lint + layout/contrast audits
     uv run studio render <slug> [--draft] render changed scenes, assemble, master -> out/
+    uv run studio verify <slug>           QA the finished master: loudness, black frames, silences, speech, captions
     uv run studio thumbnail <slug>        screenshot thumbnail/thumbnail.html -> out/
     uv run studio voices <slug>           audition Kokoro blends or Chatterbox clips on one scene -> out/voice-samples/
     uv run studio music <slug> --prompt   generate an instrumental music bed (ElevenLabs Music) sized to the episode
@@ -30,7 +31,7 @@
 import soundfile as sf
 
 from . import build as builder
-from . import music, setup, skill, voice
+from . import music, setup, skill, verify, voice
 from . import render as renderer
 from .common import (
     EPISODES,
@@ -128,7 +129,26 @@ def cmd_render(a):
     require_tools()
     ep, script, timing = _sync(a.slug)
     info("rendering")
-    renderer.render_episode(ep, script, timing, a.draft, set(a.force) if a.force else None, a.force is not None)
+    master = renderer.render_episode(ep, script, timing, a.draft, set(a.force) if a.force else None,
+                                     a.force is not None)
+    if not a.no_verify:
+        info("\nverifying")
+        if verify.report(verify.verify_master(ep, script, timing, master, deliverables=not a.draft)):
+            raise SystemExit(1)
+
+
+def cmd_verify(a):
+    require_tools()
+    ep = episode_dir(a.slug)
+    script = parse(ep / "script.md")
+    if not (ep / "timing.json").exists():
+        die(f"no timing.json: run `studio sync {a.slug}` and `studio render {a.slug}` first")
+    timing = json.loads((ep / "timing.json").read_text(encoding="utf-8"))
+    master = OUT / f"{a.slug}{'-draft' if a.draft else ''}.mp4"
+    info(f"verifying {master.relative_to(ROOT)}")
+    findings = verify.verify_master(ep, script, timing, master, speech=not a.no_speech, deliverables=not a.draft)
+    if verify.report(findings):
+        raise SystemExit(1)
 
 
 def find_browser() -> str:
@@ -299,8 +319,15 @@ def main():
     s.add_argument("slug")
     s.add_argument("--draft", action="store_true", help="fast draft quality -> out/<slug>-draft.mp4")
     s.add_argument("--force", nargs="*", metavar="SCENE", help="re-render these scenes (or all, if none named)")
+    s.add_argument("--no-verify", action="store_true", help="skip the automated QA of the finished master")
     s.set_defaults(fn=cmd_render)
 
+    s = sub.add_parser("verify", help="QA the finished master in out/: loudness, black frames, silences, speech, captions")
+    s.add_argument("slug")
+    s.add_argument("--draft", action="store_true", help="check out/<slug>-draft.mp4 (skips the captions and chapters)")
+    s.add_argument("--no-speech", action="store_true", help="skip the transcription check (the slowest step)")
+    s.set_defaults(fn=cmd_verify)
+
     s = sub.add_parser("thumbnail", help="render thumbnail/thumbnail.html to out/<slug>-thumbnail.jpg")
     s.add_argument("slug")
     s.set_defaults(fn=cmd_thumbnail)