feat(editor): transcribe every media automatically and gate the AI actions on it

[EtienneLescot]

Summary

Clicking "Smart cuts with AI" was the natural first move in the editor, and the one that could not work: nothing produced a transcript until the user went looking for the Media tab or the transcript pane and ordered one. Transcription is local and free, so the editor now produces transcripts by itself, and every affordance that needs one says what it is waiting for instead of doing nothing.

Automatic, SSOT, no loop — store/transcriptionStore.ts owns the queue, mounted once via useAutoTranscription() in the shell and driven by any document change (Media-tab import, or the asset auto-added after a screen recording). One run at a time — whisper-server is a single process and audio extraction holds decoded frames in renderer memory. The transcript itself still lives on the document (document.transcripts[]); the store only owns the job. The auto pass cannot re-trigger on its own result: an asset is enqueued only when it has no transcript, no job entry (queued / running / failed alike) and no persisted failure, and a job is dropped only after the save carrying its transcript has resolved. Runs are stamped with a run id, so a manual regenerate can supersede one mid-flight without the outgoing run clearing its successor.

Gating — the Smart-cuts row is disabled unless a usable transcript exists, with a spinner and a subtitle that replaces "With AI" by the actual reason (transcribing / no audio / no speech / failed). Same treatment for the transcript pane, the captions pane, the media cards and the source-transcript modal.

Several media — everything is per asset. The gate is resolved over the assets the timeline plays (falling back to the media bin while it is empty), and pending beats ready: no point letting the agent plan cuts over half the timeline while the other half is still being transcribed.

Errors — classifyTranscriptionError separates the deterministic verdicts (no audio track, unreadable audio codec) from transient ones. Deterministic ones are written to asset.transcriptionFailure (additive schema field, no version bump), so a silent screen recording is not re-extracted on every project open and the button is correctly greyed with the right message straight from load. Transient failures stay in memory and are retried on the next load. Silence produces no error toast — it is an expected outcome, and the UI already says so.

Cleanup along the way — removes projectStore.setTranscript (it duplicated withTranscript) and the assetStatuses / onRegenerateAsset / onTranscribe / isTranscribing prop chains: the panes read the store.

Related issue

Type of change

• Bug fix
• Feature
• Enhancement
• Documentation
• Refactor / maintenance
• Performance
• Security

Release impact

• Patch
• Minor
• Major / breaking change
• No release note needed

Desktop impact

• Windows
• macOS
• Linux
• Installer / packaging
• Not platform-specific

Screenshots / video

None: the states were verified through the DOM rather than visually (the review environment could not composite frames for a screenshot). Evidence from the running editor, with a silent screen capture as the primary asset plus a talking clip that has a transcript:

• transcript pane, per clip: 1 screen-recording.webm … No transcript for this clip — open the asset card and regenerate. / 2 talking-head.mp4 … Bonjour à tous voici la démo [silence 8.5s]
• gate: { state: "ready", reason: null, pendingCount: 0 } → Smart cuts clickable
• with only the silent capture in the project: { state: "blocked", reason: "no-audio" }, transcript pane This media has no audio track + Transcribe now disabled, captions pane This media has no audio track — there is nothing to transcribe. + Transcribe video disabled

Testing

• npx tsc --noEmit, npm run lint, npm run i18n:check (10 new keys × 13 locales), node scripts/check-docs.mjs — all clean
• npm run test — 1446 passed / 120 files, of which 29 new:
  • transcription/status.test.ts — error classification, per-asset status precedence (including a stored transcript outranks a failed retry), gate resolution, timeline-relevant assets
  • store/transcriptionStore.test.ts — sequential drain, loop guard across repeated sync, silent-verdict persistence + no auto-retry, transient failure kept in memory, manual supersede of an in-flight run, project switch, asset removal, no-engine, requestTimelineTranscripts targeting
  • TranscriptPane.gating.test.tsx, CaptionsPane.gating.test.tsx, TranscriptPane.keyboardCut.test.tsx — the rendered enabled/disabled/message states, and the per-asset read-only scoping (cutting still works while another asset transcribes; it stops, visibly, while this one does)
  • electron/stt/index.test.ts — a failed setup is retried instead of cached
• tsc && vite build passes (renderer + main + preload bundles)
• A multi-agent adversarial review over the diff surfaced two real defects, both fixed in this branch before it was opened: the panes gating on primaryAssetId instead of the timeline's assets, and a failed job masking an existing usable transcript.
• Not verified end to end: the whisper run itself. The browser preview has no window.electronAPI.stt (by design — no engine, no background pass), so the automatic run, the queue and the failure bookkeeping are covered by unit tests rather than by a live recognition pass.

First run on a fresh install

On a fresh install the GGML model (≈253 MB, HuggingFace) is not on disk. It is fetched by SttManager.prepare() inside the stt:transcribe IPC call — i.e. inside a run this store has already marked running — so it reads to the user as one single busy phase that simply takes longer the first time: no separate "downloading" step, no progress bar to stare at, nothing falsely clickable in the meantime. That is the intended behaviour, and phase: "model" is deliberately not forwarded to the renderer. Nothing imposes a timeout a slow download could trip (bare ipcRenderer.invoke, no per-request deadline in fetchWithRetry, whisper-server's 30 s readiness budget only starts after the download resolves), and the model is fetched once for a whole queue.

An adversarial review of that first-run sequence found the promise held on the happy path but broke at the edges — fixed in the second commit:

• SttManager.init cached a rejected prepare(). One dropped connection during the download failed every later transcription for the rest of the session: the remaining queued assets flipped red in the same frame, and the retry the editor offers was a dead control until the app was restarted — reconnecting the network changed nothing. The slot is now cleared on failure.
• A transient failure stops the queue instead of walking the remaining assets into the same wall: they inherit the verdict (one toast, one retry affordance) rather than each spending a full retry budget.
• Read-only is scoped per asset again. Feeding the timeline-wide gate into the transcript pane's busy made every other clip's word stream swallow Backspace and hover-bin clicks for the whole background pass, with nothing on screen to say why. A block being rewritten now shows a spinner + "Transcribing…", dims its stream and sets aria-busy; the per-clip empty line no longer tells the user to regenerate an asset that is mid-run.
• mixToMono no longer freezes the window. It called getChannelData inside the sample loop — one WebIDL call per sample per channel, ~57 M for a ten-minute stereo recording, seconds of blocked main thread with the spinners stopped mid-animation. Survivable while the pass was user-triggered; not now that it is automatic.

Two known cosmetics left, deliberately: the Smart-cuts subtitle shows "Needs a transcript" for the single frame before useAutoTranscription's effect runs (disabled either way, only the reason differs), and an offline first run surfaces the raw network message in the failure toast. The mixdown is also still synchronous — a worker is the next step if it shows up in practice.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 78367b01-16fc-4333-af88-bb8ecdd7e5d6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}