diff --git a/README.md b/README.md index 6ce5cc87c..3e0fc586e 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,19 @@ The goal of this continuation is to keep OpenScreen alive as a fully open-source - Export to MP4 or GIF in multiple aspect ratios and resolutions, rendered and encoded on the GPU (Metal on macOS, D3D11 on Windows, Vulkan on Linux) with an automatic CPU fallback. - Languages supported: Arabic, English, Spanish, French, Italian, Japanese, Korean, Portuguese (Brazil), Russian, Turkish, Vietnamese, Simplified Chinese, and Traditional Chinese. +## Command-line interface (headless) + +OpenScreen ships a CLI for scripts, CI, and AI coding agents: record the screen +headlessly, edit the `.openscreen` project JSON programmatically (zooms, +annotations, trims), and render MP4/GIF with the full export pipeline — no +visible windows, NDJSON output with `--json`. + +```bash +openscreen record --duration 20 --project demo.openscreen --json +openscreen export demo.openscreen -o demo.mp4 --json +``` + +See [docs/cli.md](./docs/cli.md). ## Installation diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 000000000..9e965bcc1 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,244 @@ +# OpenScreen CLI + +Headless command-line interface for recording the screen and exporting `.openscreen` +projects — no visible windows, machine-readable output. Designed so scripts, CI +pipelines, and AI coding agents can produce polished product demos automatically: + +```text +record → edit the project JSON programmatically → export → MP4/GIF +``` + +## Running + +Development — build once per checkout: + +```bash +npm run build-vite # renderer + main +npm run build:native:mac # capture helpers (recording) +npm run fetch:ffmpeg:mac && npm run build:native:compositor:mac # Rust compositor (export) +bash scripts/build-whisper-stt.sh # STT server (captions; model downloads on first run) +``` + +Then: + +```bash +npm run cli -- [options] +# or directly: +./node_modules/.bin/electron . [options] +``` + +Packaged app (the CLI ships inside the normal binary): + +```bash +# macOS +/Applications/Openscreen.app/Contents/MacOS/Openscreen export demo.openscreen -o demo.mp4 +# Windows +"C:\Program Files\Openscreen\Openscreen.exe" export demo.openscreen -o demo.mp4 +``` + +CLI runs skip the single-instance lock, so they work while the GUI app is open. + +## Commands + +### `openscreen record` + +Records headlessly through the same pipeline as the GUI: the native +ScreenCaptureKit helper on macOS, the WGC helper on Windows (browser capture as +fallback). Recordings land in the app's recordings directory +(`/recordings/`), exactly like GUI recordings — including the +`.cursor.json` cursor-telemetry sidecar used for editable cursors and auto-zoom. + +```bash +openscreen record --duration 30 --project demo.openscreen --json +openscreen record --window "My App" --mic --system-audio +openscreen record --display 1 --cursor system +``` + +| Option | Meaning | +|---|---| +| `--display ` | Screen index to record (default 0) | +| `--window ` | Record the first window whose title contains `<title>` | +| `--mic` / `--mic-device <name>` | Capture microphone (optionally by device-label substring) | +| `--system-audio` | Capture system audio | +| `--cursor <editable-overlay\|system>` | Hide the system cursor and record telemetry (default), or bake it into the video | +| `--duration <seconds>` | Stop automatically | +| `--project <out.openscreen>` | Write a ready-to-export project file when done | +| `--json` | NDJSON events on stdout | + +Stopping without `--duration`: send SIGINT/SIGTERM to the process, or type +`stop` + Enter on its stdin. + +Platform notes: + +- **macOS**: requires the Swift helper (`npm run build:native:mac`, needs Xcode) + and the Screen Recording permission for whatever binary hosts Electron + (your terminal during development). Webcam capture is not available in CLI + recording on macOS (same limitation as the native helper). +- **Windows**: requires the WGC helper (`npm run build:native:win`). SIGTERM + does not exist on Windows — stop recordings with Ctrl+C, stdin `stop`, or + `--duration` (a hard `taskkill` loses the recording). Microphone access is + gated by Settings → Privacy → Microphone; there is no programmatic prompt. +- **Linux**: uses the browser capture pipeline; cursor options are limited, + matching the GUI. On Wayland, capture goes through the PipeWire portal, + which may show a system picker dialog and requires a desktop session + (headless/SSH sessions without a portal cannot record). + +### `openscreen sources` + +Lists capturable displays, windows, and microphones — the same enumeration the +GUI picker uses — so scripts and agents can choose `--display`, `--window`, and +`--mic-device` values without guesswork. + +```bash +openscreen sources # human-readable +openscreen sources --json +``` + +`--json` emits the payload on the final `done` event: + +```json +{ + "event": "done", + "success": true, + "sources": { + "displays": [{ "index": 0, "id": "screen:1:0", "name": "Entire screen" }], + "windows": [{ "id": "window:210:0", "name": "My App" }], + "microphones": [{ "label": "MacBook Pro Microphone (Built-in)" }], + "microphoneLabelsUnavailable": false + } +} +``` + +### `openscreen export` + +Renders a project to MP4 or GIF using the app's real export pipeline (WebCodecs + +PixiJS, faster than realtime) in a hidden window. Falls back to SwiftShader when +no GPU is available (CI), and applies everything the editor would: zooms, trims, +speed regions, wallpaper/padding, annotations, cursor rendering, webcam layouts. + +```bash +openscreen export demo.openscreen # format/quality from the project +openscreen export demo.openscreen -o out.mp4 --quality source +openscreen export demo.openscreen -o out.gif --gif-fps 20 --gif-size large +openscreen export demo.openscreen --json | while read line; do ...; done +``` + +| Option | Meaning | +|---|---| +| `-o, --out <path>` | Output file; extension picks the format. Default: next to the project | +| `--format <mp4\|gif>` | Override the project's stored format | +| `--quality <medium\|good\|source>` | MP4 quality | +| `--gif-fps <15\|20\|25\|30>`, `--gif-size <medium\|large\|original>` | GIF settings | +| `--auto-zoom` | Add automatic zooms from cursor telemetry before rendering — the same dwell-detection engine as the editor's magic wand. Existing zoom regions are kept; suggestions never overlap them | +| `--audio <file>` | Mix a voiceover file into the MP4 (mp3/wav/m4a — anything Chromium can decode; AIFF is not supported) | +| `--audio-mode <mix\|replace>` | Layer the voiceover over the recording's audio (default `mix`) or replace it | +| `--audio-offset <seconds>` | Delay before the voiceover starts (default 0) | +| `--json` | NDJSON progress + result on stdout | + +`--audio` mixes after the native render: the exported file is read back, video +packets are copied untouched, and the audio is mixed offline +(OfflineAudioContext) and re-encoded to AAC before overwriting the output. +MP4 only. +In `mix` mode the original audio is ducked to 40% under the voiceover so the +sum cannot clip; use `replace` to drop the original entirely. + +**Media path rule**: for safety, a project's referenced media is only auto-approved +when it lives in the app's recordings directory or **next to the project file**. +Keep `.openscreen` files beside their media (or record via the CLI, which uses +the recordings directory). + +**No cancel**: the native compositor has no abort mechanism — killing the CLI +mid-export stops output but the render worker runs until process exit. + +### `openscreen pack` + +Copies a project and everything it references (screen/webcam video, cursor +telemetry sidecar) into one portable folder and rewrites the project's media +paths: + +```bash +openscreen pack demo.openscreen --out bundle/ +``` + +The folder survives being moved or shipped as a CI artifact: when the stored +absolute paths go stale, the loader falls back to files with the same basename +next to the project file. + +### `openscreen captions` + +Transcribes the project's audio with the app's on-device Whisper model (no +upload; language auto-detected) and writes the resulting caption annotations +into the project. Re-running replaces earlier auto-captions; manual annotations +are preserved. + +```bash +openscreen captions demo.openscreen --min-words 2 --max-words 7 +openscreen export demo.openscreen -o demo.mp4 # subtitles are burned in +``` + +Requires an audio track in the project's video (e.g. `record --mic`, or a +voiceover mixed in with a re-recorded source). Transcription runs on the +native whisper.cpp engine (ggml-small); the model downloads automatically on +first use. + +### `openscreen info` + +Prints a project summary (referenced media and whether it exists, format, +region counts). Exits non-zero if the referenced video is missing. + +```bash +openscreen info demo.openscreen --json +``` + +## Machine-readable output (`--json`) + +One JSON object per line on stdout (NDJSON). stderr carries diagnostics only. + +```jsonl +{"event":"started","command":"export"} +{"event":"progress","percentage":42,"currentFrame":50,"totalFrames":120,"estimatedTimeRemaining":3} +{"event":"done","success":true,"outputPath":"/path/out.mp4","format":"mp4","width":1920,"height":1080} +``` + +Record emits `log` events (`Recording started`, …), `stopping`, and a final +`done` carrying `screenVideoPath`, `cursorDataPath`, `durationMs`, and +`projectPath` when `--project` was used. Exit code is 0 on success, 1 on +failure, 2 on bad arguments. + +## Example: automated product demo (for scripts/agents) + +```bash +# 1. Record 20 seconds of the running app +openscreen record --window "MyProduct" --duration 20 --project demo.openscreen --json + +# 2. Edit the project: add a zoom and a caption (plain JSON) +node -e ' + const fs = require("fs"); + const p = JSON.parse(fs.readFileSync("demo.openscreen", "utf8")); + p.editor.zoomRegions.push({ id: "z1", startMs: 2000, endMs: 6000, depth: 3, + focus: { cx: 0.5, cy: 0.4 }, focusMode: "manual", source: "manual" }); + p.editor.annotationRegions.push({ id: "a1", startMs: 500, endMs: 4000, + type: "text", content: "One-click setup", textContent: "One-click setup", + position: { x: 8, y: 6 }, size: { width: 40, height: 12 }, + style: { fontSize: 24, color: "#fff" }, zIndex: 1 }); + fs.writeFileSync("demo.openscreen", JSON.stringify(p, null, 2)); +' + +# 3. Narrate with any TTS (macOS `say` shown; any engine producing mp3/wav/m4a works) +say -o voice.m4a --file-format=m4af "Welcome to my product. Here's a quick tour." + +# 4. Render with auto-zooms; the voiceover replaces the recording's own audio +# (drop --audio-mode replace to duck the original under the narration instead) +openscreen export demo.openscreen -o demo.mp4 --auto-zoom --audio voice.m4a --audio-mode replace --json +``` + +## Architecture + +- `electron/cli/args.ts` — pure argv parser (unit-tested in `args.test.ts`). +- `electron/cli/cliMain.ts` — headless boot: no HUD/tray/menu/dock, stdio + protocol, signal handling, exit codes. Registers the same IPC surface as the + GUI (`registerIpcHandlers`) with inert window callbacks. +- `src/cli/CliExportRunner.tsx` / `src/cli/CliRecordRunner.tsx` — hidden-window + runners (`?windowType=cli-export|cli-record`) that drive the existing + exporter classes and `useScreenRecorder` hook. +- Contracts shared between main and renderer: `src/lib/cliContracts.ts`. diff --git a/electron/cli/args.test.ts b/electron/cli/args.test.ts new file mode 100644 index 000000000..7f0b18e09 --- /dev/null +++ b/electron/cli/args.test.ts @@ -0,0 +1,203 @@ +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { parseCliArgs } from "./args"; + +const CWD = path.resolve("/work"); +const inCwd = (name: string) => path.resolve(CWD, name); + +function parse(args: string[]) { + return parseCliArgs(["electron", "app-path", ...args], 2, CWD); +} + +describe("parseCliArgs", () => { + it("returns null when no subcommand is present (GUI launch)", () => { + expect(parse([])).toBeNull(); + expect(parse(["--some-chromium-flag"])).toBeNull(); + }); + + it("skips leading Chromium switches before the subcommand (AppImage --no-sandbox)", () => { + expect(parse(["--no-sandbox", "export", "demo.openscreen"])).toMatchObject({ + kind: "export", + projectPath: inCwd("demo.openscreen"), + }); + expect(parse(["--no-sandbox", "--enable-unsafe-swiftshader", "record"])).toMatchObject({ + kind: "record", + }); + expect(parse(["--no-sandbox", "--help"])).toMatchObject({ kind: "help" }); + expect(parse(["--no-sandbox"])).toBeNull(); + }); + + it("parses a minimal export command and resolves relative paths", () => { + const cmd = parse(["export", "demo.openscreen"]); + expect(cmd).toMatchObject({ + kind: "export", + projectPath: inCwd("demo.openscreen"), + outPath: null, + format: null, + }); + }); + + it("parses export options and infers format from --out extension", () => { + const cmd = parse([ + "export", + "/p/demo.openscreen", + "-o", + "out.gif", + "--gif-fps", + "20", + "--json", + ]); + expect(cmd).toMatchObject({ + kind: "export", + outPath: inCwd("out.gif"), + format: "gif", + gifFrameRate: 20, + json: true, + }); + }); + + it("rejects a --format that conflicts with the --out extension", () => { + const cmd = parse(["export", "a.openscreen", "-o", "x.mp4", "--format", "gif"]); + expect(cmd).toMatchObject({ kind: "error" }); + }); + + it("rejects export without a project path", () => { + expect(parse(["export"])).toMatchObject({ kind: "error" }); + }); + + it("parses voiceover audio options", () => { + const cmd = parse([ + "export", + "a.openscreen", + "--audio", + "voice.mp3", + "--audio-mode", + "replace", + "--audio-offset", + "1.5", + ]); + expect(cmd).toMatchObject({ + kind: "export", + audioPath: inCwd("voice.mp3"), + audioMode: "replace", + audioOffsetSec: 1.5, + }); + }); + + it("defaults audio mode to mix and rejects --audio with gif", () => { + expect(parse(["export", "a.openscreen", "--audio", "v.mp3"])).toMatchObject({ + audioMode: "mix", + audioOffsetSec: 0, + }); + expect(parse(["export", "a.openscreen", "--audio", "v.mp3", "--format", "gif"])).toMatchObject({ + kind: "error", + }); + // gif inferred from --out, with no explicit --format + expect(parse(["export", "a.openscreen", "--audio", "v.mp3", "-o", "out.gif"])).toMatchObject({ + kind: "error", + }); + expect(parse(["export", "a.openscreen", "--audio-offset", "-1"])).toMatchObject({ + kind: "error", + }); + }); + + it("parses record defaults", () => { + expect(parse(["record"])).toMatchObject({ + kind: "record", + displayIndex: 0, + windowTitle: null, + mic: false, + systemAudio: false, + cursorMode: "editable-overlay", + durationMs: null, + }); + }); + + it("parses record options", () => { + const cmd = parse([ + "record", + "--display", + "1", + "--mic-device", + "MacBook", + "--system-audio", + "--duration", + "12.5", + "--project", + "demo.openscreen", + ]); + expect(cmd).toMatchObject({ + kind: "record", + displayIndex: 1, + mic: true, + micDevice: "MacBook", + systemAudio: true, + durationMs: 12500, + projectOut: inCwd("demo.openscreen"), + }); + }); + + it("rejects invalid record values", () => { + expect(parse(["record", "--duration", "0"])).toMatchObject({ kind: "error" }); + expect(parse(["record", "--cursor", "off"])).toMatchObject({ kind: "error" }); + expect(parse(["record", "--project", "demo.json"])).toMatchObject({ kind: "error" }); + }); + + it("parses --auto-zoom", () => { + expect(parse(["export", "a.openscreen", "--auto-zoom"])).toMatchObject({ + kind: "export", + autoZoom: true, + }); + expect(parse(["export", "a.openscreen"])).toMatchObject({ autoZoom: false }); + }); + + it("parses sources", () => { + expect(parse(["sources", "--json"])).toMatchObject({ kind: "sources", json: true }); + expect(parse(["sources", "--bogus"])).toMatchObject({ kind: "error" }); + expect(parse(["sources", "extra-arg"])).toMatchObject({ kind: "error" }); + }); + + it("parses pack", () => { + expect(parse(["pack", "demo.openscreen", "--out", "bundle"])).toMatchObject({ + kind: "pack", + projectPath: inCwd("demo.openscreen"), + outDir: inCwd("bundle"), + }); + expect(parse(["pack", "demo.openscreen"])).toMatchObject({ kind: "error" }); + expect(parse(["pack", "--out", "bundle"])).toMatchObject({ kind: "error" }); + expect(parse(["pack", "demo.openscreen", "--out", "bundle", "--bogus"])).toMatchObject({ + kind: "error", + }); + expect(parse(["pack", "a.openscreen", "b.openscreen", "--out", "bundle"])).toMatchObject({ + kind: "error", + }); + }); + + it("parses captions", () => { + expect( + parse(["captions", "demo.openscreen", "--min-words", "1", "--max-words", "5"]), + ).toMatchObject({ + kind: "captions", + projectPath: inCwd("demo.openscreen"), + minWordsPerCaption: 1, + maxWordsPerCaption: 5, + }); + expect(parse(["captions", "demo.openscreen"])).toMatchObject({ + minWordsPerCaption: 2, + maxWordsPerCaption: 7, + }); + expect( + parse(["captions", "a.openscreen", "--min-words", "9", "--max-words", "3"]), + ).toMatchObject({ kind: "error" }); + }); + + it("parses info and help", () => { + expect(parse(["info", "demo.openscreen", "--json"])).toMatchObject({ + kind: "info", + projectPath: inCwd("demo.openscreen"), + json: true, + }); + expect(parse(["help"])).toMatchObject({ kind: "help" }); + expect(parse(["--help"])).toMatchObject({ kind: "help" }); + }); +}); diff --git a/electron/cli/args.ts b/electron/cli/args.ts new file mode 100644 index 000000000..0dad8d07c --- /dev/null +++ b/electron/cli/args.ts @@ -0,0 +1,439 @@ +// Pure argv parser for the OpenScreen CLI. No Electron imports so it can be +// unit-tested under plain vitest. + +import path from "node:path"; +import type { CliExportRequest, CliRecordRequest, CliRequest } from "../../src/lib/cliContracts"; + +export interface CliInfoCommand { + kind: "info"; + projectPath: string; +} + +export interface CliHelpCommand { + kind: "help"; +} + +export interface CliErrorCommand { + kind: "error"; + message: string; +} + +export type CliCommand = ( + | CliRequest + | CliInfoCommand + | CliPackCommand + | CliHelpCommand + | CliErrorCommand +) & { + /** Machine-readable NDJSON output on stdout instead of human progress. */ + json?: boolean; +}; + +const SUBCOMMANDS = new Set([ + "export", + "record", + "sources", + "pack", + "captions", + "info", + "help", + "--help", + "-h", +]); + +export const CLI_USAGE = `OpenScreen CLI + +Usage: + openscreen export <project.openscreen> [options] Render a project to MP4/GIF + openscreen record [options] Record the screen headlessly + openscreen sources [--json] List displays, windows and microphones + openscreen pack <project.openscreen> --out <dir> Copy project + media into one portable folder + openscreen captions <project.openscreen> Add auto-captions (on-device Whisper) to a project + [--min-words <n>] [--max-words <n>] + openscreen info <project.openscreen> [--json] Inspect a project file + openscreen help Show this help + +Export options: + -o, --out <path> Output file (.mp4 or .gif). Default: next to the project file + --format <mp4|gif> Override the format stored in the project + --quality <medium|good|source> + MP4 quality (default: from project) + --gif-fps <15|20|25|30> GIF frame rate (default: from project) + --gif-size <medium|large|original> + GIF size preset (default: from project) + --auto-zoom Add automatic zooms from cursor telemetry (editor's magic wand) + --audio <file> Mix a voiceover audio file into the MP4 (mp3/wav/m4a) + --audio-mode <mix|replace> + Layer over the recording's audio (default) or replace it + --audio-offset <seconds> Delay before the voiceover starts (default 0) + --json NDJSON progress/result on stdout + +Record options (recording is saved into the app's recordings directory): + --display <n> Screen index to record (default 0) + --window <title> Record the first window whose title contains <title> + --mic Capture the default microphone + --mic-device <name> Microphone device name (implies --mic) + --system-audio Capture system audio + --cursor <editable-overlay|system> + Cursor capture mode (default editable-overlay) + --duration <seconds> Stop automatically after this long + --project <out.openscreen> + Write a ready-to-export project file when done + --json NDJSON events on stdout + +Stopping a recording: send SIGINT/SIGTERM, type "stop" + Enter on stdin, +or pass --duration. +`; + +function takeValue(argv: string[], i: number, flag: string): [string, number] { + const next = argv[i + 1]; + if (next === undefined || next.startsWith("--")) { + throw new Error(`${flag} requires a value`); + } + return [next, i + 1]; +} + +function resolvePath(p: string, cwd: string): string { + return path.isAbsolute(p) ? p : path.resolve(cwd, p); +} + +/** + * Extracts CLI arguments from process.argv. Returns null when no subcommand is + * present (normal GUI launch). `firstArgIndex` should be 1 for packaged builds + * and 2 when running via `electron <app-path> ...`. + */ +export function parseCliArgs( + argv: string[], + firstArgIndex: number, + cwd: string = process.cwd(), +): CliCommand | null { + const rawArgs = argv.slice(firstArgIndex).filter((a) => !a.startsWith("--inspect")); + + // Skip *leading* Chromium/Electron switches (e.g. the AppImage's required + // `--no-sandbox`) so `Openscreen --no-sandbox export demo.openscreen` still + // enters CLI mode. Only leading dash-tokens are skipped — everything after + // the subcommand belongs to the subcommand parser. `--help`/`-h` are ours. + let subIndex = 0; + while ( + subIndex < rawArgs.length && + rawArgs[subIndex].startsWith("-") && + rawArgs[subIndex] !== "--help" && + rawArgs[subIndex] !== "-h" + ) { + subIndex++; + } + + const args = rawArgs.slice(subIndex); + const sub = args[0]; + if (!sub || !SUBCOMMANDS.has(sub)) return null; + if (sub === "help" || sub === "--help" || sub === "-h") return { kind: "help" }; + + try { + if (sub === "export") return parseExport(args.slice(1), cwd); + if (sub === "record") return parseRecord(args.slice(1), cwd); + if (sub === "sources") return parseSources(args.slice(1)); + if (sub === "pack") return parsePack(args.slice(1), cwd); + if (sub === "captions") return parseCaptions(args.slice(1), cwd); + return parseInfo(args.slice(1), cwd); + } catch (error) { + return { kind: "error", message: error instanceof Error ? error.message : String(error) }; + } +} + +function parseExport(args: string[], cwd: string): CliCommand { + const request: CliExportRequest & { json?: boolean } = { + kind: "export", + projectPath: "", + outPath: null, + format: null, + quality: null, + gifFrameRate: null, + gifSizePreset: null, + autoZoom: false, + audioPath: null, + audioMode: "mix", + audioOffsetSec: 0, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + switch (arg) { + case "-o": + case "--out": { + const [value, next] = takeValue(args, i, arg); + request.outPath = resolvePath(value, cwd); + i = next; + break; + } + case "--format": { + const [value, next] = takeValue(args, i, arg); + if (value !== "mp4" && value !== "gif") { + throw new Error(`--format must be mp4 or gif, got "${value}"`); + } + request.format = value; + i = next; + break; + } + case "--quality": { + const [value, next] = takeValue(args, i, arg); + if (value !== "medium" && value !== "good" && value !== "source") { + throw new Error(`--quality must be medium, good or source, got "${value}"`); + } + request.quality = value; + i = next; + break; + } + case "--gif-fps": { + const [value, next] = takeValue(args, i, arg); + const fps = Number(value); + if (fps !== 15 && fps !== 20 && fps !== 25 && fps !== 30) { + throw new Error(`--gif-fps must be 15, 20, 25 or 30, got "${value}"`); + } + request.gifFrameRate = fps; + i = next; + break; + } + case "--gif-size": { + const [value, next] = takeValue(args, i, arg); + if (value !== "medium" && value !== "large" && value !== "original") { + throw new Error(`--gif-size must be medium, large or original, got "${value}"`); + } + request.gifSizePreset = value; + i = next; + break; + } + case "--auto-zoom": + request.autoZoom = true; + break; + case "--audio": { + const [value, next] = takeValue(args, i, arg); + request.audioPath = resolvePath(value, cwd); + i = next; + break; + } + case "--audio-mode": { + const [value, next] = takeValue(args, i, arg); + if (value !== "mix" && value !== "replace") { + throw new Error(`--audio-mode must be mix or replace, got "${value}"`); + } + request.audioMode = value; + i = next; + break; + } + case "--audio-offset": { + const [value, next] = takeValue(args, i, arg); + const seconds = Number(value); + if (!Number.isFinite(seconds) || seconds < 0) { + throw new Error( + `--audio-offset must be a non-negative number of seconds, got "${value}"`, + ); + } + request.audioOffsetSec = seconds; + i = next; + break; + } + case "--json": + request.json = true; + break; + default: + if (arg.startsWith("-")) throw new Error(`Unknown export option: ${arg}`); + if (request.projectPath) throw new Error(`Unexpected extra argument: ${arg}`); + request.projectPath = resolvePath(arg, cwd); + } + } + + if (!request.projectPath) throw new Error("export requires a <project.openscreen> path"); + if (request.outPath) { + const ext = path.extname(request.outPath).toLowerCase(); + if (ext !== ".mp4" && ext !== ".gif") { + throw new Error(`--out must end in .mp4 or .gif, got "${request.outPath}"`); + } + const extFormat = ext === ".gif" ? "gif" : "mp4"; + if (request.format && request.format !== extFormat) { + throw new Error(`--format ${request.format} conflicts with --out extension ${ext}`); + } + request.format = extFormat; + } + // After --out resolves the format, so `--audio … -o out.gif` is caught too. + if (request.audioPath && request.format === "gif") { + throw new Error("--audio is only supported for MP4 exports"); + } + return request; +} + +function parseRecord(args: string[], cwd: string): CliCommand { + const request: CliRecordRequest & { json?: boolean } = { + kind: "record", + displayIndex: 0, + windowTitle: null, + mic: false, + micDevice: null, + systemAudio: false, + cursorMode: "editable-overlay", + durationMs: null, + projectOut: null, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + switch (arg) { + case "--display": { + const [value, next] = takeValue(args, i, arg); + const index = Number(value); + if (!Number.isInteger(index) || index < 0) { + throw new Error(`--display must be a non-negative integer, got "${value}"`); + } + request.displayIndex = index; + i = next; + break; + } + case "--window": { + const [value, next] = takeValue(args, i, arg); + request.windowTitle = value; + i = next; + break; + } + case "--mic": + request.mic = true; + break; + case "--mic-device": { + const [value, next] = takeValue(args, i, arg); + request.mic = true; + request.micDevice = value; + i = next; + break; + } + case "--system-audio": + request.systemAudio = true; + break; + case "--cursor": { + const [value, next] = takeValue(args, i, arg); + if (value !== "editable-overlay" && value !== "system") { + throw new Error(`--cursor must be editable-overlay or system, got "${value}"`); + } + request.cursorMode = value; + i = next; + break; + } + case "--duration": { + const [value, next] = takeValue(args, i, arg); + const seconds = Number(value); + if (!Number.isFinite(seconds) || seconds <= 0) { + throw new Error(`--duration must be a positive number of seconds, got "${value}"`); + } + request.durationMs = Math.round(seconds * 1000); + i = next; + break; + } + case "--project": { + const [value, next] = takeValue(args, i, arg); + if (!value.endsWith(".openscreen")) { + throw new Error(`--project must end in .openscreen, got "${value}"`); + } + request.projectOut = resolvePath(value, cwd); + i = next; + break; + } + case "--json": + request.json = true; + break; + default: + throw new Error(`Unknown record option: ${arg}`); + } + } + return request; +} + +function parseSources(args: string[]): CliCommand { + let json = false; + for (const arg of args) { + if (arg === "--json") { + json = true; + } else { + throw new Error(`Unknown sources option: ${arg}`); + } + } + return { kind: "sources", json }; +} + +export interface CliPackCommand { + kind: "pack"; + projectPath: string; + outDir: string; +} + +function parsePack(args: string[], cwd: string): CliCommand { + let projectPath = ""; + let outDir = ""; + let json = false; + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + if (arg === "-o" || arg === "--out") { + const [value, next] = takeValue(args, i, arg); + outDir = resolvePath(value, cwd); + i = next; + } else if (arg === "--json") { + json = true; + } else if (arg.startsWith("-")) { + throw new Error(`Unknown pack option: ${arg}`); + } else if (projectPath) { + throw new Error(`Unexpected extra argument: ${arg}`); + } else { + projectPath = resolvePath(arg, cwd); + } + } + if (!projectPath) throw new Error("pack requires a <project.openscreen> path"); + if (!outDir) throw new Error("pack requires --out <directory>"); + return { kind: "pack", projectPath, outDir, json }; +} + +function parseCaptions(args: string[], cwd: string): CliCommand { + let projectPath = ""; + let minWordsPerCaption = 2; + let maxWordsPerCaption = 7; + let json = false; + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + if (arg === "--min-words" || arg === "--max-words") { + const [value, next] = takeValue(args, i, arg); + const count = Number(value); + if (!Number.isInteger(count) || count < 1) { + throw new Error(`${arg} must be a positive integer, got "${value}"`); + } + if (arg === "--min-words") minWordsPerCaption = count; + else maxWordsPerCaption = count; + i = next; + } else if (arg === "--json") { + json = true; + } else if (arg.startsWith("-")) { + throw new Error(`Unknown captions option: ${arg}`); + } else if (projectPath) { + throw new Error(`Unexpected extra argument: ${arg}`); + } else { + projectPath = resolvePath(arg, cwd); + } + } + if (!projectPath) throw new Error("captions requires a <project.openscreen> path"); + if (minWordsPerCaption > maxWordsPerCaption) { + throw new Error("--min-words cannot exceed --max-words"); + } + return { kind: "captions", projectPath, minWordsPerCaption, maxWordsPerCaption, json }; +} + +function parseInfo(args: string[], cwd: string): CliCommand { + let projectPath = ""; + let json = false; + for (const arg of args) { + if (arg === "--json") { + json = true; + } else if (arg.startsWith("-")) { + throw new Error(`Unknown info option: ${arg}`); + } else if (projectPath) { + throw new Error(`Unexpected extra argument: ${arg}`); + } else { + projectPath = resolvePath(arg, cwd); + } + } + if (!projectPath) throw new Error("info requires a <project.openscreen> path"); + return { kind: "info", projectPath, json }; +} diff --git a/electron/cli/cliMain.ts b/electron/cli/cliMain.ts new file mode 100644 index 000000000..b5f02afdf --- /dev/null +++ b/electron/cli/cliMain.ts @@ -0,0 +1,469 @@ +// Headless CLI mode: boots Electron without HUD/tray/menu, drives a hidden +// renderer window (windowType=cli-export | cli-record) that reuses the app's +// existing export and recording pipelines, and reports progress on stdio. + +import fs from "node:fs/promises"; +import path from "node:path"; +import readline from "node:readline"; +import { fileURLToPath } from "node:url"; +import { app, BrowserWindow, ipcMain, session, systemPreferences } from "electron"; +import type { + CliDoneResult, + CliProgressEvent, + CliRequest, + CliSourcesResult, +} from "../../src/lib/cliContracts"; +import { getSelectedDesktopSource, registerIpcHandlers } from "../ipc/handlers"; +import { registerSttIpc } from "../stt"; +import { ASSET_BASE_URL_ARG } from "../windows"; +import { CLI_USAGE, type CliCommand } from "./args"; +import { runInfoCommand, runPackCommand } from "./projectCommands"; + +const __dirname = path.dirname(fileURLToPath(import.meta.url)); +const VITE_DEV_SERVER_URL = process.env["VITE_DEV_SERVER_URL"]; +const RENDERER_DIST = path.join(__dirname, "..", "dist"); + +interface CliOutput { + json: boolean; + event(event: string, data?: Record<string, unknown>): void; + info(message: string): void; + error(message: string): void; + progress(p: CliProgressEvent): void; +} + +// stdout/stderr may be a closed pipe (`openscreen export | head`); writes must +// never take the process down — Electron would show a GUI error dialog. +function safeWrite(stream: NodeJS.WriteStream, text: string): void { + try { + stream.write(text); + } catch { + // EPIPE or closed stream; drop the output. + } +} + +function createOutput(json: boolean): CliOutput { + const isTty = process.stdout.isTTY === true; + let progressLineActive = false; + let lastProgressText = ""; + const clearProgressLine = () => { + if (progressLineActive) { + safeWrite(process.stdout, "\n"); + progressLineActive = false; + } + }; + return { + json, + event(event, data = {}) { + if (json) { + safeWrite(process.stdout, `${JSON.stringify({ event, ...data })}\n`); + } + }, + info(message) { + if (json) return; + clearProgressLine(); + safeWrite(process.stdout, `${message}\n`); + }, + error(message) { + clearProgressLine(); + if (json) { + safeWrite(process.stdout, `${JSON.stringify({ event: "error", message })}\n`); + } else { + safeWrite(process.stderr, `Error: ${message}\n`); + } + }, + progress(p) { + if (json) { + safeWrite(process.stdout, `${JSON.stringify({ event: "progress", ...p })}\n`); + return; + } + const frames = + p.currentFrame !== undefined && p.totalFrames + ? ` frame ${p.currentFrame}/${p.totalFrames}` + : ""; + const eta = + p.estimatedTimeRemaining !== undefined && Number.isFinite(p.estimatedTimeRemaining) + ? ` ETA ${Math.max(0, Math.round(p.estimatedTimeRemaining))}s` + : ""; + const phase = p.phase ? ` [${p.phase}]` : ""; + const text = `Exporting ${Math.round(p.percentage)}%${frames}${eta}${phase}`; + if (isTty) { + if (text === lastProgressText) return; + lastProgressText = text; + safeWrite(process.stdout, `\r\x1b[2K${text}`); + progressLineActive = true; + } else { + // Piped/non-TTY consumers get one line per whole-percent (or phase) change. + const coarse = `${Math.round(p.percentage)}%${phase}`; + if (coarse === lastProgressText) return; + lastProgressText = coarse; + safeWrite(process.stdout, `${text}\n`); + } + }, + }; +} + +function loadRunnerWindow(windowType: string): BrowserWindow { + const win = new BrowserWindow({ + width: 1280, + height: 720, + show: false, + webPreferences: { + preload: path.join(__dirname, "preload.mjs"), + additionalArguments: [ASSET_BASE_URL_ARG], + nodeIntegration: false, + contextIsolation: true, + // Same relaxation as the editor window: exporters load recording media + // via file:// URLs. + webSecurity: false, + backgroundThrottling: false, + }, + }); + + if (VITE_DEV_SERVER_URL) { + win.loadURL(`${VITE_DEV_SERVER_URL}?windowType=${windowType}`); + } else { + win.loadFile(path.join(RENDERER_DIST, "index.html"), { query: { windowType } }); + } + return win; +} + +function registerAppHandlersForCli(cliWindowRef: () => BrowserWindow | null) { + // The recording/export pipelines are driven through the same IPC surface the + // GUI uses. Window-management callbacks become no-ops; "switch-to-editor" + // (fired by the recorder hook after it stores a finished session) is handled + // by the runner itself, so an inert factory is enough. + const noop = () => { + // Intentionally empty: CLI mode has no HUD/tray/editor windows to manage. + }; + const notAvailable = () => { + throw new Error("Window not available in CLI mode"); + }; + registerIpcHandlers( + noop, // createEditorWindow: recording finished; runner drives completion + notAvailable, // createSourceSelectorWindow + notAvailable, // createCountdownOverlayWindow + notAvailable, // createNotesWindow + cliWindowRef, + () => null, + () => null, + () => null, + noop, // onRecordingStateChange: no tray to update + noop, // switchToHud + ); +} + +function printSources(output: CliOutput, sources: CliSourcesResult): void { + if (output.json) { + // The "done" event already carries the payload; nothing extra to print. + return; + } + const lines: string[] = []; + lines.push("Displays:"); + for (const display of sources.displays) { + lines.push(` ${display.index} ${display.name} (${display.id})`); + } + lines.push("Windows:"); + if (sources.windows.length === 0) { + lines.push(" (none)"); + } + for (const win of sources.windows) { + lines.push(` - ${win.name}`); + } + lines.push("Microphones:"); + if (sources.microphoneLabelsUnavailable) { + lines.push(" (labels unavailable — grant microphone permission to see device names)"); + } else if (sources.microphones.length === 0) { + lines.push(" (none)"); + } + for (const mic of sources.microphones) { + lines.push(` - ${mic.label}`); + } + output.info(lines.join("\n")); +} + +async function writeProjectFile(projectOut: string, projectData: unknown): Promise<void> { + await fs.mkdir(path.dirname(projectOut), { recursive: true }); + await fs.writeFile(projectOut, JSON.stringify(projectData, null, 2), "utf8"); +} + +function setupRecordStopSignals(stop: (reason: string) => void): void { + // SIGINT covers Ctrl+C everywhere; SIGTERM never fires on Windows, where + // stdin "stop" or --duration are the graceful alternatives (see docs/cli.md). + process.on("SIGINT", () => stop("SIGINT")); + process.on("SIGTERM", () => stop("SIGTERM")); + try { + // Touching process.stdin can throw on Windows GUI-subsystem builds when + // spawned with stdio "ignore"; signals and --duration still work then. + const rl = readline.createInterface({ input: process.stdin }); + rl.on("line", (line) => { + const trimmed = line.trim().toLowerCase(); + if (trimmed === "stop" || trimmed === "q" || trimmed === "quit") { + stop("stdin"); + } + }); + rl.on("close", () => { + // stdin EOF is not a stop signal: agents may spawn the CLI with + // stdin closed and stop it via SIGINT/--duration instead. + }); + } catch { + // stdin unavailable; signals and --duration still work. + } +} + +/** Both file-only commands write through the same EPIPE-safe stdout writer. */ +const writeStdout = (text: string) => safeWrite(process.stdout, text); + +export function runCli(command: CliCommand): void { + if (command.kind === "help") { + safeWrite(process.stdout, CLI_USAGE); + app.exit(0); + return; + } + if (command.kind === "error") { + safeWrite(process.stderr, `Error: ${command.message}\n\n${CLI_USAGE}`); + app.exit(2); + return; + } + + const output = createOutput(command.json === true); + + // stdout belongs to the CLI protocol (NDJSON / progress); reroute the app's + // own console chatter (e.g. "[native-sck] starting…") to stderr. + const stringifyArg = (value: unknown): string => { + if (typeof value === "string") return value; + if (value instanceof Error) return value.stack ?? value.message; + try { + return JSON.stringify(value) ?? String(value); + } catch { + return String(value); + } + }; + for (const level of ["log", "info", "warn", "error", "debug"] as const) { + console[level] = (...args: unknown[]) => { + safeWrite(process.stderr, `${args.map(stringifyArg).join(" ")}\n`); + }; + } + + // A consumer closing the pipe (`openscreen export | head`) must not crash the + // process, and main-process exceptions must never surface as Electron's GUI + // error dialog — report on stderr and exit non-zero instead. + const ignoreStreamError = () => { + // Intentionally empty: EPIPE from a closed consumer is not an error here. + }; + process.stdout.on("error", ignoreStreamError); + process.stderr.on("error", ignoreStreamError); + process.on("uncaughtException", (error) => { + safeWrite(process.stderr, `Fatal: ${error?.stack ?? String(error)}\n`); + app.exit(1); + }); + process.on("unhandledRejection", (reason) => { + safeWrite( + process.stderr, + `Fatal (unhandled rejection): ${reason instanceof Error ? (reason.stack ?? reason.message) : String(reason)}\n`, + ); + app.exit(1); + }); + + // Set once cli-done has been received; suppresses the window-all-closed + // failure path during the normal teardown race after a successful run. + let finished = false; + + // GPU may be unavailable in CI/servers; let Chromium fall back to SwiftShader + // so the WebGL-based export renderer still works. + app.commandLine.appendSwitch("enable-unsafe-swiftshader"); + + // Never show the dock icon for CLI runs. + if (process.platform === "darwin") { + app.dock?.hide(); + } + + app.on("window-all-closed", () => { + // Completion is signalled via cli-done; a vanished window is a failure + // only while the run is still in flight. + if (finished) return; + output.error("Renderer window closed unexpectedly"); + app.exit(1); + }); + + void app + .whenReady() + .then(async () => { + if (command.kind === "info") { + const code = await runInfoCommand(command.projectPath, command.json === true, writeStdout); + app.exit(code); + return; + } + + if (command.kind === "pack") { + const code = await runPackCommand( + command.projectPath, + command.outDir, + command.json === true, + writeStdout, + ); + app.exit(code); + return; + } + + await fs.mkdir(path.join(app.getPath("userData"), "recordings"), { recursive: true }); + + // Media/screen permissions for the renderer (mic metering, future browser + // capture paths). Mirrors the GUI allowlist. + const allowed = [ + "media", + "audioCapture", + "microphone", + "videoCapture", + "camera", + "screen", + "display-capture", + ]; + session.defaultSession.setPermissionCheckHandler((_wc, permission) => + allowed.includes(permission), + ); + session.defaultSession.setPermissionRequestHandler((_wc, permission, callback) => + callback(allowed.includes(permission)), + ); + + // Browser-pipeline recording fallback (e.g. Linux, missing native helper) + // resolves the pre-selected source exactly like the GUI does. + session.defaultSession.setDisplayMediaRequestHandler( + (request, callback) => { + const source = getSelectedDesktopSource(); + if (!request.videoRequested || !source) { + callback({}); + return; + } + callback({ + video: source, + ...(request.audioRequested && process.platform === "win32" + ? { audio: "loopback" as const } + : {}), + }); + }, + { useSystemPicker: false }, + ); + + if (command.kind === "record" && command.mic && process.platform === "darwin") { + const micStatus = systemPreferences.getMediaAccessStatus("microphone"); + if (micStatus !== "granted") { + await systemPreferences.askForMediaAccess("microphone"); + } + } + + let cliWindow: BrowserWindow | null = null; + registerAppHandlersForCli(() => cliWindow); + + // Speech-to-text backs the captions command; registered by the GUI boot + // path (main.ts) rather than registerIpcHandlers. + registerSttIpc(ipcMain); + + // Registered by the GUI boot path (main.ts) rather than registerIpcHandlers; + // the renderer's i18n init invokes it unconditionally. + ipcMain.handle("set-locale", () => { + // Locale only affects GUI menus/tray, which do not exist in CLI mode. + }); + ipcMain.handle("update-global-shortcut", () => ({ success: false })); + + const request: CliRequest = command; + ipcMain.handle("cli-get-request", () => request); + ipcMain.on("cli-log", (_event, level: string, message: string) => { + if (level === "error") { + output.error(message); + } else { + output.info(message); + output.event("log", { message }); + } + }); + ipcMain.on("cli-progress", (_event, progress: CliProgressEvent) => { + output.progress(progress); + }); + + ipcMain.handle("cli-done", async (_event, result: CliDoneResult) => { + if (finished) return; + finished = true; + + try { + if (result.success && command.kind === "record" && command.projectOut) { + if (result.projectData !== undefined) { + await writeProjectFile(command.projectOut, result.projectData); + result.projectPath = command.projectOut; + } + } + if (result.success && command.kind === "captions" && result.projectData !== undefined) { + await writeProjectFile(command.projectPath, result.projectData); + } + } catch (error) { + result.success = false; + result.error = `Run succeeded but writing the project file failed: ${String(error)}`; + } + + if (result.success) { + for (const warning of result.warnings ?? []) { + output.info(`Warning: ${warning}`); + output.event("warning", { message: warning }); + } + if (command.kind === "sources" && result.sources) { + printSources(output, result.sources); + } else if (command.kind === "captions") { + output.info( + `Added ${result.captionCount ?? 0} caption annotation(s) → ${result.projectPath}`, + ); + } else if (command.kind === "export") { + output.info(`Exported ${result.format ?? ""} → ${result.outputPath}`); + } else { + output.info(`Recording saved → ${result.screenVideoPath}`); + if (result.cursorDataPath) output.info(`Cursor data → ${result.cursorDataPath}`); + if (result.projectPath) output.info(`Project → ${result.projectPath}`); + } + output.event("done", { ...result }); + } else { + output.error(result.error ?? "Unknown failure"); + output.event("done", { success: false, error: result.error }); + } + + // Give the renderer a beat to resolve the invoke before exiting. + setTimeout(() => app.exit(result.success ? 0 : 1), 50); + }); + + if (command.kind === "record") { + const stop = (reason: string) => { + output.info(`Stopping recording (${reason})…`); + output.event("stopping", { reason }); + cliWindow?.webContents.send("cli-stop-recording"); + }; + setupRecordStopSignals(stop); + } + + const windowType = { + export: "cli-export", + record: "cli-record", + sources: "cli-sources", + captions: "cli-captions", + }[command.kind]; + cliWindow = loadRunnerWindow(windowType); + + // Surface renderer console errors/warnings on stderr — the hidden window + // has no other way to show what went wrong (toasts are invisible). + cliWindow.webContents.on("console-message", (details) => { + if (details.level === "error" || details.level === "warning") { + safeWrite(process.stderr, `[renderer] ${details.message}\n`); + } + }); + + cliWindow.webContents.on("did-fail-load", (_e, code, description) => { + output.error(`Failed to load runner window: ${description} (${code})`); + app.exit(1); + }); + cliWindow.webContents.on("render-process-gone", (_e, details) => { + output.error(`Renderer crashed: ${details.reason}`); + app.exit(1); + }); + + output.event("started", { command: command.kind }); + }) + .catch((error) => { + output.error(error instanceof Error ? (error.stack ?? error.message) : String(error)); + app.exit(1); + }); +} diff --git a/electron/cli/projectCommands.test.ts b/electron/cli/projectCommands.test.ts new file mode 100644 index 000000000..0748ad44d --- /dev/null +++ b/electron/cli/projectCommands.test.ts @@ -0,0 +1,139 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { type PackedProjectData, runInfoCommand, runPackCommand } from "./projectCommands"; + +let root = ""; + +/** Absolute path inside the throwaway root, parents created. */ +async function make(relative: string, contents = "video-bytes"): Promise<string> { + const target = path.join(root, relative); + await fs.mkdir(path.dirname(target), { recursive: true }); + await fs.writeFile(target, contents, "utf8"); + return target; +} + +async function writeProject(relative: string, data: PackedProjectData): Promise<string> { + return make(relative, JSON.stringify(data)); +} + +const readProject = async (file: string): Promise<PackedProjectData> => + JSON.parse(await fs.readFile(file, "utf8")); + +/** Collects CLI output instead of writing to stdout. */ +function recorder() { + const chunks: string[] = []; + const write = (text: string) => { + chunks.push(text); + }; + return { write, text: () => chunks.join("") }; +} + +beforeEach(async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), "os-cli-pack-")); +}); + +afterEach(async () => { + await fs.rm(root, { recursive: true, force: true }); +}); + +describe("runPackCommand", () => { + it("keeps screen and webcam apart when they share a basename", async () => { + const screen = await make("rec/a/clip.mp4", "screen"); + const webcam = await make("rec/b/clip.mp4", "webcam"); + const project = await writeProject("demo.openscreen", { + version: 1, + media: { screenVideoPath: screen, webcamVideoPath: webcam }, + }); + const outDir = path.join(root, "packed"); + + const out = recorder(); + expect(await runPackCommand(project, outDir, false, out.write)).toBe(0); + + const packed = await readProject(path.join(outDir, "demo.openscreen")); + expect(packed.media?.screenVideoPath).not.toBe(packed.media?.webcamVideoPath); + await expect(fs.readFile(packed.media?.screenVideoPath ?? "", "utf8")).resolves.toBe("screen"); + await expect(fs.readFile(packed.media?.webcamVideoPath ?? "", "utf8")).resolves.toBe("webcam"); + }); + + it("copies the cursor sidecar and drops the legacy videoPath", async () => { + const screen = await make("rec/clip.mp4", "screen"); + await make("rec/clip.mp4.cursor.json", "[]"); + const project = await writeProject("demo.openscreen", { version: 1, videoPath: screen }); + const outDir = path.join(root, "packed"); + + const out = recorder(); + expect(await runPackCommand(project, outDir, true, out.write)).toBe(0); + + const packed = await readProject(path.join(outDir, "demo.openscreen")); + expect(packed.videoPath).toBeUndefined(); + expect(packed.media?.screenVideoPath).toBe(path.join(outDir, "clip.mp4")); + await expect(fs.readFile(path.join(outDir, "clip.mp4.cursor.json"), "utf8")).resolves.toBe( + "[]", + ); + expect(JSON.parse(out.text())).toMatchObject({ event: "done", cursorData: true }); + }); + + it("falls back to media sitting next to the project when the stored path is stale", async () => { + await make("moved/clip.mp4", "screen"); + const project = await writeProject("moved/demo.openscreen", { + version: 1, + media: { screenVideoPath: path.join(root, "gone", "clip.mp4") }, + }); + const outDir = path.join(root, "packed"); + + const out = recorder(); + expect(await runPackCommand(project, outDir, false, out.write)).toBe(0); + await expect(fs.readFile(path.join(outDir, "clip.mp4"), "utf8")).resolves.toBe("screen"); + }); + + it("fails when the referenced media is nowhere to be found", async () => { + const project = await writeProject("demo.openscreen", { + version: 1, + media: { screenVideoPath: path.join(root, "gone", "clip.mp4") }, + }); + + const out = recorder(); + await expect( + runPackCommand(project, path.join(root, "packed"), false, out.write), + ).rejects.toThrow(/Referenced media not found/); + }); +}); + +describe("runInfoCommand", () => { + it("exits 1 when the project's video is missing, 0 when it is there", async () => { + const missing = await writeProject("missing.openscreen", { + version: 1, + media: { screenVideoPath: path.join(root, "gone", "clip.mp4") }, + }); + const present = await writeProject("present.openscreen", { + version: 1, + media: { screenVideoPath: await make("rec/clip.mp4") }, + }); + + const out = recorder(); + expect(await runInfoCommand(missing, false, out.write)).toBe(1); + expect(out.text()).toContain("[MISSING]"); + expect(await runInfoCommand(present, false, out.write)).toBe(0); + }); + + it("counts timeline regions in --json mode", async () => { + const project = await writeProject("demo.openscreen", { + version: 1, + media: { screenVideoPath: await make("rec/clip.mp4") }, + editor: { zoomRegions: [{}, {}], trimRegions: [{}], exportFormat: "mp4" }, + }); + + const out = recorder(); + expect(await runInfoCommand(project, true, out.write)).toBe(0); + expect(JSON.parse(out.text())).toMatchObject({ + zoomRegions: 2, + trimRegions: 1, + speedRegions: 0, + annotationRegions: 0, + exportFormat: "mp4", + screenVideoExists: true, + }); + }); +}); diff --git a/electron/cli/projectCommands.ts b/electron/cli/projectCommands.ts new file mode 100644 index 000000000..30e211e8c --- /dev/null +++ b/electron/cli/projectCommands.ts @@ -0,0 +1,174 @@ +// The two CLI commands that only touch the project file and its media: +// `openscreen pack` and `openscreen info`. They live outside cliMain.ts so they +// carry no `electron` import and stay unit-testable — the caller passes the +// writer, so nothing here knows about process.stdout either. + +import fs from "node:fs/promises"; +import path from "node:path"; + +/** Writes one already-newline-terminated chunk of CLI output. */ +export type CliWriter = (text: string) => void; + +export interface PackedProjectData { + version?: number; + media?: { screenVideoPath?: string; webcamVideoPath?: string; cursorCaptureMode?: string }; + videoPath?: string; + editor?: Record<string, unknown>; +} + +const isFile = (candidate: string): Promise<boolean> => + fs + .stat(candidate) + .then((stats) => stats.isFile()) + .catch(() => false); + +/** Copies a project and everything it references into one portable folder. */ +export async function runPackCommand( + projectPath: string, + outDir: string, + json: boolean, + out: CliWriter, +): Promise<number> { + const emit = (message: string) => { + if (!json) out(`${message}\n`); + }; + + const raw = await fs.readFile(projectPath, "utf8"); + const data = JSON.parse(raw) as PackedProjectData; + const media = data.media ?? (data.videoPath ? { screenVideoPath: data.videoPath } : undefined); + const screenVideoPath = media?.screenVideoPath; + if (!screenVideoPath) { + throw new Error("Project file does not reference a screen video"); + } + + const projectDir = path.dirname(path.resolve(projectPath)); + const resolveSource = async (mediaPath: string): Promise<string> => { + if (await isFile(mediaPath)) return mediaPath; + // Moved project: the stored absolute path is stale but the media travelled + // with the .openscreen file. Same rule as the loader's sibling fallback. + const sibling = path.join(projectDir, path.basename(mediaPath)); + if (await isFile(sibling)) return sibling; + throw new Error(`Referenced media not found: ${mediaPath}`); + }; + + await fs.mkdir(outDir, { recursive: true }); + + const copied: string[] = []; + const copyIn = async (sourcePath: string): Promise<string> => { + const ext = path.extname(sourcePath); + const stem = path.basename(sourcePath, ext); + let destination = path.join(outDir, stem + ext); + // Screen and webcam can share a basename across directories; don't overwrite. + for (let n = 1; copied.includes(destination); n++) { + destination = path.join(outDir, `${stem}-${n}${ext}`); + } + if (path.resolve(sourcePath) !== path.resolve(destination)) { + await fs.copyFile(sourcePath, destination); + } + copied.push(destination); + return destination; + }; + + const screenSource = await resolveSource(screenVideoPath); + const newScreenPath = await copyIn(screenSource); + + let newWebcamPath: string | undefined; + if (media.webcamVideoPath) { + newWebcamPath = await copyIn(await resolveSource(media.webcamVideoPath)); + } + + // Cursor telemetry sidecar sits at "<video path>.cursor.json". + const cursorSidecar = `${screenSource}.cursor.json`; + const hasCursorData = await isFile(cursorSidecar); + if (hasCursorData) { + await copyIn(cursorSidecar); + } + + const packedProject: PackedProjectData = { + ...data, + media: { + ...media, + screenVideoPath: newScreenPath, + ...(newWebcamPath ? { webcamVideoPath: newWebcamPath } : {}), + }, + }; + delete packedProject.videoPath; + const packedProjectPath = path.join(outDir, path.basename(projectPath)); + await fs.writeFile(packedProjectPath, JSON.stringify(packedProject, null, 2), "utf8"); + + if (json) { + out( + `${JSON.stringify({ + event: "done", + success: true, + projectPath: packedProjectPath, + files: [packedProjectPath, ...copied], + cursorData: hasCursorData, + })}\n`, + ); + } else { + emit(`Packed project → ${packedProjectPath}`); + for (const file of copied) { + emit(` + ${path.basename(file)}`); + } + if (!hasCursorData) { + emit(" (no cursor telemetry sidecar found)"); + } + emit( + "The folder is self-contained: if the stored paths go stale after moving it, the loader falls back to files next to the project.", + ); + } + return 0; +} + +/** Prints what a project references and whether its media is still reachable. */ +export async function runInfoCommand( + projectPath: string, + json: boolean, + out: CliWriter, +): Promise<number> { + const raw = await fs.readFile(projectPath, "utf8"); + const data = JSON.parse(raw) as PackedProjectData; + const editor = data.editor ?? {}; + const count = (key: string) => + Array.isArray(editor[key]) ? (editor[key] as unknown[]).length : 0; + const screenVideoPath = data.media?.screenVideoPath ?? data.videoPath ?? null; + const mediaExists = screenVideoPath + ? await fs + .access(screenVideoPath) + .then(() => true) + .catch(() => false) + : false; + + const summary = { + projectPath, + version: data.version ?? null, + screenVideoPath, + screenVideoExists: mediaExists, + webcamVideoPath: data.media?.webcamVideoPath ?? null, + cursorCaptureMode: data.media?.cursorCaptureMode ?? null, + exportFormat: (editor.exportFormat as string) ?? null, + exportQuality: (editor.exportQuality as string) ?? null, + aspectRatio: (editor.aspectRatio as string) ?? null, + zoomRegions: count("zoomRegions"), + trimRegions: count("trimRegions"), + speedRegions: count("speedRegions"), + annotationRegions: count("annotationRegions"), + }; + + if (json) { + out(`${JSON.stringify(summary)}\n`); + } else { + out( + [ + `Project: ${summary.projectPath} (version ${summary.version ?? "?"})`, + `Video: ${summary.screenVideoPath ?? "(none)"}${mediaExists ? "" : " [MISSING]"}`, + `Webcam: ${summary.webcamVideoPath ?? "(none)"}`, + `Cursor: ${summary.cursorCaptureMode ?? "(unknown)"}`, + `Export: ${summary.exportFormat ?? "?"} / ${summary.exportQuality ?? "?"} / ${summary.aspectRatio ?? "?"}`, + `Timeline: ${summary.zoomRegions} zooms, ${summary.trimRegions} trims, ${summary.speedRegions} speed regions, ${summary.annotationRegions} annotations`, + ].join("\n") + "\n", + ); + } + return summary.screenVideoPath && !mediaExists ? 1 : 0; +} diff --git a/electron/electron-env.d.ts b/electron/electron-env.d.ts index dd761b0b0..bbac97658 100644 --- a/electron/electron-env.d.ts +++ b/electron/electron-env.d.ts @@ -394,6 +394,12 @@ interface Window { callback: (event: import("./stt/transcriptionContract").SttStatusEvent) => void, ) => () => void; }; + // CLI mode (hidden runner windows; see electron/cli/) + cliGetRequest: () => Promise<import("../src/lib/cliContracts").CliRequest>; + cliProgress: (progress: import("../src/lib/cliContracts").CliProgressEvent) => void; + cliLog: (level: "info" | "error", message: string) => void; + cliDone: (result: import("../src/lib/cliContracts").CliDoneResult) => Promise<void>; + onCliStopRecording: (callback: () => void) => () => void; setLocale: (locale: string) => Promise<void>; saveDiagnostic: (payload: { error: string; diff --git a/electron/ipc/handlers.ts b/electron/ipc/handlers.ts index 5e0c3af82..4baa9ee8f 100644 --- a/electron/ipc/handlers.ts +++ b/electron/ipc/handlers.ts @@ -378,13 +378,40 @@ async function getApprovedProjectSession( trustedDirs.push(path.dirname(path.resolve(projectFilePath))); } - const screenVideoPath = await approveReadableVideoPath(media.screenVideoPath, trustedDirs); + // Packed/portable projects: when the stored absolute path no longer exists + // (project moved to another machine or directory), fall back to a file with + // the same basename next to the project file (see `openscreen pack`). + const resolveWithSiblingFallback = async (mediaPath: string): Promise<string> => { + if (!projectFilePath) return mediaPath; + const exists = await fs + .stat(mediaPath) + .then((stats) => stats.isFile()) + .catch(() => false); + if (exists) return mediaPath; + const sibling = path.join( + path.dirname(path.resolve(projectFilePath)), + path.basename(mediaPath), + ); + const siblingExists = await fs + .stat(sibling) + .then((stats) => stats.isFile()) + .catch(() => false); + return siblingExists ? sibling : mediaPath; + }; + + const screenVideoPath = await approveReadableVideoPath( + await resolveWithSiblingFallback(media.screenVideoPath), + trustedDirs, + ); if (!screenVideoPath) { throw new Error("Project references an invalid or unsupported screen video path"); } const webcamVideoPath = media.webcamVideoPath - ? await approveReadableVideoPath(media.webcamVideoPath, trustedDirs) + ? await approveReadableVideoPath( + await resolveWithSiblingFallback(media.webcamVideoPath), + trustedDirs, + ) : undefined; if (media.webcamVideoPath && !webcamVideoPath) { throw new Error("Project references an invalid or unsupported webcam video path"); diff --git a/electron/main.ts b/electron/main.ts index 1fa013935..7549121f3 100644 --- a/electron/main.ts +++ b/electron/main.ts @@ -12,6 +12,8 @@ import { Tray, } from "electron"; import { ShortcutBinding } from "../src/lib/shortcuts"; +import { parseCliArgs } from "./cli/args"; +import { runCli } from "./cli/cliMain"; import { isDiagnosticModeEnabled, mainLogBuffer } from "./diagnostics/main-log-buffer"; import { loadAndRegisterGlobalShortcut, @@ -33,6 +35,10 @@ import { const __dirname = path.dirname(fileURLToPath(import.meta.url)); +// CLI mode: `openscreen export|record|info|help ...` runs headless without +// HUD/tray/menu. Parsed before any GUI side effects; see electron/cli/. +const cliCommand = parseCliArgs(process.argv, app.isPackaged ? 1 : 2); + // Use Screen & System Audio Recording permissions instead of the CoreAudio Tap API on macOS. // Tap needs NSAudioCaptureUsageDescription in the parent app's Info.plist, which breaks when // running from a terminal/IDE during dev. @@ -126,11 +132,15 @@ function showMainWindow() { createWindow(); } -const stableInstanceLock = acquireStableInstanceLock(); -const hasElectronSingleInstanceLock = app.requestSingleInstanceLock(); +// CLI runs skip the single-instance lock so `openscreen export/record` works +// while the GUI app is open (they share nothing but the recordings directory). +const stableInstanceLock = cliCommand ? null : acquireStableInstanceLock(); +const hasElectronSingleInstanceLock = cliCommand ? false : app.requestSingleInstanceLock(); const hasSingleInstanceLock = Boolean(stableInstanceLock && hasElectronSingleInstanceLock); -if (hasSingleInstanceLock) { +if (cliCommand) { + runCli(cliCommand); +} else if (hasSingleInstanceLock) { app.on("second-instance", () => { showMainWindow(); }); @@ -482,11 +492,15 @@ function createCountdownOverlayWindowWrapper() { // Closing every window quits the app (tray goes too). The in-app "Return to Recorder" // button covers the editor-to-HUD round-trip, so closing the last window means "I'm done". -app.on("window-all-closed", () => { - app.quit(); -}); +// CLI mode owns its own lifecycle (see electron/cli/cliMain.ts). +if (!cliCommand) { + app.on("window-all-closed", () => { + app.quit(); + }); +} app.on("activate", () => { + if (cliCommand) return; // On macOS, re-open a window when the dock icon is clicked and none are open. const hasVisibleWindow = BrowserWindow.getAllWindows().some((window) => { if (window.isDestroyed() || !window.isVisible()) { @@ -507,7 +521,7 @@ app.on("will-quit", () => { stableInstanceLock?.release(); }); -const appReady = hasSingleInstanceLock ? app.whenReady() : null; +const appReady = !cliCommand && hasSingleInstanceLock ? app.whenReady() : null; appReady?.then(async () => { if (isDiagnosticModeEnabled()) { diff --git a/electron/preload.ts b/electron/preload.ts index dfc9db84f..f3b44d57d 100644 --- a/electron/preload.ts +++ b/electron/preload.ts @@ -428,4 +428,22 @@ contextBridge.exposeInMainWorld("electronAPI", { return () => ipcRenderer.removeListener("stt:status", listener); }, }, + // --- CLI mode (hidden runner windows; see electron/cli/) --- + cliGetRequest: (): Promise<import("../src/lib/cliContracts").CliRequest> => { + return ipcRenderer.invoke("cli-get-request"); + }, + cliProgress: (progress: import("../src/lib/cliContracts").CliProgressEvent) => { + ipcRenderer.send("cli-progress", progress); + }, + cliLog: (level: "info" | "error", message: string) => { + ipcRenderer.send("cli-log", level, message); + }, + cliDone: (result: import("../src/lib/cliContracts").CliDoneResult) => { + return ipcRenderer.invoke("cli-done", result); + }, + onCliStopRecording: (callback: () => void) => { + const listener = () => callback(); + ipcRenderer.on("cli-stop-recording", listener); + return () => ipcRenderer.removeListener("cli-stop-recording", listener); + }, }); diff --git a/electron/windows.ts b/electron/windows.ts index bf219ca9c..687a3a175 100644 --- a/electron/windows.ts +++ b/electron/windows.ts @@ -88,7 +88,7 @@ function applyContentProtection(win: BrowserWindow, label: string) { const ASSET_BASE_DIR = process.defaultApp ? path.join(__dirname, "..", "public") : process.resourcesPath; -const ASSET_BASE_URL_ARG = `--asset-base-url=${pathToFileURL(`${ASSET_BASE_DIR}${path.sep}`).toString()}`; +export const ASSET_BASE_URL_ARG = `--asset-base-url=${pathToFileURL(`${ASSET_BASE_DIR}${path.sep}`).toString()}`; let hudOverlayWindow: BrowserWindow | null = null; diff --git a/package.json b/package.json index d223b763b..2adc681ff 100644 --- a/package.json +++ b/package.json @@ -62,6 +62,7 @@ "diagnostic:run": "node scripts/diagnostic-tool/diagnostic.mjs", "diagnostic:smoke:win": "node scripts/diagnostic-tool/diagnostic.mjs --duration 3", "build-vite": "tsc && vite build", + "cli": "electron .", "test:e2e": "playwright test", "test:e2e:windows-native-checklist": "playwright test tests/e2e/windows-native-checklist.spec.ts", "prepare": "husky", diff --git a/src/App.tsx b/src/App.tsx index f0e4d1154..53d52c742 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -20,6 +20,10 @@ const VideoEditorEntry = lazy(() => default: module.default, })), ); +const CliExportRunner = lazy(() => import("./cli/CliExportRunner")); +const CliRecordRunner = lazy(() => import("./cli/CliRecordRunner")); +const CliSourcesRunner = lazy(() => import("./cli/CliSourcesRunner")); +const CliCaptionsRunner = lazy(() => import("./cli/CliCaptionsRunner")); const ShortcutsConfigDialog = lazy(() => import("./components/video-editor/ShortcutsConfigDialog").then((module) => ({ default: module.ShortcutsConfigDialog, @@ -76,6 +80,30 @@ export default function App() { return <SourceSelector />; case "countdown-overlay": return <CountdownOverlay />; + case "cli-export": + return ( + <Suspense fallback={null}> + <CliExportRunner /> + </Suspense> + ); + case "cli-record": + return ( + <Suspense fallback={null}> + <CliRecordRunner /> + </Suspense> + ); + case "cli-sources": + return ( + <Suspense fallback={null}> + <CliSourcesRunner /> + </Suspense> + ); + case "cli-captions": + return ( + <Suspense fallback={null}> + <CliCaptionsRunner /> + </Suspense> + ); case "editor": return ( <ShortcutsProvider> diff --git a/src/cli/CliCaptionsRunner.tsx b/src/cli/CliCaptionsRunner.tsx new file mode 100644 index 000000000..620210ee2 --- /dev/null +++ b/src/cli/CliCaptionsRunner.tsx @@ -0,0 +1,168 @@ +// Hidden-window runner for `openscreen captions`: transcribes the project's +// audio with the on-device Whisper worker and writes the resulting caption +// annotations back into the project. Mirrors VideoEditor.generateAutoCaptions. + +import { useEffect, useRef, useState } from "react"; +import { + normalizeProjectEditor, + resolveProjectMedia, + toFileUrl, + validateProjectData, +} from "@/components/video-editor/projectPersistence"; +import type { AnnotationRegion, TrimRegion } from "@/components/video-editor/types"; +import { extractMono16kFromVideoUrl } from "@/lib/captioning/extractMono16k"; +import { transcribeMono16kToSegments } from "@/lib/captioning/transcribe"; +import type { CliCaptionsRequest, CliDoneResult } from "@/lib/cliContracts"; +import { nativeBridgeClient } from "@/native"; +import { captionSegmentsToAnnotationRegions } from "./captionAnnotations"; +import { + shiftTrimRegionsMsForCaptionBuffer, + trimLeadingSilenceMono16k, +} from "./vendor/leadingSilence"; + +/** Highest trailing number across existing region ids, so new ids never collide. */ +function nextNumericIdFrom(regions: { id: string }[]): number { + let max = 0; + for (const region of regions) { + const match = /(\d+)$/.exec(region.id); + if (match) max = Math.max(max, Number(match[1])); + } + return max + 1; +} + +async function runCaptions(request: CliCaptionsRequest): Promise<CliDoneResult> { + const loaded = await nativeBridgeClient.project.loadProjectFileFromPath(request.projectPath); + if (!loaded.success || loaded.project === undefined) { + throw new Error(loaded.error ?? loaded.message ?? "Failed to load project file"); + } + if (!validateProjectData(loaded.project)) { + throw new Error("Project file is not a valid .openscreen project"); + } + const project = loaded.project; + const media = resolveProjectMedia(project); + if (!media) { + throw new Error("Project file does not reference any recorded media"); + } + const editor = normalizeProjectEditor(project.editor ?? {}); + const trimRegions: TrimRegion[] = editor.trimRegions; + + window.electronAPI.cliLog("info", "Extracting audio…"); + const videoUrl = toFileUrl(media.screenVideoPath); + const { samples, durationSec } = await extractMono16kFromVideoUrl(videoUrl); + if (!Number.isFinite(durationSec) || durationSec <= 0 || samples.length < 800) { + throw new Error("The project's video has no usable audio track to transcribe"); + } + + const { samples: speechSamples, trimSec } = trimLeadingSilenceMono16k(samples); + if (speechSamples.length < 800) { + throw new Error("No speech detected in the project's audio"); + } + + const trimMs = Math.round(trimSec * 1000); + const trimRegionsForTranscribe = shiftTrimRegionsMsForCaptionBuffer(trimRegions, trimMs); + + const transcribeOptions = { + onStatus: (phase: "model" | "transcribe") => { + window.electronAPI.cliLog( + "info", + phase === "model" ? "Loading caption model…" : "Transcribing…", + ); + }, + }; + + let { segments: segmentsRaw, granularity } = await transcribeMono16kToSegments(speechSamples, { + trimRegions: trimRegionsForTranscribe, + ...transcribeOptions, + }); + let transcribedFromTrimmedBuffer = true; + + // Leading-silence trimming can return empty even when the full source has + // speech. Retry once against the untrimmed buffer before giving up. + if (segmentsRaw.length === 0 && trimSec > 0) { + ({ segments: segmentsRaw, granularity } = await transcribeMono16kToSegments(samples, { + trimRegions, + ...transcribeOptions, + })); + transcribedFromTrimmedBuffer = false; + } + + const segments = + transcribedFromTrimmedBuffer && trimSec > 0 + ? segmentsRaw.map((segment) => ({ + ...segment, + startSec: segment.startSec + trimSec, + endSec: segment.endSec + trimSec, + })) + : segmentsRaw; + + // Re-running the command replaces earlier auto-captions instead of stacking + // duplicates; manually added annotations are preserved. + const manualAnnotations: AnnotationRegion[] = editor.annotationRegions.filter( + (annotation) => annotation.annotationSource !== "auto-caption", + ); + const startNumericId = nextNumericIdFrom([...editor.annotationRegions, ...editor.zoomRegions]); + const startZIndex = manualAnnotations.reduce((max, a) => Math.max(max, a.zIndex + 1), 1); + + let regions = captionSegmentsToAnnotationRegions(segments, startNumericId, startZIndex, { + minWordsPerCaption: request.minWordsPerCaption, + maxWordsPerCaption: request.maxWordsPerCaption, + timestampGranularity: granularity, + }); + if (regions.length === 0 && segments.length > 0) { + regions = captionSegmentsToAnnotationRegions(segments, startNumericId, startZIndex, { + minWordsPerCaption: 1, + maxWordsPerCaption: Number.MAX_SAFE_INTEGER, + timestampGranularity: granularity, + }); + } + if (regions.length === 0) { + throw new Error("Transcription produced no caption segments"); + } + + const updatedProject = { + ...project, + editor: { + ...editor, + annotationRegions: [...manualAnnotations, ...regions], + }, + }; + + return { + success: true, + projectPath: request.projectPath, + projectData: updatedProject, + captionCount: regions.length, + }; +} + +export function CliCaptionsRunner() { + const startedRef = useRef(false); + const [status] = useState("Generating captions…"); + + useEffect(() => { + if (startedRef.current) return; + startedRef.current = true; + + void (async () => { + try { + const request = await window.electronAPI.cliGetRequest(); + if (request.kind !== "captions") { + throw new Error(`cli-captions window received a ${request.kind} request`); + } + const result = await runCaptions(request); + await window.electronAPI.cliDone(result); + } catch (error) { + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + await window.electronAPI.cliDone({ success: false, error: message }); + } + })(); + }, []); + + return ( + <div className="flex h-screen items-center justify-center bg-[#09090b] text-white/60 text-sm"> + {status} + </div> + ); +} + +export default CliCaptionsRunner; diff --git a/src/cli/CliExportRunner.tsx b/src/cli/CliExportRunner.tsx new file mode 100644 index 000000000..f9c57c682 --- /dev/null +++ b/src/cli/CliExportRunner.tsx @@ -0,0 +1,361 @@ +// Hidden-window runner for `openscreen export`. Loads an .openscreen project, +// migrates it to the AxcutDocument the native Rust compositor consumes, and +// drives exportMultiNative/exportGifNative — mirroring the v4 ExportDialog so +// CLI exports and GUI exports stay pixel-identical. The hidden window does no +// compositing itself: the render runs in the main process; this runner only +// builds the clip list + scene JSON and relays progress. + +import { useEffect, useRef, useState } from "react"; +import { + normalizeProjectEditor, + resolveProjectMedia, + toFileUrl, + validateProjectData, +} from "@/components/video-editor/projectPersistence"; +import type { CursorTelemetryPoint } from "@/components/video-editor/types"; +import { migrateProjectDataToAxcutDocument } from "@/lib/ai-edition/document/migrate"; +import { + collectEffectiveClipDims, + type Dims, + pickExtremeDims, + resolveAspectRatioValue, +} from "@/lib/ai-edition/document/outputFormat"; +import { applyProbedDuration } from "@/lib/ai-edition/document/timeline"; +import type { AxcutDocument } from "@/lib/ai-edition/schema"; +import { getEditorSettings } from "@/lib/ai-edition/store/editorSettings"; +import { resolveClipSourceEndSec } from "@/lib/ai-edition/timeline/clipDuration"; +import { DEFAULT_ZOOM_DEPTH, ZOOM_DEPTH_SCALES } from "@/lib/ai-edition/timeline/zoom-scale"; +import { buildAutoZoomSuggestions } from "@/lib/ai-edition/timeline/zoom-suggestions"; +import type { CliDoneResult, CliExportRequest } from "@/lib/cliContracts"; +import { GIF_SIZE_PRESETS, type GifSizePreset } from "@/lib/exporter"; +import { calculateMp4ExportSettings } from "@/lib/exporter/mp4ExportSettings"; +import { mixVoiceoverIntoVideo } from "@/lib/exporter/voiceoverMix"; +import { exportGifNative, exportMultiNative, nativeBridgeClient } from "@/native"; +import type { CompositorClipInput } from "@/native/contracts"; +import { buildSceneDescription, resolveVisibleClips } from "@/native/sceneDescription"; +import { clampZoomFocus } from "./vendor/zoomHelpers"; + +const MP4_EXPORT_FPS = 60; + +function probeVideoDimensions( + url: string, +): Promise<{ width: number; height: number; durationMs: number }> { + return new Promise((resolve, reject) => { + const video = document.createElement("video"); + video.preload = "metadata"; + video.muted = true; + const cleanup = () => { + clearTimeout(timer); + video.removeAttribute("src"); + video.load(); + }; + // A stalled load fires neither event; without a deadline the CLI hangs. + const timer = setTimeout(() => { + cleanup(); + reject(new Error(`Timed out reading video metadata: ${url}`)); + }, 30_000); + video.onloadedmetadata = () => { + const width = video.videoWidth; + const height = video.videoHeight; + const durationMs = Number.isFinite(video.duration) ? Math.round(video.duration * 1000) : 0; + cleanup(); + resolve({ width, height, durationMs }); + }; + video.onerror = () => { + cleanup(); + reject(new Error(`Failed to load video metadata: ${url}`)); + }; + video.src = url; + }); +} + +function replaceExtension(filePath: string, newExtension: string): string { + return filePath.replace(/\.(openscreen|json)$/i, "") + newExtension; +} + +/** Mirrors ExportDialog.buildNativeClipList: trim-narrowed visible clips mapped + * onto the native multiclip contract. Kept in lock-step with + * buildSceneDescription so export and scene agree on the clip stream. */ +function buildNativeClipList(axcutDocument: AxcutDocument): CompositorClipInput[] { + const assetById = new Map(axcutDocument.assets.map((asset) => [asset.id, asset])); + return resolveVisibleClips(axcutDocument).flatMap((clip) => { + const asset = assetById.get(clip.assetId); + if (!asset?.originalPath) { + return []; + } + const cam = asset.cameraTrack; + const sourceEndSec = resolveClipSourceEndSec(clip, asset); + return [ + { + screenPath: asset.originalPath, + webcamPath: cam?.sourcePath ?? asset.originalPath, + sourceStartSec: clip.sourceStartSec, + sourceEndSec, + webcamOffsetSec: cam ? (cam.startMs + cam.offsetMs) / 1000 : 0, + hasAudio: true, + }, + ]; + }); +} + +/** Mirrors ExportDialog.gifOutputDims: cap height at the preset, keep even. */ +function gifOutputDims( + preset: GifSizePreset, + tierDims: { width: number; height: number } | null, +): { width?: number; height?: number } { + if (!tierDims) return {}; + const maxHeight = GIF_SIZE_PRESETS[preset].maxHeight; + if (!Number.isFinite(maxHeight) || tierDims.height <= maxHeight) { + return { width: tierDims.width, height: tierDims.height }; + } + const scale = maxHeight / tierDims.height; + const even = (n: number) => Math.max(2, Math.round(n * scale) & ~1); + return { width: even(tierDims.width), height: even(tierDims.height) }; +} + +function appendAutoZoomRanges( + axcutDocument: AxcutDocument, + cursorTelemetry: CursorTelemetryPoint[], + totalMs: number, +): number { + const suggestions = buildAutoZoomSuggestions({ + cursorTelemetry, + totalMs, + existingRegions: axcutDocument.zoomRanges, + defaultDurationMs: Math.max(1000, Math.round(totalMs * 0.05)), + }); + let nextId = 1; + for (const suggestion of suggestions) { + axcutDocument.zoomRanges.push({ + id: `cli-auto-zoom-${nextId++}`, + startMs: Math.round(suggestion.span.start), + endMs: Math.round(suggestion.span.end), + depth: DEFAULT_ZOOM_DEPTH, + customScale: ZOOM_DEPTH_SCALES[DEFAULT_ZOOM_DEPTH], + focus: clampZoomFocus(suggestion.focus), + focusMode: "auto", + source: "auto", + }); + } + return suggestions.length; +} + +async function runExport(request: CliExportRequest): Promise<CliDoneResult> { + const loaded = await nativeBridgeClient.project.loadProjectFileFromPath(request.projectPath); + if (!loaded.success || loaded.project === undefined) { + throw new Error(loaded.error ?? loaded.message ?? "Failed to load project file"); + } + if (!validateProjectData(loaded.project)) { + throw new Error("Project file is not a valid .openscreen project"); + } + const project = loaded.project; + const media = resolveProjectMedia(project); + if (!media) { + throw new Error("Project file does not reference any recorded media"); + } + // Prefer the main process's approved session paths: they carry the + // packed-project sibling fallback when the stored absolute paths are stale. + try { + const sessionResult = await window.electronAPI.getCurrentRecordingSession(); + const session = sessionResult?.session; + if (session?.screenVideoPath) { + media.screenVideoPath = session.screenVideoPath; + if (media.webcamVideoPath && session.webcamVideoPath) { + media.webcamVideoPath = session.webcamVideoPath; + } + } + } catch { + // Fall back to the paths stored in the project file. + } + const editor = normalizeProjectEditor(project.editor ?? {}); + + const format = request.format ?? editor.exportFormat; + if (request.audioPath && format === "gif") { + throw new Error( + "--audio is only supported for MP4 exports (this project's stored format is gif; pass --format mp4)", + ); + } + const quality = request.quality ?? editor.exportQuality; + const gifFrameRate = request.gifFrameRate ?? editor.gifFrameRate; + const gifSizePreset = request.gifSizePreset ?? editor.gifSizePreset; + const outPath = + request.outPath ?? replaceExtension(request.projectPath, format === "gif" ? ".gif" : ".mp4"); + // Cursor telemetry: only needed to compute --auto-zoom suggestions. The + // native compositor discovers the `<video>.cursor.json` sidecar itself. + let cursorTelemetry: CursorTelemetryPoint[] = []; + if (request.autoZoom) { + try { + cursorTelemetry = await nativeBridgeClient.cursor.getTelemetry(media.screenVideoPath); + } catch { + cursorTelemetry = []; + } + } + + const probed = await probeVideoDimensions(toFileUrl(media.screenVideoPath)); + + // Migrate the .openscreen project onto the AxcutDocument the native + // compositor consumes. The migration is pure and carries zooms, annotations, + // trims and the legacy editor settings; the clip's duration is unknown until + // probed, so applyProbedDuration must run or the export is a single frame. + let axcutDocument = migrateProjectDataToAxcutDocument({ + ...project, + media, + editor, + }); + const primaryAssetId = axcutDocument.project.primaryAssetId ?? axcutDocument.assets[0]?.id; + if (!primaryAssetId) { + throw new Error("Project migration produced no media asset"); + } + if (probed.durationMs > 0) { + axcutDocument = applyProbedDuration(axcutDocument, primaryAssetId, probed.durationMs / 1000); + } + + if (request.autoZoom) { + const added = appendAutoZoomRanges(axcutDocument, cursorTelemetry, probed.durationMs); + window.electronAPI.cliLog("info", `Auto-zoom: added ${added} region(s) from cursor telemetry`); + } + + // Output sizing mirrors the ExportDialog: crop-aware smallest clip on the + // timeline, normalized to the document's aspect ratio. + const probedAssetDims: Record<string, Dims> = { + [primaryAssetId]: { width: probed.width, height: probed.height }, + }; + const smallestSource = + pickExtremeDims(collectEffectiveClipDims(axcutDocument, probedAssetDims), "smallest") ?? + ({ width: probed.width, height: probed.height } as Dims); + const aspectRatioValue = resolveAspectRatioValue( + axcutDocument, + getEditorSettings(axcutDocument).aspectRatio, + ); + const outDims = calculateMp4ExportSettings({ + quality, + sourceWidth: smallestSource.width, + sourceHeight: smallestSource.height, + aspectRatioValue, + }); + + const clips = buildNativeClipList(axcutDocument); + if (clips.length === 0) { + throw new Error("The project's timeline has no visible clips to export"); + } + const sceneJson = JSON.stringify(buildSceneDescription(axcutDocument)); + + // Progress: native pushes raw encoded-frame counts; totals and pacing are + // computed here, mirroring the ExportDialog. + const outFps = format === "gif" ? gifFrameRate : MP4_EXPORT_FPS; + const totalFrames = Math.max( + 1, + Math.round( + clips.reduce((sum, clip) => sum + Math.max(0, clip.sourceEndSec - clip.sourceStartSec), 0) * + outFps, + ), + ); + const exportStartedAt = Date.now(); + const unsubscribeProgress = window.electronAPI.onNativeExportProgress?.((frames: number) => { + const elapsedSec = (Date.now() - exportStartedAt) / 1000; + const rate = frames > 0 ? frames / Math.max(elapsedSec, 0.001) : 0; + window.electronAPI.cliProgress({ + percentage: Math.min(100, (frames / totalFrames) * 100), + currentFrame: frames, + totalFrames, + estimatedTimeRemaining: rate > 0 ? Math.max(0, (totalFrames - frames) / rate) : 0, + }); + }); + + try { + if (format === "gif") { + const dims = gifOutputDims(gifSizePreset, outDims); + await exportGifNative(clips, outPath, sceneJson, { + ...dims, + fps: gifFrameRate, + loopCount: editor.gifLoop ? 0 : 1, + }); + return { + success: true, + outputPath: outPath, + format, + width: dims.width, + height: dims.height, + }; + } + + // MP4: native writes outPath directly. When a voiceover is requested, mix + // it afterwards (the native pipeline has no extra-audio-track concept) and + // overwrite the same file. + await exportMultiNative(clips, outPath, sceneJson, { + width: outDims.width, + height: outDims.height, + fps: MP4_EXPORT_FPS, + codec: "h264", + }); + + if (request.audioPath) { + window.electronAPI.cliProgress({ percentage: 100, phase: "mixing-voiceover" }); + const [videoResponse, audioResponse] = await Promise.all([ + fetch(toFileUrl(outPath)), + fetch(toFileUrl(request.audioPath)), + ]); + if (!videoResponse.ok) { + throw new Error(`Failed to read the exported video back for mixing: ${outPath}`); + } + if (!audioResponse.ok) { + throw new Error(`Failed to read voiceover file: ${request.audioPath}`); + } + const mixed = await mixVoiceoverIntoVideo(await videoResponse.blob(), { + voiceoverData: await audioResponse.arrayBuffer(), + mode: request.audioMode, + offsetSec: request.audioOffsetSec, + }); + const saveResult = await window.electronAPI.writeExportToPath( + await mixed.arrayBuffer(), + outPath, + ); + if (!saveResult.success) { + throw new Error(saveResult.message ?? `Failed to write mixed output to ${outPath}`); + } + } + + return { + success: true, + outputPath: outPath, + format, + width: outDims.width, + height: outDims.height, + }; + } finally { + unsubscribeProgress?.(); + } +} + +export function CliExportRunner() { + const startedRef = useRef(false); + const [status, setStatus] = useState("Starting export…"); + + useEffect(() => { + if (startedRef.current) return; + startedRef.current = true; + + void (async () => { + try { + const request = (await window.electronAPI.cliGetRequest()) as CliExportRequest; + if (request.kind !== "export") { + throw new Error(`cli-export window received a ${request.kind} request`); + } + setStatus(`Exporting ${request.projectPath}…`); + const result = await runExport(request); + await window.electronAPI.cliDone(result); + } catch (error) { + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + await window.electronAPI.cliDone({ success: false, error: message }); + } + })(); + }, []); + + return ( + <div className="flex h-screen items-center justify-center bg-[#09090b] text-white/60 text-sm"> + {status} + </div> + ); +} + +export default CliExportRunner; diff --git a/src/cli/CliRecordRunner.tsx b/src/cli/CliRecordRunner.tsx new file mode 100644 index 000000000..e19c451f9 --- /dev/null +++ b/src/cli/CliRecordRunner.tsx @@ -0,0 +1,301 @@ +// Hidden-window runner for `openscreen record`. Reuses the full recording +// pipeline via useScreenRecorder (native macOS/Windows helpers with browser +// fallback), driven by the CLI controller instead of the HUD. + +import { useEffect, useRef, useState } from "react"; +import { + normalizeProjectEditor, + PROJECT_VERSION, +} from "@/components/video-editor/projectPersistence"; +import { useScreenRecorder } from "@/hooks/useScreenRecorder"; +import type { CliRecordRequest } from "@/lib/cliContracts"; + +type Phase = "init" | "recording" | "stopping" | "done"; + +async function pickSource(request: CliRecordRequest): Promise<ProcessedDesktopSource> { + const sources = await window.electronAPI.getSources({ + types: ["screen", "window"], + thumbnailSize: { width: 32, height: 18 }, + }); + + if (request.windowTitle) { + const needle = request.windowTitle.toLowerCase(); + const match = sources.find( + (source) => source.id.startsWith("window:") && source.name.toLowerCase().includes(needle), + ); + if (!match) { + const windows = sources + .filter((s) => s.id.startsWith("window:")) + .map((s) => ` - ${s.name}`) + .join("\n"); + throw new Error( + `No window title contains "${request.windowTitle}". Open windows:\n${windows}`, + ); + } + return match; + } + + const screens = sources.filter((source) => source.id.startsWith("screen:")); + const screen = screens[request.displayIndex]; + if (!screen) { + throw new Error( + `Display index ${request.displayIndex} not found (${screens.length} screen(s) available)`, + ); + } + return screen; +} + +async function resolveMicDeviceId(deviceNameFilter: string | null): Promise<{ + deviceId: string | undefined; + deviceName: string | undefined; +}> { + if (!deviceNameFilter) return { deviceId: undefined, deviceName: undefined }; + + // Labels require an active permission grant; a short-lived stream unlocks them. + let probeStream: MediaStream | null = null; + try { + probeStream = await navigator.mediaDevices.getUserMedia({ audio: true }); + } catch { + // Enumeration below may still work with empty labels. + } + try { + const devices = await navigator.mediaDevices.enumerateDevices(); + const needle = deviceNameFilter.toLowerCase(); + const match = devices.find( + (device) => device.kind === "audioinput" && device.label.toLowerCase().includes(needle), + ); + if (!match) { + const labels = devices + .filter((d) => d.kind === "audioinput" && d.label) + .map((d) => ` - ${d.label}`) + .join("\n"); + throw new Error(`No microphone label contains "${deviceNameFilter}". Devices:\n${labels}`); + } + return { deviceId: match.deviceId, deviceName: match.label }; + } finally { + probeStream?.getTracks().forEach((track) => track.stop()); + } +} + +/** A minimal .openscreen project referencing the finished recording, with all + * editor settings at their defaults — ready for `openscreen export` or the GUI. */ +function buildDefaultProject(session: { + screenVideoPath: string; + webcamVideoPath?: string; + cursorCaptureMode?: string; +}) { + return { + version: PROJECT_VERSION, + media: { + screenVideoPath: session.screenVideoPath, + ...(session.webcamVideoPath ? { webcamVideoPath: session.webcamVideoPath } : {}), + ...(session.cursorCaptureMode ? { cursorCaptureMode: session.cursorCaptureMode } : {}), + }, + editor: normalizeProjectEditor({}), + }; +} + +export function CliRecordRunner() { + const recorder = useScreenRecorder(); + const startedRef = useRef(false); + const requestRef = useRef<CliRecordRequest | null>(null); + // Re-render trigger once the bootstrap has applied all recorder settings; + // the start effect below cannot rely on recorder state deps alone because + // default-valued settings (no mic, no system audio) never change. + const [requestReady, setRequestReady] = useState<CliRecordRequest | null>(null); + const phaseRef = useRef<Phase>("init"); + const recordingStartedAtRef = useRef<number | null>(null); + // A stop (SIGINT/stdin) can land while the capture helper is still starting; + // remember it and apply as soon as recording flips on. + const stopRequestedRef = useRef(false); + const [status, setStatus] = useState("Preparing recording…"); + + const { + recording, + saving, + startRecordingImmediately, + toggleRecording, + setMicrophoneEnabled, + setMicrophoneDeviceId, + setMicrophoneDeviceName, + setSystemAudioEnabled, + setCursorCaptureMode, + } = recorder; + + // Keep latest values in refs for the stop/finish effects. + const toggleRecordingRef = useRef(toggleRecording); + const recordingRef = useRef(recording); + useEffect(() => { + toggleRecordingRef.current = toggleRecording; + recordingRef.current = recording; + }); + + const fail = async (error: unknown) => { + phaseRef.current = "done"; + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + await window.electronAPI.cliDone({ success: false, error: message }); + }; + + // Bootstrap: pick source, configure recorder, start. + // biome-ignore lint/correctness/useExhaustiveDependencies: intentional run-once bootstrap; startedRef guards re-entry + useEffect(() => { + if (startedRef.current) return; + startedRef.current = true; + + void (async () => { + try { + // The recorder hook surfaces some failures via blocking alert(); a + // hidden window must never show (or hang on) a modal. + window.alert = (message?: unknown) => { + window.electronAPI.cliLog("error", `Recorder: ${String(message)}`); + }; + + const request = (await window.electronAPI.cliGetRequest()) as CliRecordRequest; + if (request.kind !== "record") { + throw new Error(`cli-record window received a ${request.kind} request`); + } + requestRef.current = request; + + const source = await pickSource(request); + await window.electronAPI.selectSource(source); + window.electronAPI.cliLog("info", `Recording source: ${source.name}`); + + if (request.mic) { + const mic = await resolveMicDeviceId(request.micDevice); + setMicrophoneEnabled(true); + setMicrophoneDeviceId(mic.deviceId); + setMicrophoneDeviceName(mic.deviceName); + if (mic.deviceName) { + window.electronAPI.cliLog("info", `Microphone: ${mic.deviceName}`); + } + } + setSystemAudioEnabled(request.systemAudio); + setCursorCaptureMode(request.cursorMode); + setStatus("Starting recording…"); + setRequestReady(request); + } catch (error) { + await fail(error); + } + })(); + }, []); + + // The setters above land on the *next* render; start only once they have. + const configuredRef = useRef(false); + // biome-ignore lint/correctness/useExhaustiveDependencies: fires when recorder settings match the request; other referenced values are stable refs/callbacks + useEffect(() => { + const request = requestReady; + if (!request || configuredRef.current || phaseRef.current !== "init") return; + const micReady = !request.mic || recorder.microphoneEnabled; + const systemAudioReady = recorder.systemAudioEnabled === request.systemAudio; + const cursorReady = recorder.cursorCaptureMode === request.cursorMode; + if (!micReady || !systemAudioReady || !cursorReady) return; + + configuredRef.current = true; + phaseRef.current = "recording"; + void (async () => { + try { + await startRecordingImmediately(); + // The hook reports start failures via toast/console, not by + // rejecting — without a deadline a failed start would hang the + // CLI forever. + setTimeout(() => { + if (recordingStartedAtRef.current === null && phaseRef.current === "recording") { + void fail( + new Error( + "Recording did not start within 30s — see stderr for the underlying capture error", + ), + ); + } + }, 30_000); + } catch (error) { + await fail(error); + } + })(); + }, [ + requestReady, + recorder.microphoneEnabled, + recorder.systemAudioEnabled, + recorder.cursorCaptureMode, + ]); + + // Recording state transitions: report start and arm the duration timer. + useEffect(() => { + const request = requestRef.current; + if (recording && recordingStartedAtRef.current === null) { + recordingStartedAtRef.current = Date.now(); + setStatus("Recording…"); + window.electronAPI.cliLog("info", "Recording started"); + + if (stopRequestedRef.current) { + phaseRef.current = "stopping"; + setStatus("Stopping…"); + toggleRecordingRef.current(); + return; + } + + if (request?.durationMs) { + const timer = setTimeout(() => { + if (recordingRef.current && phaseRef.current === "recording") { + phaseRef.current = "stopping"; + window.electronAPI.cliLog("info", `Duration reached (${request.durationMs}ms)`); + toggleRecordingRef.current(); + } + }, request.durationMs); + return () => clearTimeout(timer); + } + } + }, [recording]); + + // External stop (SIGINT / stdin via main process). + useEffect(() => { + return window.electronAPI.onCliStopRecording(() => { + if (recordingRef.current && phaseRef.current === "recording") { + phaseRef.current = "stopping"; + setStatus("Stopping…"); + toggleRecordingRef.current(); + } else { + // Capture is still starting; stop as soon as it comes up. + stopRequestedRef.current = true; + } + }); + }, []); + + // Completion: recording flipped off and the session finished saving. + // biome-ignore lint/correctness/useExhaustiveDependencies: completion is keyed on recording/saving; fail is stable + useEffect(() => { + if (recordingStartedAtRef.current === null) return; + if (recording || saving) return; + if (phaseRef.current === "done") return; + phaseRef.current = "done"; + + void (async () => { + try { + const sessionResult = await window.electronAPI.getCurrentRecordingSession(); + const session = sessionResult?.session; + if (!session?.screenVideoPath) { + throw new Error("Recording finished but no session manifest was stored"); + } + const durationMs = Date.now() - (recordingStartedAtRef.current ?? Date.now()); + const request = requestRef.current; + await window.electronAPI.cliDone({ + success: true, + screenVideoPath: session.screenVideoPath, + webcamVideoPath: session.webcamVideoPath, + cursorDataPath: `${session.screenVideoPath}.cursor.json`, + durationMs, + ...(request?.projectOut ? { projectData: buildDefaultProject(session) } : {}), + }); + } catch (error) { + await fail(error); + } + })(); + }, [recording, saving]); + + return ( + <div className="flex h-screen items-center justify-center bg-[#09090b] text-white/60 text-sm"> + {status} + </div> + ); +} + +export default CliRecordRunner; diff --git a/src/cli/CliSourcesRunner.tsx b/src/cli/CliSourcesRunner.tsx new file mode 100644 index 000000000..a1e464e23 --- /dev/null +++ b/src/cli/CliSourcesRunner.tsx @@ -0,0 +1,87 @@ +// Hidden-window runner for `openscreen sources`: enumerates capturable +// displays/windows (via the same get-sources IPC the GUI picker uses) and +// microphone inputs, then hands the payload to the CLI controller to print. + +import { useEffect, useRef, useState } from "react"; +import type { CliSourcesResult } from "@/lib/cliContracts"; + +async function enumerateMicrophones(): Promise<{ + microphones: { label: string }[]; + microphoneLabelsUnavailable: boolean; +}> { + const listInputs = async () => + (await navigator.mediaDevices.enumerateDevices()).filter( + (device) => device.kind === "audioinput", + ); + + let inputs = await listInputs(); + + // Labels are blank until a getUserMedia grant exists; a short-lived probe + // stream unlocks them without leaving anything recording. + if (inputs.length > 0 && inputs.every((device) => !device.label)) { + let probeStream: MediaStream | null = null; + try { + probeStream = await navigator.mediaDevices.getUserMedia({ audio: true }); + inputs = await listInputs(); + } catch { + // Permission denied — report devices without labels. + } finally { + probeStream?.getTracks().forEach((track) => track.stop()); + } + } + + const labeled = inputs.filter((device) => device.label); + return { + microphones: labeled.map((device) => ({ label: device.label })), + microphoneLabelsUnavailable: inputs.length > 0 && labeled.length === 0, + }; +} + +async function enumerateSources(): Promise<CliSourcesResult> { + const sources = await window.electronAPI.getSources({ + types: ["screen", "window"], + thumbnailSize: { width: 32, height: 18 }, + }); + + const displays = sources + .filter((source) => source.id.startsWith("screen:")) + .map((source, index) => ({ index, id: source.id, name: source.name })); + const windows = sources + .filter((source) => source.id.startsWith("window:")) + .map((source) => ({ id: source.id, name: source.name })); + + const { microphones, microphoneLabelsUnavailable } = await enumerateMicrophones(); + return { displays, windows, microphones, microphoneLabelsUnavailable }; +} + +export function CliSourcesRunner() { + const startedRef = useRef(false); + const [status] = useState("Enumerating sources…"); + + useEffect(() => { + if (startedRef.current) return; + startedRef.current = true; + + void (async () => { + try { + const request = await window.electronAPI.cliGetRequest(); + if (request.kind !== "sources") { + throw new Error(`cli-sources window received a ${request.kind} request`); + } + const sources = await enumerateSources(); + await window.electronAPI.cliDone({ success: true, sources }); + } catch (error) { + const message = error instanceof Error ? (error.stack ?? error.message) : String(error); + await window.electronAPI.cliDone({ success: false, error: message }); + } + })(); + }, []); + + return ( + <div className="flex h-screen items-center justify-center bg-[#09090b] text-white/60 text-sm"> + {status} + </div> + ); +} + +export default CliSourcesRunner; diff --git a/src/cli/captionAnnotations.test.ts b/src/cli/captionAnnotations.test.ts new file mode 100644 index 000000000..34cee80d9 --- /dev/null +++ b/src/cli/captionAnnotations.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from "vitest"; +import type { CaptionSegment } from "@/lib/captioning/transcribe"; +import { captionSegmentsToAnnotationRegions } from "./captionAnnotations"; + +const words = (...texts: string[]): CaptionSegment[] => + texts.map((text, i) => ({ text, startSec: i * 0.5, endSec: i * 0.5 + 0.4 })); + +describe("captionSegmentsToAnnotationRegions", () => { + it("numbers ids and z-indexes from the given start, with non-empty spans", () => { + const regions = captionSegmentsToAnnotationRegions(words("one", "two", "three", "four"), 7, 3); + + expect(regions.length).toBeGreaterThan(0); + expect(regions.map((r) => r.id)).toEqual(regions.map((_, i) => `annotation-${7 + i}`)); + expect(regions.map((r) => r.zIndex)).toEqual(regions.map((_, i) => 3 + i)); + for (const region of regions) { + expect(region.endMs).toBeGreaterThan(region.startMs); + expect(region.annotationSource).toBe("auto-caption"); + expect(region.content.trim()).not.toBe(""); + } + }); + + it("groups phrase-granularity segments one line at a time", () => { + const regions = captionSegmentsToAnnotationRegions( + [ + { text: "hello there", startSec: 0, endSec: 1 }, + { text: "second line", startSec: 1.2, endSec: 2 }, + ], + 1, + 1, + { timestampGranularity: "phrase" }, + ); + + expect(regions.map((r) => r.content)).toEqual(["hello there", "second line"]); + }); +}); diff --git a/src/cli/captionAnnotations.ts b/src/cli/captionAnnotations.ts new file mode 100644 index 000000000..890f54b2b --- /dev/null +++ b/src/cli/captionAnnotations.ts @@ -0,0 +1,76 @@ +// CLI-only: the v4 editor renders captions natively, but `openscreen captions` +// still writes caption *annotations* into .openscreen projects. Only the +// annotation conversion lives here — the segment grouping/dedupe helpers come +// from the live captioning module so the CLI can't drift from the editor. + +import type { AnnotationRegion, AnnotationTextStyle } from "@/components/video-editor/types"; +import { + type CaptionSegmentLayoutOptions, + dedupeAdjacentCaptionRepeats, + finalizeCaptionSegmentsForPlayback, + groupPhraseCaptionSegmentsIntoLines, + groupTimedCaptionWordsIntoLines, +} from "@/lib/captioning/annotationsFromCaptions"; +import type { CaptionSegment } from "@/lib/captioning/transcribe"; + +/** Wide lower-third bar; `position.x` is top-left as % of container, so center with (100 - width) / 2. */ +const CAPTION_WIDTH = 92; +const CAPTION_HEIGHT = 12; +const CAPTION_BOTTOM_MARGIN = 2; + +const CAPTION_POSITION = { + x: (100 - CAPTION_WIDTH) / 2, + y: 100 - CAPTION_HEIGHT - CAPTION_BOTTOM_MARGIN, +}; + +const CAPTION_SIZE = { width: CAPTION_WIDTH, height: CAPTION_HEIGHT }; + +const CAPTION_STYLE: AnnotationTextStyle = { + color: "#ffffff", + backgroundColor: "rgba(255, 255, 255, 0)", + fontSize: 24, + fontFamily: "Inter", + fontWeight: "normal", + fontStyle: "normal", + textDecoration: "none", + textAlign: "center", +}; + +export function captionSegmentsToAnnotationRegions( + segments: CaptionSegment[], + startNumericId: number, + startZIndex: number, + layout?: CaptionSegmentLayoutOptions, +): AnnotationRegion[] { + // Don't echo-collapse raw word tokens before grouping: repeated words ("I … I") share a + // normalized key and would merge spans while keeping only the first token's text. + const minW = layout?.minWordsPerCaption ?? 2; + const maxW = layout?.maxWordsPerCaption ?? 7; + const granularity = layout?.timestampGranularity ?? "word"; + + const grouped = + granularity === "phrase" + ? groupPhraseCaptionSegmentsIntoLines(segments, minW, maxW) + : groupTimedCaptionWordsIntoLines(segments, minW, maxW); + + const finalized = finalizeCaptionSegmentsForPlayback(dedupeAdjacentCaptionRepeats(grouped)); + + let nid = startNumericId; + let z = startZIndex; + return finalized.map((seg) => { + const startMs = Math.round(seg.startSec * 1000); + const endMs = Math.max(Math.round(seg.endSec * 1000), startMs + 1); + return { + id: `annotation-${nid++}`, + startMs, + endMs, + type: "text", + content: seg.text, + annotationSource: "auto-caption", + position: { ...CAPTION_POSITION }, + size: { ...CAPTION_SIZE }, + style: { ...CAPTION_STYLE }, + zIndex: z++, + }; + }); +} diff --git a/src/cli/vendor/leadingSilence.ts b/src/cli/vendor/leadingSilence.ts new file mode 100644 index 000000000..1f6b1b846 --- /dev/null +++ b/src/cli/vendor/leadingSilence.ts @@ -0,0 +1,80 @@ +// Vendored for the CLI: @/lib/captioning/leadingSilence was deleted in the 1.8 line. The CLI's captions command still trims leading silence before transcription, so the module lives on here verbatim. + +/** Caption path is always mono 16 kHz after `extractMono16kFromVideoUrl`. */ +import type { TrimRegion } from "@/components/video-editor/types"; + +const SAMPLE_RATE = 16_000; + +/** Window length for peak detection (~50 ms). */ +const WINDOW_SAMPLES = 800; + +/** Coarse hop so long intros scan quickly (~50 ms steps). */ +const HOP_SAMPLES = 800; + +/** Max |sample| in a window below this counts as silence (float PCM ~[-1, 1]). */ +const PEAK_THRESHOLD = 0.012; + +/** Keep a little audio before the first peak so word onsets are not clipped. */ +const PRE_ROLL_SEC = 0.12; + +/** Do not scan more than this much audio for leading silence (performance + pathological files). */ +const MAX_LEADING_SCAN_SEC = 15 * 60; + +/** + * Drops quiet audio at the beginning so Whisper is not fed a long silent prefix (which can skew + * the first phrase and wastes work). Returned `trimSec` must be added back to every segment time. + */ +export function trimLeadingSilenceMono16k(samples: Float32Array): { + samples: Float32Array; + trimSec: number; +} { + if (samples.length < WINDOW_SAMPLES) { + return { samples, trimSec: 0 }; + } + + const maxIndex = Math.min( + samples.length - WINDOW_SAMPLES, + Math.floor(MAX_LEADING_SCAN_SEC * SAMPLE_RATE), + ); + + let firstSpeechSample = -1; + for (let i = 0; i <= maxIndex; i += HOP_SAMPLES) { + let peak = 0; + for (let j = 0; j < WINDOW_SAMPLES; j++) { + peak = Math.max(peak, Math.abs(samples[i + j]!)); + } + if (peak > PEAK_THRESHOLD) { + firstSpeechSample = i; + break; + } + } + + if (firstSpeechSample <= 0) { + return { samples, trimSec: 0 }; + } + + const preRollSamples = Math.round(PRE_ROLL_SEC * SAMPLE_RATE); + const start = Math.max(0, firstSpeechSample - preRollSamples); + return { + samples: samples.subarray(start), + trimSec: start / SAMPLE_RATE, + }; +} + +/** + * When audio is trimmed from the front, Whisper times are relative to the shortened buffer. + * Shift trim regions by the same offset so `segmentOverlapsTrim` still uses consistent coordinates. + */ +export function shiftTrimRegionsMsForCaptionBuffer( + regions: TrimRegion[], + trimMs: number, +): TrimRegion[] { + if (trimMs <= 0) return regions; + return regions + .map((r) => ({ + ...r, + startMs: Math.max(0, r.startMs - trimMs), + endMs: Math.max(0, r.endMs - trimMs), + })) + .filter((r) => r.endMs > r.startMs); +} diff --git a/src/cli/vendor/zoomHelpers.ts b/src/cli/vendor/zoomHelpers.ts new file mode 100644 index 000000000..e2fc32a4a --- /dev/null +++ b/src/cli/vendor/zoomHelpers.ts @@ -0,0 +1,17 @@ +// Vendored for the CLI: clampFocusToDepth was deleted from +// @/components/video-editor/types in the 1.8 line with no successor. It is a +// tiny pure predicate the CLI export runner still needs. + +export interface ZoomFocusPoint { + cx: number; + cy: number; +} + +function clamp(value: number, min: number, max: number): number { + if (Number.isNaN(value)) return (min + max) / 2; + return Math.min(max, Math.max(min, value)); +} + +export function clampZoomFocus(focus: ZoomFocusPoint): ZoomFocusPoint { + return { cx: clamp(focus.cx, 0, 1), cy: clamp(focus.cy, 0, 1) }; +} diff --git a/src/hooks/useScreenRecorder.ts b/src/hooks/useScreenRecorder.ts index 19afd52f5..1da548b14 100644 --- a/src/hooks/useScreenRecorder.ts +++ b/src/hooks/useScreenRecorder.ts @@ -55,6 +55,8 @@ type UseScreenRecorderReturn = { saving: boolean; elapsedSeconds: number; toggleRecording: () => void; + /** Starts recording with no countdown overlay. Used by the headless CLI runner. */ + startRecordingImmediately: () => Promise<void>; togglePaused: () => void; canPauseRecording: boolean; restartRecording: () => void; @@ -2062,6 +2064,7 @@ export function useScreenRecorder(): UseScreenRecorderReturn { saving, elapsedSeconds, toggleRecording, + startRecordingImmediately: () => startRecording(), togglePaused, canPauseRecording, restartRecording, diff --git a/src/lib/cliContracts.ts b/src/lib/cliContracts.ts new file mode 100644 index 000000000..186613a85 --- /dev/null +++ b/src/lib/cliContracts.ts @@ -0,0 +1,115 @@ +// Shared request/response contracts between the CLI entry in the Electron main +// process (electron/cli/) and the hidden renderer runners (src/cli/). Keep this +// file dependency-free so both build targets can import it. + +import type { ExportQuality, GifFrameRate, GifSizePreset } from "./exporter/types"; + +export type CliCursorCaptureMode = "editable-overlay" | "system"; + +export interface CliExportRequest { + kind: "export"; + /** Absolute path to the .openscreen project file. */ + projectPath: string; + /** Absolute output path; null = derive from projectPath + format. */ + outPath: string | null; + /** null = use the format stored in the project. */ + format: "mp4" | "gif" | null; + /** null = use the quality stored in the project. */ + quality: ExportQuality | null; + gifFrameRate: GifFrameRate | null; + gifSizePreset: GifSizePreset | null; + /** + * Reference preview box used to scale annotation text and border radii the + * same way the editor's on-screen preview does. The composition is fitted + * into this box, mirroring the editor layout. Defaults to 1280x720. + */ + /** + * Add automatic zoom regions derived from cursor-dwell telemetry (same + * suggestion engine as the editor's magic wand) before rendering. Existing + * zoom regions are preserved; suggestions never overlap them. + */ + autoZoom: boolean; + /** Absolute path to a voiceover audio file to mix into the export (MP4 only). */ + audioPath: string | null; + /** "mix" layers the voiceover over the recording's audio; "replace" drops the original. */ + audioMode: "mix" | "replace"; + /** Delay before the voiceover starts, in seconds. */ + audioOffsetSec: number; +} + +export interface CliRecordRequest { + kind: "record"; + /** Index into the available screen sources (0 = primary display). */ + displayIndex: number; + /** Case-insensitive substring match against window titles; overrides displayIndex. */ + windowTitle: string | null; + mic: boolean; + /** Microphone device label substring; null = system default. */ + micDevice: string | null; + systemAudio: boolean; + cursorMode: CliCursorCaptureMode; + /** Auto-stop after this many milliseconds; null = stop via signal/stdin. */ + durationMs: number | null; + /** When set, write a ready-to-export .openscreen project here after recording. */ + projectOut: string | null; +} + +export interface CliSourcesRequest { + kind: "sources"; +} + +export interface CliCaptionsRequest { + kind: "captions"; + /** Absolute path to the .openscreen project file (updated in place). */ + projectPath: string; + minWordsPerCaption: number; + maxWordsPerCaption: number; +} + +export type CliRequest = + | CliExportRequest + | CliRecordRequest + | CliSourcesRequest + | CliCaptionsRequest; + +export interface CliSourcesResult { + displays: { index: number; id: string; name: string }[]; + windows: { id: string; name: string }[]; + microphones: { label: string }[]; + /** True when microphone labels required a permission the user hasn't granted. */ + microphoneLabelsUnavailable: boolean; +} + +export interface CliProgressEvent { + percentage: number; + currentFrame?: number; + totalFrames?: number; + estimatedTimeRemaining?: number; + phase?: string; +} + +export interface CliDoneResult { + success: boolean; + error?: string; + warnings?: string[]; + /** Export: the written output file. */ + outputPath?: string; + format?: string; + width?: number; + height?: number; + /** Record: produced artifacts. */ + screenVideoPath?: string; + webcamVideoPath?: string; + cursorDataPath?: string; + projectPath?: string; + durationMs?: number; + /** + * Record: a ready-to-save .openscreen project object built by the runner. + * The main process writes it to the --project path (renderer has no fs). + */ + projectData?: unknown; + /** Sources: enumeration payload printed by the main process. */ + sources?: CliSourcesResult; + /** Captions: number of caption annotations generated. */ + captionCount?: number; +} diff --git a/src/lib/exporter/voiceoverMix.ts b/src/lib/exporter/voiceoverMix.ts new file mode 100644 index 000000000..0c6305627 --- /dev/null +++ b/src/lib/exporter/voiceoverMix.ts @@ -0,0 +1,147 @@ +// Post-export voiceover mixing for the CLI (`openscreen export --audio`). +// +// Takes the finished MP4 blob, copies its video packets untouched (no +// re-encode), renders a new audio track with OfflineAudioContext — the +// original audio and the voiceover mixed, or the voiceover alone — and +// re-muxes both into a new MP4 via mediabunny. + +import { + ALL_FORMATS, + AudioBufferSource, + BlobSource, + BufferTarget, + EncodedPacketSink, + EncodedVideoPacketSource, + Input, + Mp4OutputFormat, + Output, +} from "mediabunny"; + +export type VoiceoverMixMode = "mix" | "replace"; + +export interface VoiceoverMixOptions { + /** Encoded audio file bytes (mp3/wav/m4a — anything decodeAudioData accepts). */ + voiceoverData: ArrayBuffer; + mode: VoiceoverMixMode; + /** Delay before the voiceover starts, in seconds. */ + offsetSec: number; + /** Gain applied to the original track in "mix" mode (0..1). */ + originalGain?: number; +} + +// Duck the original bed under the voiceover by default so the unity-gain sum +// of two loud sources doesn't hard-clip. +const DEFAULT_ORIGINAL_GAIN = 0.4; + +const OUTPUT_SAMPLE_RATE = 48_000; +const OUTPUT_CHANNELS = 2; +const VOICEOVER_AUDIO_BITRATE = 192_000; + +async function decodeToBuffer( + context: OfflineAudioContext, + data: ArrayBuffer, +): Promise<AudioBuffer> { + // decodeAudioData detaches the buffer, so hand it a copy. + return context.decodeAudioData(data.slice(0)); +} + +/** Renders the final audio track: original bed (optional) + offset voiceover. */ +async function renderMixedAudio( + videoData: ArrayBuffer | null, + durationSec: number, + options: VoiceoverMixOptions, +): Promise<AudioBuffer> { + const frameCount = Math.max(1, Math.ceil(durationSec * OUTPUT_SAMPLE_RATE)); + const context = new OfflineAudioContext(OUTPUT_CHANNELS, frameCount, OUTPUT_SAMPLE_RATE); + + const voiceover = await decodeToBuffer(context, options.voiceoverData); + const voiceoverNode = context.createBufferSource(); + voiceoverNode.buffer = voiceover; + voiceoverNode.connect(context.destination); + voiceoverNode.start(Math.max(0, options.offsetSec)); + + if (options.mode === "mix" && videoData) { + try { + const original = await decodeToBuffer(context, videoData); + const originalNode = context.createBufferSource(); + originalNode.buffer = original; + const gainNode = context.createGain(); + gainNode.gain.value = options.originalGain ?? DEFAULT_ORIGINAL_GAIN; + originalNode.connect(gainNode); + gainNode.connect(context.destination); + originalNode.start(0); + } catch { + // The exported video has no decodable audio track; the voiceover + // becomes the only audio, same as "replace". + } + } + + return context.startRendering(); +} + +/** + * Returns a new MP4 blob with the same video stream and the mixed audio track. + * The video packets are copied without re-encoding. + */ +export async function mixVoiceoverIntoVideo( + videoBlob: Blob, + options: VoiceoverMixOptions, +): Promise<Blob> { + const input = new Input({ source: new BlobSource(videoBlob), formats: ALL_FORMATS }); + try { + const videoTrack = await input.getPrimaryVideoTrack(); + if (!videoTrack) { + throw new Error("Exported file has no video track to remux"); + } + const codec = videoTrack.codec; + if (!codec) { + throw new Error("Exported file's video codec was not recognized"); + } + const decoderConfig = await videoTrack.getDecoderConfig(); + if (!decoderConfig) { + throw new Error("Exported file's video decoder config could not be read"); + } + const durationSec = await input.computeDuration(); + + // The full-file bytes are only needed to decode the original bed in + // "mix" mode; "replace" skips the copy entirely. + const videoData = options.mode === "mix" ? await videoBlob.arrayBuffer() : null; + const mixedAudio = await renderMixedAudio(videoData, durationSec, options); + + const target = new BufferTarget(); + const output = new Output({ + format: new Mp4OutputFormat({ fastStart: "in-memory" }), + target, + }); + try { + const videoSource = new EncodedVideoPacketSource(codec); + output.addVideoTrack(videoSource); + const audioSource = new AudioBufferSource({ + codec: "aac", + bitrate: VOICEOVER_AUDIO_BITRATE, + }); + output.addAudioTrack(audioSource); + await output.start(); + + const sink = new EncodedPacketSink(videoTrack); + let isFirstPacket = true; + for await (const packet of sink.packets()) { + await videoSource.add(packet, isFirstPacket ? { decoderConfig } : undefined); + isFirstPacket = false; + } + await audioSource.add(mixedAudio); + + await output.finalize(); + } catch (error) { + await output.cancel().catch(() => undefined); + throw error; + } + const buffer = target.buffer; + if (!buffer) { + throw new Error("Voiceover remux produced no output"); + } + return new Blob([buffer], { type: "video/mp4" }); + } finally { + input.dispose(); + } +}