diff --git a/README.md b/README.md index 5ac56e5..3b728d7 100644 --- a/README.md +++ b/README.md @@ -423,7 +423,7 @@ Yes. MakeMyClip Editor transcribes with whisper.cpp on your machine; install it ### Can Claude add captions or subtitles to a video? -Yes. After transcription, MakeMyClip Editor's `clip timeline captions` (or the UI's Captions button) puts captions timed to the words on a track above the video, and export burns them in. Captions come as short phrases or one word at a time, with your choice of size, color, position, and background. Non-Latin scripts use an installed font that has the glyphs. In the Premiere export, captions become `.srt` files. On Linux, the bundled FFmpeg can't draw text, so set `MAKEMYCLIP_FFMPEG_PATH` to an FFmpeg built with libfreetype and libharfbuzz. +Yes. After transcription, MakeMyClip Editor's `clip timeline captions` (or the UI's Captions button) puts captions timed to the words on a track above the video, and export burns them in. Captions come as short phrases or one word at a time, with your choice of size, color, position, and background. On a vertical canvas they sit above the bottom quarter, clear of the Shorts, Reels and TikTok buttons, and a line is never wider than fits at its font size. Non-Latin scripts use an installed font that has the glyphs. In the Premiere export, captions become `.srt` files. On Linux, the bundled FFmpeg can't draw text, so set `MAKEMYCLIP_FFMPEG_PATH` to an FFmpeg built with libfreetype and libharfbuzz. ### Can it cut Shorts or Reels out of a podcast? diff --git a/SKILL.md b/SKILL.md index 3e8389e..ef44f04 100644 --- a/SKILL.md +++ b/SKILL.md @@ -327,7 +327,7 @@ To read what's said in a clip, run `timeline transcribe `: it returns th To cut by words, use the `index` values from `timeline transcribe`: `timeline cut-words ` removes those words (inclusive) and moves later clips on the track up. Add `--transcript ` from the same `timeline transcribe` output so the cut is refused, not misplaced, if the clip was transcribed again since. The transcript also lists `retakes`, phrases said twice in a row; `timeline remove-retakes ` cuts the earlier take of each (`--dry-run` lists them first). Each is one edit that `timeline undo` reverses. A cut splits the clip, so transcribe the resulting clips again before the next cut, and run `timeline check` afterwards. -To caption speech, run `timeline captions` once the cuts are done: it turns the stored transcripts into text clips on a `captions` track above the video, timed to the words, and export burns them in. With no clip id it captions every transcribed clip and replaces the whole caption track; `timeline captions ` replaces only the captions over that clip. Captions are short phrases (up to 7 words, broken at pauses and sentence ends); `--words 1` shows one word at a time. Style with `--font --color --anchor --background ` (default white on a dark box, bottom center). Captions don't move with later cuts, so run it again after one. It's one edit that `timeline undo` reverses, and `timeline frame ` shows a captioned frame. Text outside Latin (Cyrillic, Chinese, Japanese, Korean) is drawn with an installed font that has its glyphs; if none does, export says which characters are missing — install a font (for CJK, Noto Sans CJK) or set `MAKEMYCLIP_FALLBACK_FONT` to a font file. Nothing is downloaded. +To caption speech, run `timeline captions` once the cuts are done: it turns the stored transcripts into text clips on a `captions` track above the video, timed to the words, and export burns them in. With no clip id it captions every transcribed clip and replaces the whole caption track; `timeline captions ` replaces only the captions over that clip. Captions are short phrases (up to 7 words, broken at pauses and sentence ends); `--words 1` shows one word at a time. Style with `--font --color --anchor --background ` (default white on a dark box, bottom center). On a vertical (portrait) canvas captions sit above the bottom quarter and keep 14% in from each side, clear of the Shorts, Reels and TikTok overlays; `--margin ` sets the distance from the bottom (or top) edge. A line never gets wider than fits at its font size (22 characters at the default size on 1080x1920), so a bigger `--font` makes shorter lines. Captions don't move with later cuts, so run it again after one. It's one edit that `timeline undo` reverses, and `timeline frame ` shows a captioned frame. Text outside Latin (Cyrillic, Chinese, Japanese, Korean) is drawn with an installed font that has its glyphs; if none does, export says which characters are missing — install a font (for CJK, Noto Sans CJK) or set `MAKEMYCLIP_FALLBACK_FONT` to a font file. Nothing is downloaded. To make a vertical (9:16), square (1:1) or feed (4:5) version, run `timeline reframe 9:16`. It changes the canvas shape, keeping its short side (1920x1080 becomes 1080x1920), and fills it: `--fit cover` (the default) crops what overflows, `--fit contain` shows the whole picture with bars. Look at `timeline frame ` first to see where the speaker is, then pass `--x` (0 shows the left edge of the source, 0.5 the center, 1 the right edge; `--y` likewise, top to bottom) to keep them in frame. `--x` sets every media clip on the timeline now; clips added later start centered. To frame one clip differently, run `timeline framing --x 0.3` (also `--scale` 1–4 to punch in, `--fit cover|contain|inherit`, `--reset`). Check with `timeline frame ` after; the `timeline check` contact sheet shows the source frames, not the crop. Captions keep their size, since the short side doesn't change. Each is one edit that `timeline undo` reverses. A reframed cut exports as mp4; the editor-project export refuses it. diff --git a/docs/backlog/caption-safe-zone.md b/docs/backlog/caption-safe-zone.md deleted file mode 100644 index 7a7d784..0000000 --- a/docs/backlog/caption-safe-zone.md +++ /dev/null @@ -1,3 +0,0 @@ -# Caption safe zone for vertical video - -Captions sit 40 px from the bottom edge (`EDGE` in `src/timeline/compile.ts`) whatever the canvas shape. On a 9:16 export, the Shorts, Reels and TikTok overlays (caption text, buttons) cover roughly the bottom fifth of the frame, so bottom-anchored captions land under them. Raise the default caption position on portrait canvases, or add a safe-zone offset to the caption style. diff --git a/docs/backlog/caption-width-follows-font-size.md b/docs/backlog/caption-width-follows-font-size.md deleted file mode 100644 index 6b0e133..0000000 --- a/docs/backlog/caption-width-follows-font-size.md +++ /dev/null @@ -1,3 +0,0 @@ -# Caption width follows the font size - -`groupCaptions` breaks captions at 32 Latin characters (`MAX_CAPTION_WIDTH` in `src/timeline/captions.ts`) whatever the font size, and drawtext doesn't wrap. With the default size (the canvas short side / 20) a caption fits, but a larger `--font` makes long captions run off both edges. Derive the limit from the effective font size and the canvas width. diff --git a/docs/backlog/caption-width-font-metrics.md b/docs/backlog/caption-width-font-metrics.md new file mode 100644 index 0000000..b34fc58 --- /dev/null +++ b/docs/backlog/caption-width-font-metrics.md @@ -0,0 +1,3 @@ +# Caption width from the font's own metrics + +Caption lines are fitted by `textWidthEm` in `src/timeline/captions.ts`, an estimate per character class that errs wide for Arial and DejaVu Sans (W and m at 1 em, other capitals 0.8, non-Latin letters 0.95–1.1). A font wider than those, or a glyph outside the classes, can still overflow the safe area. Read the advance widths from the font file the caption is drawn with (`fontFileFor` in `src/ffmpeg/font.ts` already parses its character map) and fit lines by them. diff --git a/llms.txt b/llms.txt index fd5e3a9..fcc3577 100644 --- a/llms.txt +++ b/llms.txt @@ -18,7 +18,7 @@ - **Transcribe locally:** `clip timeline transcribe ` returns each word with its timeline time, via whisper.cpp on the user's machine; `clip timeline transcribe --sentences` lists a long recording as numbered sentences - **Edit by transcript:** `clip timeline cut-words ` cuts a range of words - **Remove repeated takes:** `clip timeline remove-retakes ` cuts the earlier take of each phrase said twice in a row -- **Captions:** `clip timeline captions` burns in captions timed to the words (short phrases or one word at a time) +- **Captions:** `clip timeline captions` burns in captions timed to the words (short phrases or one word at a time); on a vertical canvas they stay clear of the Shorts, Reels and TikTok overlays, and their width follows the font size - **Reframe:** `clip timeline reframe 9:16` turns a landscape cut vertical (also 1:1, 4:5); `--x` picks the part that shows, `clip timeline framing ` sets one clip's zoom and position - **Loudness:** `clip timeline loudness -14` normalizes the exported mp4 to -14 LUFS (or `-16` for podcasts) in two passes - **Shorts from a long recording:** `clip timeline short best-1 ` (MCP: the `make_short` verb) turns a range of transcript words into a 9:16 short on a timeline of its own, captioned and at -14 LUFS, keeping the cuts made on the main timeline; `--timeline ` points any command at a short diff --git a/src/cli-timeline.ts b/src/cli-timeline.ts index 72633ac..0027091 100644 --- a/src/cli-timeline.ts +++ b/src/cli-timeline.ts @@ -1,5 +1,10 @@ import { appendOp } from './session/store.js'; -import { CaptionError, DEFAULT_CAPTION_WORDS, planCaptions } from './timeline/captions.js'; +import { + CaptionError, + captionSideMargin, + DEFAULT_CAPTION_WORDS, + planCaptions, +} from './timeline/captions.js'; import { invalidColorMessage, isColor } from './timeline/color.js'; import { buildFrameAtPlan, @@ -90,7 +95,7 @@ Every command works on the main timeline unless --timeline names another clip timeline transcribe [] [--sentences] [--from S] [--to S] [--model ] [--language ] clip timeline cut-words [--transcript ] clip timeline remove-retakes [--dry-run] [--transcript ] - clip timeline captions [] [--words N] [--track T] [--font N --color C --anchor A --background C|none] [--dry-run] + clip timeline captions [] [--words N] [--track T] [--font N --color C --anchor A --background C|none --margin PX] [--dry-run] clip timeline reframe <9:16|1:1|4:5|16:9|WxH> [--fit cover|contain] [--x 0..1] [--y 0..1] [--dry-run] clip timeline framing [--fit cover|contain|inherit] [--scale 0.1..4] [--x 0..1] [--y 0..1] [--reset] clip timeline loudness <-14|-16|LUFS|off> @@ -722,6 +727,15 @@ export async function runTimeline(args: string[]): Promise { style.background = flags.background === 'none' ? null : flags.background; if (flags.anchor) style.anchor = AnchorSchema.parse(flags.anchor); const comp = await readComposition(timeline); + const margin = optionalNum('margin', flags.margin); + if (margin !== undefined) { + if (!Number.isInteger(margin) || margin < 0) { + throw new CaptionError( + `--margin must be a whole number of pixels, got "${flags.margin}".`, + ); + } + style.margin = { x: captionSideMargin(comp), y: margin }; + } const media = await buildMediaMap(); const plan = await planCaptions( comp, diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 8ddc8b6..a48c7d4 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -217,7 +217,7 @@ export function buildMcpServer(): McpServer { { title: 'Edit the timeline', description: - 'Apply one or more editing verbs to the document as ONE undoable change. Verbs: add_media, add_text, add_color, trim, move, split, remove, remove_silence, cut_words, remove_retakes, caption_speech, reframe, set_framing, set_loudness, transition. remove_silence cuts the pauses out of a media clip (sensitivity 0–1, default 0.5) and moves later clips on its track up; it applies nothing when no pause is found. cut_words cuts words fromIndex..toIndex out of a clip, sending the transcriptId from the timeline_transcript call they came from; remove_retakes cuts the earlier take of each phrase said twice in a row. caption_speech turns stored transcripts into captions on a track above the video (short phrases, or one word at a time with maxWords 1), replacing captions already there; export burns them in. Captions do not move with later cuts, so caption after cutting, or run it again after a cut. reframe changes the output shape (aspect 9:16 for Shorts/Reels/TikTok, 1:1, 4:5, 16:9) and fills it (fit cover, the default, crops; contain shows bars); x/y (0 = left/top, 0.5 = center, 1 = right/bottom) pick the part of every current media clip that shows. set_framing sets the fit, zoom (scale 0.1–4) and position of one media clip. Check the result with timeline_show atSec. set_loudness sets the integrated loudness the mp4 export normalizes to (targetLufs -14 for YouTube/Shorts/Reels/TikTok, -16 for podcasts, null for off). ' + + 'Apply one or more editing verbs to the document as ONE undoable change. Verbs: add_media, add_text, add_color, trim, move, split, remove, remove_silence, cut_words, remove_retakes, caption_speech, reframe, set_framing, set_loudness, transition. remove_silence cuts the pauses out of a media clip (sensitivity 0–1, default 0.5) and moves later clips on its track up; it applies nothing when no pause is found. cut_words cuts words fromIndex..toIndex out of a clip, sending the transcriptId from the timeline_transcript call they came from; remove_retakes cuts the earlier take of each phrase said twice in a row. caption_speech turns stored transcripts into captions on a track above the video (short phrases, or one word at a time with maxWords 1), replacing captions already there; export burns them in. Captions do not move with later cuts, so caption after cutting, or run it again after a cut. On a portrait canvas they sit above the bottom quarter and stay narrow enough to keep clear of the Shorts, Reels and TikTok overlays (margin sets the distance from the edge), and their width follows the font size. reframe changes the output shape (aspect 9:16 for Shorts/Reels/TikTok, 1:1, 4:5, 16:9) and fills it (fit cover, the default, crops; contain shows bars); x/y (0 = left/top, 0.5 = center, 1 = right/bottom) pick the part of every current media clip that shows. set_framing sets the fit, zoom (scale 0.1–4) and position of one media clip. Check the result with timeline_show atSec. set_loudness sets the integrated loudness the mp4 export normalizes to (targetLufs -14 for YouTube/Shorts/Reels/TikTok, -16 for podcasts, null for off). ' + "make_short turns a long recording into a vertical short on a timeline of its own: send it with `timeline` set to a new id such as \"best-1\" (never main). It plays the given word ranges (read the source timeline's sentences with timeline_transcript and no clipId; the source defaults to main) with a little padding, cut from the source's clips so its silence and retake cuts carry in, on a 9:16 canvas filled edge to edge (x picks the part of the frame that shows), normalized to -14 LUFS, with captions. Pick whole sentences (15–60 s in all is typical); the result's warnings name a range that starts or ends mid-sentence. Look at a frame with timeline_show (same timeline, atSec), then adjust with reframe x. Undo on that timeline removes it; replace rebuilds it. Every other verb also takes `timeline`, so a short can be edited like main. " + 'Batch related verbs into a single call. Media paths must be inside the workspace. Returns the updated document summary, including whether it can now render.', inputSchema: { timeline: timelineInput, verbs: z.array(EditableVerbSchema).min(1) }, diff --git a/src/timeline/captions.ts b/src/timeline/captions.ts index 7c59c62..c74dc58 100644 --- a/src/timeline/captions.ts +++ b/src/timeline/captions.ts @@ -1,4 +1,4 @@ -import { videoLayers } from './compile.js'; +import { TEXT_EDGE_PX, videoLayers } from './compile.js'; import { type Anchor, type Clip, @@ -24,6 +24,12 @@ export const DEFAULT_CAPTION_WORDS = 7; /** Widest caption, in Latin characters (a CJK character counts as two). */ const MAX_CAPTION_WIDTH = 32; +/** The box drawtext draws behind a caption reaches this far past its text. */ +const CAPTION_BOX_PX = 12; +/** Share of a portrait canvas the vertical feeds' overlays cover: the caption, + * buttons and progress bar at the bottom, the action column at the sides. */ +const PORTRAIT_BOTTOM_SHARE = 0.25; +const PORTRAIT_SIDE_SHARE = 0.14; /** A pause this long between words starts a new caption, or a new sentence. */ export const PAUSE_SEC = 0.7; /** How long a caption stays up after its last word, unless the next one starts. */ @@ -108,8 +114,9 @@ export async function planCaptions( const removed = (track?.clips.length ?? 0) - kept.length; const style = { ...defaultCaptionStyle(comp), ...request.style }; + const maxWidth = captionWidthBudget(comp, style); const added: CompositionOp[] = sources.flatMap(({ clip, transcript }) => - groupCaptions(clipWords(transcript, clip), clipEndSec(clip), maxWords).map( + groupCaptions(clipWords(transcript, clip), clipEndSec(clip), maxWords, maxWidth).map( (cue): CompositionOp => ({ op: 'addClip', trackId, @@ -118,7 +125,7 @@ export async function planCaptions( text: cue.text, startSec: cue.startSec, durationSec: roundMs(cue.endSec - cue.startSec), - style, + style: fitCaptionStyle(cue.text, style, comp), }), }), ), @@ -196,23 +203,78 @@ async function everyTranscribedClip( return found; } -/** Captions sized to the canvas: white on a dark box, near the bottom. */ -export function defaultCaptionStyle(comp: Composition): TextStyle { +/** + * Where captions stay clear of what a platform draws over the video. On a + * portrait canvas (Shorts, Reels, TikTok) that is the bottom quarter and 14% + * at each side: none of them publishes pixel specs for organic posts and + * third-party estimates conflict, so one conservative box. Other shapes keep + * the usual 40 px edge. + */ +export function safeArea(comp: Composition): { side: number; bottom: number } { + if (comp.height <= comp.width) return { side: TEXT_EDGE_PX, bottom: TEXT_EDGE_PX }; return { + side: Math.round(PORTRAIT_SIDE_SHARE * comp.width), + bottom: Math.round(PORTRAIT_BOTTOM_SHARE * comp.height), + }; +} + +/** Captions sized to the canvas: white on a dark box, near the bottom, and on + * a portrait canvas inside its safe area. */ +export function defaultCaptionStyle(comp: Composition): TextStyle { + const style: TextStyle = { fontSize: Math.round(Math.min(comp.width, comp.height) / 20), color: 'white', background: 'black@0.6', anchor: 'bottom-center' satisfies Anchor, }; + if (comp.height <= comp.width) return style; + // The box drawtext draws behind the text reaches past it, so the text keeps + // that much further in for the box to stay inside the safe area too. + const { bottom } = safeArea(comp); + return { ...style, margin: { x: captionSideMargin(comp), y: bottom + CAPTION_BOX_PX } }; +} + +/** How far caption text keeps from the side edges: inside the safe area with + * its box on a portrait canvas, the usual 40 px otherwise. */ +export function captionSideMargin(comp: Composition): number { + return comp.height <= comp.width ? TEXT_EDGE_PX : safeArea(comp).side + CAPTION_BOX_PX; +} + +/** Pixels a caption line's text may take between the style's side margins. */ +function captionRoomPx(comp: Composition, style: TextStyle): number { + return Math.max(1, comp.width - 2 * (style.margin?.x ?? TEXT_EDGE_PX)); +} + +/** + * How wide a caption line may be, in em of its font: what fits between the + * style's side margins at its font size. drawtext doesn't wrap, so a wider + * line runs off the frame or into a platform's buttons. + */ +export function captionWidthBudget(comp: Composition, style: TextStyle): number { + return captionRoomPx(comp, style) / style.fontSize; +} + +/** The caption style a line is drawn with: a single word too wide for the + * room even on a line of its own gets a font size that fits it. */ +export function fitCaptionStyle(text: string, style: TextStyle, comp: Composition): TextStyle { + const width = textWidthEm(text); + const room = captionRoomPx(comp, style); + if (width * style.fontSize <= room) return style; + return { ...style, fontSize: Math.max(1, Math.floor(room / width)) }; } /** * Group a clip's words (timeline times) into captions: a new one starts after - * `maxWords` words, past the width limit, after a pause, or after a sentence - * ends. Each stays up a moment after its last word, but never into the next - * one or past `endSec`. + * `maxWords` words, past 32 characters or `maxWidthEm` em of estimated width, + * after a pause, or after a sentence ends. Each stays up a moment after its + * last word, but never into the next one or past `endSec`. */ -export function groupCaptions(words: ClipWord[], endSec: number, maxWords: number): CaptionCue[] { +export function groupCaptions( + words: ClipWord[], + endSec: number, + maxWords: number, + maxWidthEm: number = Number.POSITIVE_INFINITY, +): CaptionCue[] { const groups: ClipWord[][] = []; let current: ClipWord[] = []; for (const word of words) { @@ -222,7 +284,7 @@ export function groupCaptions(words: ClipWord[], endSec: number, maxWords: numbe (current.length >= maxWords || word.startSec - prev.endSec > PAUSE_SEC || endsSentence(prev.text) || - displayWidth(joinWords([...current, word].map((w) => w.text))) > MAX_CAPTION_WIDTH) + tooWide(joinWords([...current, word].map((w) => w.text)), maxWidthEm)) ) { groups.push(current); current = []; @@ -285,6 +347,38 @@ function displayWidth(text: string): number { return width; } +function tooWide(text: string, maxWidthEm: number): boolean { + return displayWidth(text) > MAX_CAPTION_WIDTH || textWidthEm(text) > maxWidthEm; +} + +const WIDEST = /[WMmw@%#&ÆŒæœ]/u; +const NARROW = /[ijlItfr.,:;'’!|()[\]-]/u; +/** ASCII math signs: DejaVu draws them at about 0.84 em. */ +const SIGNS = /[+=<>~^]/u; +const ASCII = /^[\x20-\x7e]$/; + +/** + * A line's width in em, estimated from the characters' advances in common + * sans-serif caption fonts: for each class at least what Arial and DejaVu Sans + * draw (W and m are about 0.95–1.0 em), so the estimate errs wide and a line it + * lets through fits on screen. Accented Latin counts as its base letter; other + * scripts count as their widest letters (DejaVu's Щ and Ю are about 1.1 em). + */ +export function textWidthEm(text: string): number { + let width = 0; + for (const ch of text) { + const base = ch.normalize('NFD').charAt(0); + if (base === ' ') width += 0.32; + else if (WIDE.test(ch) || WIDEST.test(base)) width += 1; + else if (!ASCII.test(base)) width += base === base.toLowerCase() ? 0.95 : 1.1; + else if (SIGNS.test(base)) width += 0.85; + else if (NARROW.test(base)) width += 0.4; + else if (base !== base.toLowerCase()) width += 0.8; + else width += 0.64; + } + return width; +} + function roundMs(sec: number): number { return Math.round(sec * 1000) / 1000; } diff --git a/src/timeline/compile.ts b/src/timeline/compile.ts index 66d5be1..5db459e 100644 --- a/src/timeline/compile.ts +++ b/src/timeline/compile.ts @@ -608,8 +608,6 @@ function even(size: number): number { return Math.max(2, Math.round(size / 2) * 2); } -const CENTER_POS = { x: '(w-text_w)/2', y: '(h-text_h)/2' }; - /** Seconds for a filter expression: millisecond precision, no float noise. */ function seconds(sec: number): string { return String(Math.round(sec * 1000) / 1000); @@ -643,7 +641,7 @@ function drawtextFilter( enable?: string, ): string { const { style } = clip; - const pos = ANCHOR_EXPR[style.anchor] ?? CENTER_POS; + const pos = anchorPosition(style.anchor, style.margin); const opts = [`textfile=${quoteFilterArg(textfile)}`]; if (fontFile) opts.push(`fontfile=${quoteFilterArg(fontFile)}`); opts.push( @@ -660,18 +658,27 @@ function drawtextFilter( return `drawtext=${opts.join(':')}`; } -const EDGE = 40; -const ANCHOR_EXPR: Record = { - 'top-left': { x: `${EDGE}`, y: `${EDGE}` }, - 'top-center': { x: '(w-text_w)/2', y: `${EDGE}` }, - 'top-right': { x: `w-text_w-${EDGE}`, y: `${EDGE}` }, - 'center-left': { x: `${EDGE}`, y: '(h-text_h)/2' }, - center: { x: '(w-text_w)/2', y: '(h-text_h)/2' }, - 'center-right': { x: `w-text_w-${EDGE}`, y: '(h-text_h)/2' }, - 'bottom-left': { x: `${EDGE}`, y: `h-text_h-${EDGE}` }, - 'bottom-center': { x: '(w-text_w)/2', y: `h-text_h-${EDGE}` }, - 'bottom-right': { x: `w-text_w-${EDGE}`, y: `h-text_h-${EDGE}` }, -}; +/** Inset of anchored text from the frame edges when its style sets none. */ +export const TEXT_EDGE_PX = 40; + +/** drawtext x/y for an anchor, kept `margin` pixels in from the edges it is + * anchored to. */ +function anchorPosition( + anchor: string, + margin: { x: number; y: number } = { x: TEXT_EDGE_PX, y: TEXT_EDGE_PX }, +): { x: string; y: string } { + const x = anchor.endsWith('left') + ? `${margin.x}` + : anchor.endsWith('right') + ? `w-text_w-${margin.x}` + : '(w-text_w)/2'; + const y = anchor.startsWith('top') + ? `${margin.y}` + : anchor.startsWith('bottom') + ? `h-text_h-${margin.y}` + : '(h-text_h)/2'; + return { x, y }; +} const CANVAS_BACKGROUND = 'Canvas background'; diff --git a/src/timeline/composition.ts b/src/timeline/composition.ts index 1f9f1fc..b09127a 100644 --- a/src/timeline/composition.ts +++ b/src/timeline/composition.ts @@ -117,6 +117,11 @@ export const TextStyleSchema = z.object({ /** Box color behind the text, or null for none. */ background: z.string().nullable().default(null), anchor: AnchorSchema.default('center'), + /** How far, in pixels, anchored text keeps from the frame edges: `x` from + * the left or right edge, `y` from the top or bottom. Absent means 40. */ + margin: z + .object({ x: z.number().int().nonnegative(), y: z.number().int().nonnegative() }) + .optional(), }); export type TextStyle = z.infer; diff --git a/src/timeline/interchange-fcpxml.ts b/src/timeline/interchange-fcpxml.ts index 8afe048..35e9bed 100644 --- a/src/timeline/interchange-fcpxml.ts +++ b/src/timeline/interchange-fcpxml.ts @@ -111,14 +111,19 @@ function titlePlacement( height: number, ): { position: string; alignment: 'left' | 'center' | 'right' } { const { anchor, fontSize } = title.style; + const margin = title.style.margin ?? { x: EDGE, y: EDGE }; const alignment = anchor.endsWith('left') ? 'left' : anchor.endsWith('right') ? 'right' : 'center'; const x = - alignment === 'left' ? -(width / 2 - EDGE) : alignment === 'right' ? width / 2 - EDGE : 0; - const rowOffset = height / 2 - EDGE - fontSize / 2; + alignment === 'left' + ? -(width / 2 - margin.x) + : alignment === 'right' + ? width / 2 - margin.x + : 0; + const rowOffset = height / 2 - margin.y - fontSize / 2; const y = anchor.startsWith('top') ? rowOffset : anchor.startsWith('bottom') ? -rowOffset : 0; return { position: `${num(x)} ${num(y)}`, alignment }; } diff --git a/src/timeline/shorts.ts b/src/timeline/shorts.ts index 608f8dc..7ba17bc 100644 --- a/src/timeline/shorts.ts +++ b/src/timeline/shorts.ts @@ -1,8 +1,10 @@ import { + captionWidthBudget, DEFAULT_CAPTION_TRACK, DEFAULT_CAPTION_WORDS, defaultCaptionStyle, endsSentence, + fitCaptionStyle, groupCaptions, PAUSE_SEC, } from './captions.js'; @@ -276,7 +278,9 @@ export async function planShort( // Each piece captions only its own range's words: the lead reaches into the // word before, which whisper.cpp stretches over the pause, and that word // isn't part of the short even when another range asks for it. - const style = defaultCaptionStyle(applyOps(target, ops)); + const built = applyOps(target, ops); + const style = defaultCaptionStyle(built); + const maxWidth = captionWidthBudget(built, style); const cues = placed.flatMap(({ piece, clip }) => { const words = piece.asked.flatMap((r) => wordsInside(piece.transcript.words.slice(r.fromIndex, r.toIndex + 1), r.start, r.end), @@ -285,6 +289,7 @@ export async function planShort( clipWords({ ...piece.transcript, words }, clip), clipEndSec(clip), request.maxWords ?? DEFAULT_CAPTION_WORDS, + maxWidth, ); }); if (cues.length > 0) { @@ -298,7 +303,7 @@ export async function planShort( text: cue.text, startSec: cue.startSec, durationSec: roundMs(cue.endSec - cue.startSec), - style, + style: fitCaptionStyle(cue.text, style, built), }), }); } diff --git a/src/timeline/verbs.ts b/src/timeline/verbs.ts index 6e96d54..b0ad625 100644 --- a/src/timeline/verbs.ts +++ b/src/timeline/verbs.ts @@ -1,5 +1,10 @@ import { z } from 'zod'; -import { DEFAULT_CAPTION_TRACK, DEFAULT_CAPTION_WORDS, planCaptions } from './captions.js'; +import { + captionSideMargin, + DEFAULT_CAPTION_TRACK, + DEFAULT_CAPTION_WORDS, + planCaptions, +} from './captions.js'; import { ColorSchema } from './color.js'; import type { MediaInfo } from './compile.js'; import { @@ -210,6 +215,14 @@ const EDITABLE_VERB_SCHEMAS = [ .optional() .describe('Box color behind the text (default black@0.6), or null for none.'), anchor: AnchorSchema.optional().describe('Where captions sit (default bottom-center).'), + margin: z + .number() + .int() + .nonnegative() + .optional() + .describe( + 'How far, in pixels, captions keep from the bottom (or top) edge. Default: a quarter of the height on a portrait canvas, clear of the Shorts, Reels and TikTok overlays; 40 otherwise.', + ), }), z.object({ verb: z.literal('reframe'), @@ -551,6 +564,7 @@ export async function lowerVerb( if (verb.color !== undefined) style.color = verb.color; if (verb.background !== undefined) style.background = verb.background; if (verb.anchor !== undefined) style.anchor = verb.anchor; + if (verb.margin !== undefined) style.margin = { x: captionSideMargin(comp), y: verb.margin }; const plan = await planCaptions( comp, { diff --git a/tests/compile.test.ts b/tests/compile.test.ts index 05ecaa5..133ed72 100644 --- a/tests/compile.test.ts +++ b/tests/compile.test.ts @@ -132,6 +132,39 @@ describe('compileTimeline — generated clips', () => { }); }); +describe('compileTimeline — text margins', () => { + const placed = (style: Record) => { + const comp = oneTrack({ + op: 'addClip', + trackId: 'v0', + clip: textClip({ id: 't', text: 'Hi', durationSec: 1, startSec: 0, style }), + }); + const step = compileTimeline(comp, ctx({ fontFor: () => undefined })).steps[0]; + const fc = step?.args[step.args.indexOf('-filter_complex') + 1] ?? ''; + const options = parseFilter(fc.slice(fc.indexOf('drawtext='))).options; + return { x: options.x, y: options.y }; + }; + + it('keeps text its margin in from the edges it is anchored to', () => { + const margin = { x: 151, y: 480 }; + expect(placed({ anchor: 'bottom-center', margin })).toEqual({ + x: '(w-text_w)/2', + y: 'h-text_h-480', + }); + expect(placed({ anchor: 'bottom-left', margin })).toEqual({ x: '151', y: 'h-text_h-480' }); + expect(placed({ anchor: 'top-right', margin })).toEqual({ x: 'w-text_w-151', y: '480' }); + expect(placed({ anchor: 'center', margin })).toEqual({ + x: '(w-text_w)/2', + y: '(h-text_h)/2', + }); + }); + + it('keeps 40 px without a margin, as before', () => { + expect(placed({ anchor: 'bottom-center' })).toEqual({ x: '(w-text_w)/2', y: 'h-text_h-40' }); + expect(placed({ anchor: 'top-left' })).toEqual({ x: '40', y: '40' }); + }); +}); + describe('compileTimeline — effects', () => { it('maps adjust → eq, volume → volume, speed → setpts/atempo, fades → fade/afade', () => { const comp = oneTrack( diff --git a/tests/integration/timeline-caption-safe-zone.test.ts b/tests/integration/timeline-caption-safe-zone.test.ts new file mode 100644 index 0000000..097d164 --- /dev/null +++ b/tests/integration/timeline-caption-safe-zone.test.ts @@ -0,0 +1,121 @@ +import { mkdtemp, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { listFilters, runFfmpeg } from '../../src/ffmpeg/run.js'; +import { + captionWidthBudget, + defaultCaptionStyle, + fitCaptionStyle, + groupCaptions, +} from '../../src/timeline/captions.js'; +import { buildFrameAtPlan } from '../../src/timeline/compile.js'; +import { emptyComposition } from '../../src/timeline/composition.js'; +import { applyOps, colorClip, textClip, videoTrack } from '../../src/timeline/ops.js'; +import { runPlan } from '../../src/timeline/run-plan.js'; + +// Captions drawn by real FFmpeg on a 9:16 frame, measured with bbox: text and +// box must stay inside the safe area (x 151–929, bottom at or above y 1440). +// Runs where FFmpeg can draw text (CI's Linux build can't). +const IS_CI = process.env.CI === 'true'; +const HAS_DRAWTEXT = IS_CI && (await listFilters()).has('drawtext'); + +let workspace: string; +beforeEach(async () => { + workspace = await mkdtemp(join(tmpdir(), 'mmc-safe-zone-')); +}); +afterEach(async () => { + await rm(workspace, { recursive: true, force: true }); +}); + +/** Bounding box of the bright pixels of an image. */ +async function brightBox( + image: string, +): Promise<{ x1: number; x2: number; y1: number; y2: number }> { + const { stderr } = await runFfmpeg([ + ...['-hide_banner', '-i', image, '-vf', 'bbox=min_val=128', '-f', 'null', '-'], + ]); + const match = /x1:(\d+)\s+x2:(\d+)\s+y1:(\d+)\s+y2:(\d+)/.exec(stderr); + if (!match) throw new Error(`no bbox in:\n${stderr.slice(-400)}`); + const [x1, x2, y1, y2] = match.slice(1).map(Number) as [number, number, number, number]; + return { x1, x2, y1, y2 }; +} + +describe.skipIf(!HAS_DRAWTEXT)('captions inside the safe area, drawn by real FFmpeg', () => { + it('keeps the widest all-caps captions on 1080x1920 inside x 151–929, above y 1440', async () => { + const canvas = emptyComposition({ width: 1080, height: 1920, background: 'black' }); + const style = defaultCaptionStyle(canvas); + const budget = captionWidthBudget(canvas, style); + + // All-caps speech grouped the way captions are: the longest lines it makes. + const text = + 'SUBSCRIBE FOR MORE VIDEOS ABOUT EDITING PODCASTS WITHOUT WASTING HOURS EVERY WEEKEND WHATEVER HAPPENS'; + const words = text.split(' ').map((t, i) => ({ + index: i, + text: t, + startSec: i * 0.4, + endSec: i * 0.4 + 0.35, + })); + const lines = groupCaptions(words, 60, 12, budget) + .map((c) => c.text) + .sort((a, b) => b.length - a.length) + .slice(0, 3); + // The widest letters, and a 96 px font: a W-heavy line and a long word. + const big = { ...style, fontSize: 96 }; + const wide = groupCaptions( + 'WOW WOW WOW WOW' + .split(' ') + .map((t, i) => ({ index: i, text: t, startSec: i, endSec: i + 0.5 })), + 60, + 12, + captionWidthBudget(canvas, big), + ).map((c) => c.text); + // The widest lowercase (m in DejaVu is 0.97 em) and Cyrillic capitals. + const grouped = (text: string) => + groupCaptions( + text.split(' ').map((t, i) => ({ index: i, text: t, startSec: i, endSec: i + 0.5 })), + 60, + 12, + budget, + ).map((c) => c.text); + const cases = [ + ...lines.map((line) => ({ line, lineStyle: style })), + ...wide.map((line) => ({ line, lineStyle: big })), + { line: 'UNBELIEVABLE', lineStyle: fitCaptionStyle('UNBELIEVABLE', big, canvas) }, + ...grouped('mmm mmmm mmm mmmm mmm mmmm').map((line) => ({ line, lineStyle: style })), + ...grouped('ШИРОКИЕ ЖЁЛТЫЕ ЩИТЫ ЮЖНЫХ ПЛОЩАДЕЙ').map((line) => ({ line, lineStyle: style })), + ]; + + for (const [i, { line, lineStyle }] of cases.entries()) { + const doc = applyOps(canvas, [ + { op: 'addTrack', track: videoTrack({ id: 'v0' }) }, + { + op: 'addClip', + trackId: 'v0', + clip: colorClip({ id: 'bg', color: 'black', durationSec: 1, startSec: 0 }), + }, + { op: 'addTrack', track: videoTrack({ id: 'captions' }) }, + { + op: 'addClip', + trackId: 'captions', + // A light box, so the box's own edges are measured, not just the text. + clip: textClip({ + id: 'c', + text: line, + durationSec: 1, + startSec: 0, + style: { ...lineStyle, background: 'white@0.6' }, + }), + }, + ]); + const frame = join(workspace, `line-${i}.png`); + await runPlan( + buildFrameAtPlan(doc, { media: new Map(), dir: workspace, output: frame }, 0.5), + ); + const box = await brightBox(frame); + expect(box.x1, line).toBeGreaterThanOrEqual(151); + expect(box.x2, line).toBeLessThanOrEqual(929); + expect(box.y2, line).toBeLessThanOrEqual(1440); + } + }, 60_000); +}); diff --git a/tests/timeline-captions.test.ts b/tests/timeline-captions.test.ts index 79dfd99..9c95436 100644 --- a/tests/timeline-captions.test.ts +++ b/tests/timeline-captions.test.ts @@ -4,10 +4,15 @@ import { join } from 'node:path'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { CaptionError, + captionWidthBudget, + defaultCaptionStyle, + fitCaptionStyle, groupCaptions, joinWords, planCaptions, + safeArea, type TranscriptLookup, + textWidthEm, } from '../src/timeline/captions.js'; import { checkExportable } from '../src/timeline/compile.js'; import { type Composition, emptyComposition, findClip } from '../src/timeline/composition.js'; @@ -27,7 +32,7 @@ import { type Transcript, TranscriptError, } from '../src/timeline/transcript.js'; -import { lowerVerb, type VerbContext } from '../src/timeline/verbs.js'; +import { EditableVerbSchema, lowerVerb, type VerbContext } from '../src/timeline/verbs.js'; const M = 'm_aaaaaaaaaaaa' as MediaId; const N = 'm_bbbbbbbbbbbb' as MediaId; @@ -329,6 +334,17 @@ describe('caption_speech verb', () => { defaultTrack: 'v0', }); + it('takes a margin from the edge, keeping the safe sides', async () => { + const vertical = applyOps(cutRecording(), [{ op: 'setCanvas', width: 1080, height: 1920 }]); + const ops = await lowerVerb( + vertical, + { verb: 'caption_speech', clipId: 'a', margin: 300 }, + verbCtx(), + ); + expect(ops[1]).toMatchObject({ clip: { style: { margin: { x: 163, y: 300 } } } }); + expect(() => EditableVerbSchema.parse({ verb: 'caption_speech', margin: -1 })).toThrow(); + }); + it('lowers style options onto the caption clips', async () => { const ops = await lowerVerb( cutRecording(), @@ -376,3 +392,96 @@ describe('caption_speech verb', () => { }); }); }); + +describe('captions on a vertical canvas', () => { + const vertical = emptyComposition({ width: 1080, height: 1920 }); + const landscape = emptyComposition({ width: 1920, height: 1080 }); + + it('keep clear of the bottom quarter and 14% at each side on a portrait canvas', () => { + expect(safeArea(vertical)).toEqual({ side: 151, bottom: 480 }); + expect(safeArea(emptyComposition({ width: 720, height: 1280 }))).toEqual({ + side: 101, + bottom: 320, + }); + expect(safeArea(landscape)).toEqual({ side: 40, bottom: 40 }); + expect(safeArea(emptyComposition({ width: 1080, height: 1080 }))).toEqual({ + side: 40, + bottom: 40, + }); + }); + + it('get that margin by default there, box included, and none on landscape (unchanged documents)', () => { + expect(defaultCaptionStyle(vertical)).toEqual({ + fontSize: 54, + color: 'white', + background: 'black@0.6', + anchor: 'bottom-center', + margin: { x: 163, y: 492 }, + }); + expect('margin' in defaultCaptionStyle(landscape)).toBe(false); + }); + + it('fit their width to the room between the side margins, at the font size', () => { + // Text keeps 163 px from each side (151 + the 12 px box): (1080 − 326) / 54 px + // = 13.96 em; landscape keeps 40 px: (1920 − 80) / 54 = 34.1 em. + expect(captionWidthBudget(vertical, defaultCaptionStyle(vertical))).toBeCloseTo(13.963, 3); + expect(captionWidthBudget(landscape, defaultCaptionStyle(landscape))).toBeCloseTo(34.07, 2); + // A larger font leaves room for less, on any canvas. + expect( + captionWidthBudget(vertical, { ...defaultCaptionStyle(vertical), fontSize: 96 }), + ).toBeCloseTo(7.854, 3); + }); + + it('estimate a line wide, so capitals and W or M take more room than lowercase', () => { + expect(textWidthEm('WOW WOW WOW')).toBeGreaterThan(textWidthEm('wow wow wow')); + // Arial draws W at 0.94 em and DejaVu Sans at 1.0: the estimate takes 1. + expect(textWidthEm('W')).toBe(1); + expect(textWidthEm('MOM')).toBeCloseTo(2.8, 6); + // DejaVu Sans draws m at 0.97 em, Щ and Ю at about 1.1, ж and ш under 0.95. + expect(textWidthEm('mm')).toBe(2); + expect(textWidthEm('ЩЮ')).toBeCloseTo(2.2, 6); + expect(textWidthEm('жш')).toBeCloseTo(1.9, 6); + // Math signs are about 0.84 em in DejaVu: 2+2=4 is 3 digits and 2 signs. + expect(textWidthEm('2+2=4')).toBeCloseTo(3 * 0.64 + 2 * 0.85, 6); + // Accented Latin counts as its base letter. + expect(textWidthEm('é')).toBe(textWidthEm('e')); + }); + + it('break a line before it passes the width budget, even all in capitals', () => { + const budget = captionWidthBudget(vertical, { ...defaultCaptionStyle(vertical), fontSize: 96 }); + // 11 characters, within a count of 12, but about 9.6 em of W and O at 96 px: too wide. + const wow = groupCaptions(spoken('WOW WOW WOW'), 30, 12, budget); + expect(wow.map((c) => c.text)).toEqual(['WOW WOW', 'WOW']); + const words = spoken('these words make a caption longer than twenty two'); + const narrow = groupCaptions( + words, + 30, + 12, + captionWidthBudget(vertical, defaultCaptionStyle(vertical)), + ); + expect(narrow.length).toBeGreaterThan(groupCaptions(words, 30, 12).length); + for (const cue of narrow) expect(textWidthEm(cue.text)).toBeLessThanOrEqual(13.97); + }); + + it('shrink the font of a single word too wide for the room', () => { + const big = { ...defaultCaptionStyle(vertical), fontSize: 96 }; + // 754 px of room; "UNBELIEVABLE" is about 9.4 em, 900 px at 96 px. + const fitted = fitCaptionStyle('UNBELIEVABLE', big, vertical); + expect(fitted.fontSize).toBeLessThan(96); + expect(textWidthEm('UNBELIEVABLE') * fitted.fontSize).toBeLessThanOrEqual(754); + // 13 of Cyrillic's widest capital (about 1.1 em each in DejaVu) are fitted too. + const shch = fitCaptionStyle('ЩЩЩЩЩЩЩЩЩЩЩЩЩ', defaultCaptionStyle(vertical), vertical); + expect(13 * 1.095 * shch.fontSize).toBeLessThanOrEqual(754); + // A word that fits keeps the style as it is. + expect(fitCaptionStyle('YES', big, vertical)).toBe(big); + }); + + it('are planned with the safe margin and the narrow budget on a portrait timeline', async () => { + currentDoc = applyOps(cutRecording(), [{ op: 'setCanvas', width: 1080, height: 1920 }]); + const plan = await planCaptions(currentDoc, {}, lookup(new Map([[M, RECORDING]])), counter()); + const doc = applyOps(currentDoc, plan.ops); + for (const clip of doc.tracks[1]?.clips ?? []) { + expect(clip.kind === 'text' && clip.style.margin).toEqual({ x: 163, y: 492 }); + } + }); +}); diff --git a/tests/timeline-interchange.test.ts b/tests/timeline-interchange.test.ts index 9245b72..f0a6ff3 100644 --- a/tests/timeline-interchange.test.ts +++ b/tests/timeline-interchange.test.ts @@ -270,6 +270,30 @@ describe('interchange — FCPXML', () => { expect(t1?.getAttribute('ref')).toBe(basic?.getAttribute('id')); }); + it('places a title by its margin, as the mp4 does', () => { + const root = fcpxml( + doc([ + { id: 'v0', clips: [media('a', 0, 20, 30)] }, + { + id: 'captions', + clips: [ + textClip({ + id: 't1', + text: 'Safe', + startSec: 1, + durationSec: 2, + style: { anchor: 'bottom-center', fontSize: 54, margin: { x: 151, y: 480 } }, + }), + ], + }, + ]), + ); + const title = one(root, 'title'); + const position = all(title, 'param').find((p) => p.getAttribute('name') === 'Position'); + // 1080/2 − 480 − 54/2 = 33 px below center. + expect(position?.getAttribute('value')).toBe('0 -33'); + }); + it('keeps one title across a transition, in output time', () => { const root = fcpxml( doc([ diff --git a/tests/timeline-shorts.test.ts b/tests/timeline-shorts.test.ts index b8a2656..f69b371 100644 --- a/tests/timeline-shorts.test.ts +++ b/tests/timeline-shorts.test.ts @@ -366,7 +366,14 @@ describe('planShort: the edit', () => { text: 'w2 w3 w4', startSec: 0.25, durationSec: 2.5, - style: { fontSize: 54, color: 'white', background: 'black@0.6', anchor: 'bottom-center' }, + style: { + fontSize: 54, + color: 'white', + background: 'black@0.6', + anchor: 'bottom-center', + // Clear of the vertical feeds' overlays: the bottom quarter, 14% a side. + margin: { x: 163, y: 492 }, + }, }), }, ]);