Skip to content

feat(timeline): captions inside the vertical feeds' safe area (shorts slice 3) - #85

Merged
yultyyev merged 7 commits into
feat/shortsfrom
feat/caption-safe-zone
Nov 16, 2026
Merged

yultyyev merged 7 commits into
feat/shortsfrom
feat/caption-safe-zone

Conversation

@yultyyev

Copy link
Copy Markdown
Collaborator

Slice 3 of 4 toward v0.5.2 (long recording → vertical shorts). Targets the integration branch feat/shorts; merging it releases nothing. feat/shorts is merged in, after #78, #83 and #84.

Docs: README before · after (one sentence in the captions FAQ); SKILL.md before · after (the "To caption speech" paragraph).

What

TextStyle.margin (src/timeline/composition.ts)

  • An optional { x, y }: how many pixels anchored text keeps from the side and top/bottom edges.
  • Absent means 40, so existing documents compile to the same FFmpeg arguments, byte for byte. The 90 exact-args compile tests pass unchanged.
  • In compile.ts, anchorPosition() replaces the fixed ANCHOR_EXPR table. FCPXML places Basic Titles by the same margin.

Safe area on portrait canvases (src/timeline/captions.ts)

  • safeArea(comp): the bottom 25% and 14% at each side on a portrait canvas, and 40/40 otherwise (decision 7).

  • defaultCaptionStyle sets margin on portrait canvases only, and includes the 12 px box drawtext draws behind the text:

    • x = side + 12 = 163 on 1080x1920;
    • y = bottom + 12 = 492.

    So the box itself stays inside x 151–929 and above y 1440. Landscape styles get no margin, so they stay as they were.

  • make_short picks this up through defaultCaptionStyle.

Caption width follows the font (decision 8, changed while building; see Notes)

  • groupCaptions breaks a line when it passes 32 characters, as before, or captionWidthBudget(comp, style): the room between the side margins in em of the font. That is 13.96 em at 54 px on 1080x1920 and 34.1 em on landscape.

  • Width comes from textWidthEm, a per-character-class estimate that errs wide for Arial and DejaVu Sans:

    Characters em
    W M m w @ % # & 1.0
    other capitals 0.8
    + = < > ~ ^ 0.85
    narrow (i l t f r . , : ; ' ! | ( ) [ ] -) 0.4
    other lowercase and digits 0.64
    space 0.32
    non-Latin letters 0.95 lowercase, 1.1 capitals
    CJK 1.0

    Accented Latin counts as its base letter.

  • fitCaptionStyle: a single word too wide for the room, even on a line of its own, gets a font size that fits it.

  • A larger --font makes shorter lines on every canvas.

Margin option

  • caption_speech gains margin (pixels from the bottom or top edge; the sides keep the safe area). The CLI gets clip timeline captions … --margin PX.
  • The MCP timeline_edit description, SKILL.md, the README FAQ and llms.txt mention the safe area.
  • Deletes docs/backlog/caption-safe-zone.md and docs/backlog/caption-width-follows-font-size.md.

Why

On a 9:16 export, bottom-anchored captions sat 40 px from the edge, under the caption text, buttons and progress bar that Shorts, Reels and TikTok draw. A 32-character line in capitals ran almost edge to edge.

How to test

clip timeline new --width 1080 --height 1920
clip timeline add-media talk.mp4
clip timeline transcribe <clipId>
clip timeline captions
clip timeline frame 3 check.jpg      # captions above the bottom quarter, inside the sides

Checks

  • pnpm lint: clean (the same 7 pre-existing warnings in tests/speed.test.ts and tests/transition.test.ts).

  • pnpm type-check: clean.

  • pnpm test: 953 passed, 33 skipped (on the merged tip).

  • pnpm build: OK.

  • Unit tests:

    • Safe area: 1080x1920 → 151/480, 720x1280 → 101/320, square and landscape → 40/40.
    • Default style: the margin {163, 492} on portrait and none on landscape.
    • Width: the budget in em; textWidthEm for W, M, m, Cyrillic, signs and accents; WOW WOW WOW at 96 px breaks; a long word and 13×Щ get fitted fonts.
    • Compile: x/y for anchors with a margin, and 40 without.
    • FCPXML: the title position by margin.
    • caption_speech: the margin and its schema bound; portrait captions planned with the margin.
  • CI-gated tests/integration/timeline-caption-safe-zone.test.ts draws each caption with real drawtext on a black 1080x1920 frame, with a light box so the box's own edges are measured. It runs where FFmpeg has drawtext, so locally with CI=true; it skips on CI's Linux FFmpeg.

    Caption Box, x Box bottom
    widest all-caps lines at 54 px 208–870 1439
    WOW WOW at 96 px 257–821 1439
    UNBELIEVABLE, fitted to 81 px 223–855 1439
    m-heavy lowercase 265–813 1439
    Cyrillic capitals 265–813 1439
    • Control, the old style: 32-character capitals drew text at x 11–1068 with its bottom at y 1879.
    • Integration suite (CI=true): 32 passed, 4 skipped. timeline-review timed out once at vitest's default 5 s, then passed twice; that's the known flake in docs/backlog/review-test-timeout.md.
  • Codex review (gpt-6.1-sol), whole slice with the diff inlined:

    • Pass 1, two P2, fixed:
      • a character count let W- and M-heavy capitals overflow (now an em estimate);
      • a long single word passed (now fitted).
    • Pass 2, two P2, fixed:
      • DejaVu's m and Cyrillic were wider than the estimate;
      • the caption's background box crossed the safe line (margins now include the box, and the test measures it).
    • Pass 3, one P2, fixed: Щ at 1.095 em (non-Latin capitals now 1.1).
    • Pass 4, on the merged branch, one P2, fixed: ASCII math signs at 0.84 em.
    • The last commit, reviewed alone: "No [P1] or [P2] findings in the supplied diff."

Notes

  • Changed from the plan:
    • margin is { x, y }, not one number. A single value would put a left- or right-anchored portrait caption 480 px from the side.
    • The margins include the 12 px box. The plan's 480/151 are where the box stops, so the text sits 12 px further in.
    • Width is an em estimate per character class, not "22 Latin characters". A count can't tell iiii from WWWW. The 32-character cap stays, so landscape grouping is unchanged for real text.
  • Known limit: the estimate covers Arial and DejaVu Sans. A wider font, or glyphs outside the classes, can still overflow. Measuring the drawn font's own advance widths is a backlog note (docs/backlog/caption-width-font-metrics.md).
  • Linux: caption fonts there (DejaVu Sans) are wider and weren't measured by drawtext here, because CI's FFmpeg has none. The estimate takes DejaVu's widths into account.
  • Mixed versions: an older clip drops style.margin from documents it rewrites.

yultyyev and others added 7 commits November 15, 2026 18:32
Text styles gain an optional margin ({ x, y } pixels from the anchored
edges; absent stays 40, so existing documents render byte for byte).
On a portrait canvas captions default to the safe area: above the bottom
quarter and 14% in from each side, clear of the Shorts, Reels and TikTok
overlays. The caption line limit now follows the font size, so a line
fits between the side margins (22 characters at 54 px on 1080x1920, 32 on
landscape as before), and a bigger font makes shorter lines everywhere.
caption_speech and `clip timeline captions` take a margin; FCPXML places
titles by it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e word

A character count let W- and M-heavy capitals overflow (WOW WOW WOW at
96 px fit 12 characters but not the 754 px of room). Lines now break by
an em-width estimate that errs wide, at least Arial's and DejaVu Sans's
advance per character class, with the 32-character cap kept. A single
word too wide for the room on a line of its own gets a font size that
fits it. Measured on 1080x1920: every case inside x 151–929, above 1440.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stimate

The box drawtext draws behind a caption reaches 12 px past its text, so
on a portrait canvas the text now keeps 163 px from the sides and 492 px
from the bottom and the box itself stays within x 151–929, above y 1440.
The width estimate counts m, w, # and & as 1 em (DejaVu draws m at 0.97)
and any non-Latin letter as 0.9–1 em; accented Latin counts as its base.
The CI-gated check now measures each caption's box, for capitals, a
96 px font, a long word, m-heavy lowercase and Cyrillic capitals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DejaVu Sans draws Щ and Ю at about 1.1 em, so a run of them could pass the
caption width check and overflow. Non-Latin capitals now count as 1.1 em
and lowercase as 0.95.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…afe-zone

# Conflicts:
#	src/mcp/server.ts
#	src/timeline/captions.ts
#	src/timeline/shorts.ts
DejaVu Sans draws + = < > ~ ^ at about 0.84 em, so a caption of them
could pass the width check and overflow; they now count as 0.85 em.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yultyyev
yultyyev merged commit 6c94b5b into feat/shorts Nov 16, 2026
1 check passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.5.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant