Skip to content

feat(timeline): named timelines and make_short (shorts slice 1) - #73

Merged
yultyyev merged 6 commits into
feat/shortsfrom
feat/timeline-shorts
Nov 15, 2026
Merged

yultyyev merged 6 commits into
feat/shortsfrom
feat/timeline-shorts

Conversation

@yultyyev

Copy link
Copy Markdown
Collaborator

Slice 1 of 4 toward v0.5.2 (long recording → vertical shorts). Targets the integration branch feat/shorts; merging it releases nothing.

README: before · after. Look at the new "Quick start: a long recording → vertical shorts" section, the "Shorts from a long recording" row in the feature table, and the FAQ "Can it cut Shorts or Reels out of a podcast?".

What

Named timelines (src/timeline/document-store.ts)

  • A workspace holds several timelines. main stays at composition.json / composition-ops.json, so existing workspaces and older clip versions keep working. Every other timeline lives in timelines/<id>/ with the same two files.
  • Ids match ^[a-z0-9][a-z0-9-]{0,47}$ (TimelineIdSchema, MAIN_TIMELINE in schema.ts). There is no "current timeline" pointer.
  • Each timeline has its own revisioned store (lock, rev compare-and-swap) and its own undo history.
  • Every exported store function takes an optional timeline id, defaulting to main. A missing timeline other than main throws NoTimelineError. Only make_short (as the first verb of a batch) and clip timeline new --timeline create one, so a typo can't edit an empty document.
  • listTimelines() returns main first, then each timeline by id. A document that doesn't parse is listed with its error. deleteTimeline(id) refuses main.
  • Paths go through resolveInWorkspace, and the timeline folder is checked too, not only its files.

Shorts (new src/timeline/shorts.ts, make_short verb in verbs.ts)

  • planShort resolves each word range through the stored transcript (transcriptId and source-changed checks included).
  • It cuts the padded window out of the source timeline's picture clips of that file, so main's silence and retake cuts carry in. A moment main plays twice is taken once. Pieces a split left continuous are joined. A piece holding none of the range's words is dropped.
  • It builds a fresh v0 track on a canvasFor(source, '9:16') cover canvas at the source's fps, framed by x/y (through planReframe). It sets loudness to -14 LUFS (null turns it off) and adds captions made only from each piece's own range of words.
  • Pieces keep adjust and volume; fades are dropped and a speed clip is refused. Without replace, a timeline that already holds a short is refused.
  • The whole short is one edit, so one undo empties it.

CLI (src/cli-timeline.ts)

  • A global --timeline <id> on every clip timeline subcommand.
  • clip timeline short <id> <clipId> <from> <to> [...] with --source --aspect --x --y --words --no-captions --loudness --replace --dry-run. It prints the timeline, duration, pieces and warnings.
  • clip timeline list and clip timeline delete <id>.

MCP (src/mcp/server.ts)

  • An optional timeline input on timeline_show, timeline_check, timeline_edit, timeline_transcript, timeline_undo, timeline_redo and timeline_export. No new tool.
  • timeline_show adds timeline and timelines.
  • make_short is added to the timeline_edit description.
  • NoTimelineError and ShortError come back as { error }.

Elsewhere

  • clip ui: the verbs route answers a ShortError with 422 instead of 500. The UI itself still edits main only (slice 4).
  • Docs: README, SKILL.md (a trigger and a workflow paragraph), llms.txt, mcpb/manifest.json. src/index.ts exports the new API.

Why

Finish line (approved 2026-11-15): a long talking-head or podcast recording becomes N vertical shorts from the CLI, MCP and the UI. Each short is picked by transcript, reframed to 9:16, captioned inside the safe zone and normalized to -14 LUFS. This slice adds the document model (several timelines, decision 1–2 of the plan) and the make_short verb (decisions 3–6), which the later slices build on.

Edge padding, tuned (plan decision 6)

The plan's starting values were first word − 0.10 s and last word + 0.25 s, each clamped to the neighboring word's edge. Measured, they don't work with our transcripts, so this PR ships:

Plan Shipped
Lead 0.10 s, clamped at the previous word's end 0.25 s, clamped halfway into the previous word, and at 0
Tail 0.25 s, clamped at the next word's start none: a range ends where its last word ends

How it was measured:

  • Recording. 30 sentences from macOS say (Samantha and Daniel alternating, pauses of 0.15–0.6 s, a -63 dBFS noise floor), 83 s. Each sentence's true speech edges come from its own samples.
  • Transcript. The editor's own clip timeline transcribe, with whisper.cpp 1.9.5 and ggml-base.en.bin.
  • Cross-check. Real speech: jfk.wav, bundled with Homebrew's whisper.cpp, with pauses found from its energy.

Findings:

  • whisper.cpp words abut. With -ml 1 -sow, 234 of 234 word pairs share an edge: each pause goes to the word before it. So the edge clamp cancelled the padding entirely; every lead and tail value gave identical cuts.
  • Sentence boundaries land late. In all 29 pauses the boundary sits 0.15–0.74 s after the true end of speech. In 18 of 29 it is past the pause, inside the next sentence's first syllable, by up to 0.345 s. jfk.wav agrees: next words start 0–0.13 s late, and the final word runs 0.64 s past the speech.
  • Ends. With no tail, no sentence end was clipped (0 of 30). A 0.25 s tail made 29 of 29 ends run into the next sentence, by up to 0.595 s.
  • Starts. A 0.10 s lead clipped 7 of 29 sentence starts (worst 0.245 s). 0.25 s clips 3 of 29 (worst 0.095 s, the first fricative's onset). It lets in at most 0.07 s of the previous word's release, with the midpoint clamp; the clamp at the previous word's start would let in 0.105 s.
  • Not solved by padding. 18 of 29 short ends still include up to 0.345 s of the next sentence, because that is where whisper puts the boundary. Capping the last word's length didn't help, because the whole word grid runs late there. The fix is to snap edges to the pause in the audio: docs/backlog/short-edges-snap-to-pause.md (a candidate for slice 2).

How to test

clip timeline new
clip timeline add-media podcast.mp4                 # prints the clip id
clip timeline transcribe <clipId>
clip timeline silence <clipId>
clip timeline short best-1 <clipId> 120 160 --x 0.4
clip timeline --timeline best-1 frame 3 check.jpg
clip timeline --timeline best-1 export best-1.mp4   # 9:16, -14 LUFS, captions
clip timeline list
clip timeline --timeline best-1 undo                # empties the short; main untouched

Checks

  • pnpm lint: clean (7 pre-existing warnings in tests/speed.test.ts and tests/transition.test.ts).
  • pnpm type-check: clean.
  • pnpm test: 890 passed, 28 skipped.
  • pnpm build: OK.
  • New unit tests:
    • tests/timeline-shorts.test.ts (33): padding and the midpoint clamp; stopping at the file start and at a source clip's end; three silence-cut sub-clips giving three pieces; words cut out of main (including an exact cut and a cut leaving only the gap between words) refused; padding-only slivers dropped; neighboring ranges not replaying shared padding; multi-range order; two recordings; a repeated moment taken once; split joins; effects kept and dropped; speed refused; the exact op list (canvas, clip, framing, setLoudness, captions); captions only from each piece's own words; 720p and 1:1 canvases; replace and the non-empty refusal; the verb's lowering, refusal on main and schema bounds; the store path (one undo empties the short, main untouched, a timeline created only by a batch that starts with make_short).
    • tests/document-store.test.ts (+10): paths per timeline, malformed ids, NoTimelineError from every entry point, separate rev and undo history, per-timeline write serialization, listTimelines with a corrupt timeline and stray entries, deleteTimeline, and a timelines/ symlink out of the workspace refused, including one whose files link back inside.
    • tests/mcp-server.test.ts (+2): timeline_edit with timeline: 'clip-1' and make_short gives 1080x1920, fit: cover, -14 LUFS and exportable; main is unchanged; timeline_show lists both; one undo empties the short; a missing timeline and a short on main answer { error }; a malformed id is refused at the boundary. The tool-name pin is unchanged.
  • Integration, CI=true, real FFmpeg 6.0: the new tests/integration/timeline-shorts.test.ts passes (11 s).
    • It loops a 30 s 720p test-pattern clip with a stepped tone into a 10-minute source by stream copy.
    • It cuts words 545–547 out of main, then makes a short from two ranges (540–552 near minute 9, and 100–105).
    • The pieces skip the cut. The export probes 720x1280; ebur128 reads it within 1 LU of -14. Main keeps its rev, and one undo empties the short.
    • The whole integration suite: 28 passed, 4 skipped. On an earlier run timeline-review timed out at vitest's default 5 s and passed on rerun. It does the same on feat/shorts (docs/backlog/review-test-timeout.md).
  • Duration tolerance. The integration test allows one frame per piece plus one AAC frame, not the plan's ±1 frame. Export renders each clip as its own segment, and each comes out up to a frame long: a three-piece 16.200 s short exported as 488 frames and 16.25 s of audio. That is the compiler's behavior, not the short's: docs/backlog/segment-duration-rounding.md.
  • Codex review, 4 passes (codex review --base feat/shorts, gpt-6.1-sol).
    • Pass 1, two P2, fixed:
      • a piece that only touched the words (cut out exactly) passed, giving a padding-only short;
      • neighboring ranges replayed their shared padding.
    • Pass 2, one P1 and two P2, fixed:
      • P1: a timelines/ symlink out of the workspace, with files linking back in, passed the leaf-path check, so temp files and deleteTimeline's recursive delete would act outside. The folder is now checked too, with a test that fails without the fix.
      • captions included the word the lead padding reaches into;
      • a range whose words were all cut passed on the gap between them.
    • Pass 3, one P2, fixed: captions used the union of all ranges' words, so reordered neighboring ranges could caption another range's word in a lead. Each piece now captions its own ranges.
    • Pass 4: "No actionable regressions were identified."

Notes

  • Differences from the plan's sketch:
    • planShort takes no media map, and ShortRequest has no source; the caller passes the source document.
    • Captions use groupCaptions and defaultCaptionStyle directly instead of planCaptions, so each piece captions only its own words.
    • warnings is always empty until slice 2 adds sentence-edge warnings.
  • Mixed versions: an older clip ignores timelines/, so it sees and edits main only.
  • New backlog notes: short-edges-snap-to-pause.md, segment-duration-rounding.md, review-test-timeout.md.

yultyyev and others added 6 commits November 15, 2026 14:58
A workspace now holds several timelines: main stays at composition.json,
others live in timelines/<id>/ with their own rev, lock and undo history.
The make_short verb builds a vertical short on its own timeline from
transcript word ranges of a source timeline: padded edges, cut from the
source's clips so its silence and retake cuts carry in, a 9:16 cover
canvas, -14 LUFS and captions, as one undoable edit.

The CLI gains --timeline, `short`, `list` and `delete`; every MCP
timeline tool takes an optional timeline id and timeline_show lists them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A piece that only touches a range's words (they were cut out exactly) is
dropped, so make_short refuses instead of playing padding alone. A range
that continues the one before it in the same file starts where that one's
padding ended, so neighboring words play straight through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the long recording → shorts quick start, the `short` command, the
make_short verb in the MCP lists and an FAQ entry, and polishes the
missing-timeline and non-empty-short errors for CLI users.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
whisper.cpp words abut, each pause going to the word before it, so the
neighbor-word clamp removed all padding. Measured on a 30-sentence say
recording with base.en and on jfk.wav: a range now starts 0.25 s before
its first word (stopping halfway into the word before) and ends with its
last word, which already runs through the pause after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ords

A timeline's folder is now checked against the workspace too, not only
its files, so a timelines/ symlink out of the workspace can't receive
temp files or a recursive delete through files that link back in.
Captions on a short come only from the chosen words, not the word the
lead padding reaches into, and a range whose words were all cut is
refused even when a clip still covers the gap between them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Captions came from the union of every range's words, so with reordered
neighboring ranges one range's word could show in another's lead. Each
piece now captions only the ranges it plays, from the transcripts the
ranges were already read with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yultyyev
yultyyev merged commit a2b0895 into feat/shorts Nov 15, 2026
1 check passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.5.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant