Skip to content

feat(timeline): long recordings — sentence outline, edge warnings, pause-snapped edges, fast frames (shorts slice 2) - #78

Merged
yultyyev merged 15 commits into
feat/shortsfrom
feat/long-recordings
Nov 16, 2026
Merged

yultyyev merged 15 commits into
feat/shortsfrom
feat/long-recordings

Conversation

@yultyyev

@yultyyev yultyyev commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Slice 2 of 4 toward v0.5.2 (long recording → vertical shorts). Targets the integration branch feat/shorts, after #73 (slice 1); merging it releases nothing. feat/shorts gained #72 (color validation) while this was open; it is merged in, not rebased.

Docs: README before · after (one new line in the shorts quick start); SKILL.md before · after (the transcribe and shorts paragraphs under "Build a multi-clip composition").

What

Sentence outline (src/timeline/transcript.ts)

  • outlineSentences(words, played?) groups words into sentences. A sentence ends at sentence-ending punctuation, before a pause over 0.7 s, or after 40 words. It also ends where the timeline goes back in the source, or skips ahead over words it plays elsewhere.
  • timelineOutline(comp, media, opts) walks the picture track's media clips in timeline order and transcribes each source once (or reads its stored transcript).
    • A sentence runs on across silence cuts within one source, and a word a cut splits is listed once.
    • A word that a clip plays again stays two words.
    • Each sentence carries startSec/endSec (timeline time), clipId, fromIndex..toIndex and text, plus mediaId when the timeline has more than one source.
  • Pages hold at most 40 KB of compact JSON, with nextFromSec. A page never splits sentences that start at the same time, so the cursor always moves on. fromSec/toSec select a stretch, and clipId one clip.
  • endsSentence and PAUSE_SEC are now exported from captions.ts instead of copied.

Transcript views

  • MCP timeline_transcript: clipId is optional. view: 'words' | 'sentences' defaults to words with a clip and sentences without one. fromSec/toSec are new. The sentence view is returned as compact JSON, the form the page size is counted in.
  • CLI: clip timeline transcribe [<clipId>] [--sentences] [--from S] [--to S] prints one line per sentence, with no page limit:
    transcript m_ab12…: 56c5be80e9c5
    12:34  #1400–1413  clip_ab12  text…
    
    Piping it to head ends the listing cleanly.

Sentence-edge warnings

  • planShort warns when a range starts right after, or ends right before, a word of the same sentence (no sentence end and no real pause between). The warning names the words on either side; the short is still made.
  • They reach MCP through a warn hook on VerbContext: applyVerbs collects them, and timeline_edit returns warnings.

Frame preview from a short window (src/timeline/compile.ts)

  • The video chain is factored out of buildSegmentStep into clipChains, unchanged byte for byte (the 90 exact-args compile tests pass as they were).
  • A media clip's frame is now one step:
    -noaccurate_seek -ss <in + max(0, local − 1)> -i src
    -filter_complex [0:v]setpts=PTS+round(seek/TB)-round(in/TB),trim=start=0,<segment chain>,trim=start=<first canvas frame at or after t>[v]
    -frames:v 1
    
    • The seek reads from a second early and keeps every frame from the keyframe before, so a frame export repeats is still read: a slower source, slow motion, or a screen recording holding a still.
    • The shift back is done in whole stream ticks, so timestamps match export's own seek exactly.
    • Frames before the in-point are dropped, as export's seek drops them.
    • The trim boundary is snapped up to the canvas grid and capped at the segment's last encoded frame. Encoding to the clip's length keeps round(length × fps) frames.
  • Text and color cards keep the two-step path.
  • Deletes docs/backlog/frame-preview-short-window.md.

A short's edges land in the pause the audio shows (src/timeline/pauses.ts, src/ffmpeg/args/speech-levels.ts, snapToPauses in shorts.ts)

Added after review: the user folded it in after a five-lens council on the slice 1 finding.

  • Why it was needed. whisper.cpp gives each pause to the word before it, and at a sentence end it runs up to 0.74 s late. A short cut by the transcript alone kept up to 0.35 s of the next sentence's first syllable, at 18 of 29 sentence ends.
  • How it measures. make_short now reads the source's levels around each edge: one FFmpeg read of 2.5 s, 10 ms RMS frames at 16 kHz mono. Gaps and a late start are padded with aresample=async=1:first_pts=0, the same as transcription.
  • What counts as a pause. A frame is quiet below min(floor + 10 dB, speech − 20 dB). Floor and speech are that window's own 5th and 95th percentiles, with digital silence left out of the floor. A pause is at least 80 ms; sound under 40 ms inside one is ignored.
  • Which pause. It takes the longest pause reaching into [boundary − 0.9 s, boundary + 0.05 s] that is still going after the word before the boundary has started.
  • The cut. It goes 0.1 s into the pause (half of it, for a shorter pause).
  • No clear pause. If there is none, or two pauses of 0.12 s or more compete, that edge keeps the transcript rule and warnings say so. The council split on this; the majority chose this fallback. The warning is skipped when the range was cut mid-sentence anyway, which already warns.
  • Audio that can't be read. The short is still made, by the transcript, with a warning. A source without sound follows the transcript quietly.
  • Captions. A word whisper places past the cut is pulled just inside it, in order, so it keeps its caption. The check that a piece holds speech uses those same times.
  • Cost. Two FFmpeg reads per range, about 80 ms each.

A speed clip stays on the frame grid

  • An fps filter now follows the speed setpts. Without it, retimed frames reached the encoder off the grid, and its frame dropping repeated some frames and shifted others by up to two. That made export unsteady at 2x, and a preview couldn't match it.
  • This changes exported frames of sped-up and slowed clips only.

Why

On a long recording the agent couldn't read the transcript: 60 minutes of words is about 983 KB as one result. It also couldn't look at a frame: the preview encoded the whole clip, 53.6 s for a 10-minute one. This slice adds the outline the agent picks shorts from, warnings when it cuts mid-sentence, and a frame preview whose cost doesn't grow with the clip.

How to test

clip timeline new
clip timeline add-media podcast.mp4
clip timeline transcribe <clipId>
clip timeline transcribe --sentences | head -40
clip timeline short best-1 <clipId> <from> <to> --dry-run     # see warnings
clip timeline frame 540 late.jpg                              # well under a second

Checks

  • pnpm lint: clean (the same 7 pre-existing warnings in tests/speed.test.ts and tests/transition.test.ts).

  • pnpm type-check: clean.

  • pnpm test: 937 passed, 32 skipped (after merging feat/shorts).

  • pnpm build: OK.

  • New unit tests:

    • tests/timeline-outline.test.ts (10):
      • grouping;
      • indices and times that match clipWords across three silence-cut sub-clips, with a word split by a cut listed once;
      • replayed words kept as two;
      • a sentence ending where clips moved out of source order;
      • each source read once, one without sound skipped, and a break between sources (mediaId per sentence there, none with one source);
      • paging by size and range, pages put back together giving every sentence once;
      • tied start times never repeating or stalling a page;
      • one clip, including one off the picture track;
      • a 9,000-word, 60-minute transcript paged at ≤ 40 KB: 643 sentences across pages.
    • tests/timeline-introspect.test.ts: the media frame is one step, and its graph equals setpts…,trim=start=0, + export's segment chain + trim=…. Also covered:
      • the in-point seek with the 1 s preroll;
      • the last frame;
      • a time between two frames;
      • a clip whose length is off the frame grid;
      • a second clip;
      • 2x speed.
    • tests/timeline-shorts.test.ts: warnings at a mid-sentence start and end, none for a whole sentence, the recording's edges or after a long pause.
    • tests/mcp-server.test.ts:
      • timeline_transcript without a clip returns the compact sentence outline;
      • the words view without a clip errors;
      • timeline_edit with make_short returns its warnings.
  • Integration, CI=true, bundled FFmpeg 6.0: the new tests/integration/timeline-frame-window.test.ts compares the window frame with the frame of the fully encoded clip (export's own path). The test pattern turns its hue 24° a frame, so a frame off by one scores about 21 dB.

    Case PSNR
    mid-clip, with a caption window 50.2 dB
    between two frames (3.01 s) 50.8 dB
    last frame of a 1.01 s clip 50.7 dB
    inside a fade 52.0 dB
    2x speed 50.0 dB
    24 fps source on a 30 fps canvas (1.5 s, 1.391 s, 3.999 s) 51.4, 51.0, 51.1 dB
    0.25x slow motion just past a keyframe 51.8 dB
    sparse source (a frame every 5 s) 56.4 dB
    • The very last frame of a clip that ends with its source renders.
    • A frame 9 minutes into a 10-minute clip renders in 84 ms, against 53.6 s on the old path when planning.
    • Each case was checked failing without its fix: 2x speed 35.9 dB without the fps filter; 3.01 s 21.3 dB; the 1.01 s clip 21.0 dB; 1.391 s 21.2 dB; slow motion 21.0 dB; the sparse source 19.5 dB; the end of the source gave no image.
    • The whole integration suite: 32 passed, 4 skipped.
  • Pause snapping:

    • Unit tests: tests/timeline-pauses.test.ts (11) covers:

      • the argument array and the parser;
      • pauses found by the window's own floor in a noisy room;
      • a click inside a pause;
      • digital silence before late audio;
      • no pause in steady speech or a stepped tone;
      • the pause a late boundary points at;
      • never the pause before the last word;
      • an ambiguous edge.

      tests/timeline-shorts.test.ts covers:

      • both edges moving into the pauses;
      • fallback with warnings;
      • one warning when the range is mid-sentence anyway;
      • a source without sound;
      • a measuring error;
      • a last word past the cut keeping its caption;
      • crowded last words keeping every caption at maxWords: 1;
      • a one-word range kept;
      • a range across a split captioned once.
    • CI-gated tests/integration/timeline-pauses.test.ts (real FFmpeg, AAC, no drawtext): the pause found lies inside the real one, by at most 40 ms (AAC smears a sharp ending about 30 ms). The same holds for audio that starts 1 s late. A steady tone has no pause.

    • Measured through the CLI. Every sentence of the 30-sentence say recording as its own short (whisper base.en):

      • 0 starts clipped, 0 keeping the previous sentence;
      • 0 ends clipped, 0 keeping the next one;
      • no fallbacks;
      • the tightest cut is 35 ms from speech.

      Before: 18 ends kept the next sentence (up to 0.345 s) and 3 starts were clipped (up to 0.095 s).

    • jfk.wav: every cut lands in a measured pause, 0.07–0.12 s from the speech. The whole speech ends at 10.48 s instead of whisper's 11.0 s.

  • CLI, by hand:

    • transcribe --sentences on a real whisper base.en transcript of a say recording;
    • --from 60 --to 70;
    • short --dry-run warnings;
    • a 644-line listing through | head -3 under pipefail, exit 0 with nothing on stderr (an unhandled EPIPE before the fix).
  • Codex review (gpt-6.1-sol): every finding was fixed, none rebutted.

    • Pass 1, whole diff, two P2:
      • the window seek rounded to the millisecond, past a clip's last frame (no image);
      • a word replayed by consecutive clips was merged as if a cut split it.
    • Pass 2, whole diff, three P2:
      • a source slower than the canvas lost the frame export repeats;
      • the MCP description suggested cut_words for sentences that can span clips;
      • an explicit clipId off the picture track outlined nothing.
    • Pass 3: the whole-diff review timed out twice (Codex ran the test suite itself). The commits after pass 2 were reviewed one by one instead:
      • trim=start rounded an off-grid time to the nearer frame (fixed);
      • a clip whose length is off the frame grid could preview a frame export leaves out (fixed);
      • then clean.
    • Pass 4, whole diff with the diff inlined, three P2:
      • a sparse or variable-frame-rate source lost a frame held longer than the 1 s preroll (now -noaccurate_seek);
      • sentences had no link to their transcriptId with several sources (mediaId added);
      • paging could repeat or stall on tied start times.
    • Pass 5, three P2:
      • an outline range could take in words played elsewhere (clips moved out of source order);
      • the seek's whole-tick rounding against a fractional setpts could move a frame one canvas frame;
      • the CLI lines lacked the source with several sources.
    • Pass 6, two P2:
      • slow motion needs the frame before the keyframe the seek lands on (the 1 s preroll is back, alongside -noaccurate_seek);
      • an EPIPE crash under | head.
    • The final commit was reviewed alone: "No actionable regressions were identified."

Notes

  • Export change: the fps filter after a speed setpts changes the frames of exported speed clips. They are now steady; before, some frames repeated and others were shifted up to two frames.
  • Cost: a frame now runs the chain over at most a second plus one GOP of frames, not the whole clip.
  • Pause-snapping review (Codex):
    • First pass, three P2, fixed:
      • audio that starts late or has gaps could shift a pause;
      • a pulled word could lose its predecessor's caption;
      • a one-word range could be rejected.
    • Second pass, three P2, fixed:
      • an accurate last word with a long pause before it could be cut;
      • a range across a split captioned twice;
      • words crowded at the cut could share a start.
    • The last fix commit, reviewed alone: "No [P1] or [P2] defects found." The whole-diff passes timed out with Codex running the toolchain, so these were reviewed commit by commit.
    • A fourth problem turned up in the CI-gated test: digital silence before late audio set the floor so low that room tone read as sound. Fixed.
  • Known limits. The tests use a synthetic recording with a quiet room plus 11 s of real speech. Breaths inside a pause, music beds and noisy rooms aren't measured yet; there, an edge falls back with a warning.
  • New backlog notes:
    • cut-words-pause-edges: cut_words and remove_retakes have the mirror problem.
    • retake-test-model: a whisper-gated integration test fails with base.en, on feat/shorts too.
    • short-edges-snap-to-pause is deleted.

yultyyev and others added 15 commits November 15, 2026 15:41
The frame preview encoded the clip's whole segment to grab one frame,
which took minutes on a long recording. A media clip's frame is now one
step: seek the source to the frame and run the segment's own video chain
on timestamps offset to the frame's time, so fades, captions and speed
read the same time they do in the segment. Text and color cards keep the
two-step path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After setpts, a sped-up clip's frames reached the encoder off the canvas
frame grid, and the encoder's frame dropping repeated some frames and
shifted others by up to two frames. An fps filter right after the speed
setpts picks the nearest frame for each slot, so export is steady and the
frame preview shows the frame export plays.

Adds the CI-gated check: the window frame matches the frame of the fully
encoded clip at PSNR >= 40 dB mid-clip, inside a fade and at 2x, and a
frame 9 minutes into a 10-minute clip renders in under a second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
timelineOutline lists the timeline's speech as numbered sentences, across
silence-cut sub-clips, one 40 KB page at a time with nextFromSec. MCP
timeline_transcript returns it when no clipId is given (view, fromSec and
toSec added), and `clip timeline transcribe --sentences [--from --to]`
prints one line per sentence. make_short now warns when a range starts or
ends mid-sentence, and timeline_edit passes the warnings on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The window frame rounded its seek to the millisecond, which could land
past a clip's last frame (no image) or its out-point; it now rounds down
at microsecond precision. The outline merged a word repeated by
consecutive clips as if a cut split it; it merges only when the next clip
carries on later in the source, and a word played again starts a new
sentence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source slower than the canvas shows a frame export repeats into later
slots, and seeking straight to the frame skipped it (at a clip's end, no
image at all). The window now reads from a second earlier and trims to
the first frame at the frame's segment time, timestamped from the seek
point. An outline of an explicit clip off the picture track lists its
sentences, and the MCP outline no longer suggests cut_words for sentences
that can span clips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…time

trim rounds an off-grid start to the nearest frame, so a preview between
two frames could show the earlier one where export shows the later. The
boundary is now the next canvas frame. The CI-gated test's pattern turns
its hue 24 degrees a frame, so a preview off by one frame fails it (21 dB
with the old boundary), and it adds a case between two frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Encoding a clip to its length keeps round(length x fps) frames, so on a
clip whose length is off the frame grid the next-frame boundary could
pick a frame export leaves out. The boundary now stops at the segment's
last frame, with a unit test and a CI-gated case (21 dB without it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frame preview now keeps every frame from the keyframe before its seek
(-noaccurate_seek) instead of a 1 s preroll, so a sparse source, such as
a screen recording holding a still, previews the frame export holds
(19.5 dB before, now matching). Frames before the clip's in-point are
dropped as export's seek drops them.

The outline names each sentence's source when the timeline has more than
one, so its transcriptId can be found, and a page never splits sentences
that start at the same time, so paging always moves on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frame preview restored its timestamps with a fractional offset while
the seek rounds to whole stream ticks, so a 24 fps frame on a 30 fps
canvas could land one canvas frame early (21 dB at 1.391 s). It now
undoes the seek in whole ticks, leaving the timestamps export's own seek
gives.

An outline sentence ends where the timeline skips ahead over words it
plays elsewhere (clips moved out of source order), so its index range
never takes in words it doesn't show. The CLI sentence lines name the
source when the timeline has more than one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on a closed pipe

A slow-motion frame can need the source frame before the keyframe its
own time falls just after; the preview now seeks a second early as well
as keeping the frames from the keyframe before (21 dB before at 0.25x).

`clip timeline transcribe --sentences | head` ended with an unhandled
EPIPE once head closed the pipe on a long listing; a closed pipe now
just ends it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
whisper.cpp gives each pause to the word before it and runs late at
sentence ends, so a short cut by the transcript kept up to 0.35 s of the
next sentence. make_short now measures the source's levels around each
edge (one short FFmpeg read, 10 ms RMS frames), finds the pause the
transcript boundary points at by a threshold set from those levels, and
cuts 0.1 s into it. An edge with no clear pause keeps the transcript rule
and warns; a word whisper places past the cut keeps its caption.

On the 30-sentence say recording every sentence-short is clean: no edge
clipped, none keeping a neighbor's speech (before: 18 ends kept the next
sentence, 3 starts were clipped). On jfk.wav every cut lands in a pause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rder

Levels are measured with the transcription's aresample=async=1:first_pts=0,
so a late audio start or a timestamp gap can't shift a pause; digital
silence counts as quiet but is left out of the floor estimate, or room
tone between sentences would read as sound. A word pulled inside a cut
keeps its place after the word before it, and the speech check uses the
pulled words, so a one-word range whisper places past its cut is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in order

A pause that ends by the time the last word starts is the one before it,
so an accurately timed last word with a long pause ahead of it is no
longer cut off. A range across a source split captions its words once,
and words crowded against a cut move earlier to make room instead of
sharing a start, so no caption shown a word at a time is lost.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yultyyev yultyyev changed the title feat(timeline): long recordings — sentence outline, edge warnings, fast frames (shorts slice 2) feat(timeline): long recordings — sentence outline, edge warnings, pause-snapped edges, fast frames (shorts slice 2) Nov 16, 2026
@yultyyev
yultyyev merged commit 7c6110a into feat/shorts Nov 16, 2026
1 check passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.5.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant