Repository navigation
feat(timeline): long recordings — sentence outline, edge warnings, pause-snapped edges, fast frames (shorts slice 2) - #78
Merged
Conversation
The frame preview encoded the clip's whole segment to grab one frame, which took minutes on a long recording. A media clip's frame is now one step: seek the source to the frame and run the segment's own video chain on timestamps offset to the frame's time, so fades, captions and speed read the same time they do in the segment. Text and color cards keep the two-step path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After setpts, a sped-up clip's frames reached the encoder off the canvas frame grid, and the encoder's frame dropping repeated some frames and shifted others by up to two frames. An fps filter right after the speed setpts picks the nearest frame for each slot, so export is steady and the frame preview shows the frame export plays. Adds the CI-gated check: the window frame matches the frame of the fully encoded clip at PSNR >= 40 dB mid-clip, inside a fade and at 2x, and a frame 9 minutes into a 10-minute clip renders in under a second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
timelineOutline lists the timeline's speech as numbered sentences, across silence-cut sub-clips, one 40 KB page at a time with nextFromSec. MCP timeline_transcript returns it when no clipId is given (view, fromSec and toSec added), and `clip timeline transcribe --sentences [--from --to]` prints one line per sentence. make_short now warns when a range starts or ends mid-sentence, and timeline_edit passes the warnings on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The window frame rounded its seek to the millisecond, which could land past a clip's last frame (no image) or its out-point; it now rounds down at microsecond precision. The outline merged a word repeated by consecutive clips as if a cut split it; it merges only when the next clip carries on later in the source, and a word played again starts a new sentence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source slower than the canvas shows a frame export repeats into later slots, and seeking straight to the frame skipped it (at a clip's end, no image at all). The window now reads from a second earlier and trims to the first frame at the frame's segment time, timestamped from the seek point. An outline of an explicit clip off the picture track lists its sentences, and the MCP outline no longer suggests cut_words for sentences that can span clips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…time trim rounds an off-grid start to the nearest frame, so a preview between two frames could show the earlier one where export shows the later. The boundary is now the next canvas frame. The CI-gated test's pattern turns its hue 24 degrees a frame, so a preview off by one frame fails it (21 dB with the old boundary), and it adds a case between two frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Encoding a clip to its length keeps round(length x fps) frames, so on a clip whose length is off the frame grid the next-frame boundary could pick a frame export leaves out. The boundary now stops at the segment's last frame, with a unit test and a CI-gated case (21 dB without it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frame preview now keeps every frame from the keyframe before its seek (-noaccurate_seek) instead of a 1 s preroll, so a sparse source, such as a screen recording holding a still, previews the frame export holds (19.5 dB before, now matching). Frames before the clip's in-point are dropped as export's seek drops them. The outline names each sentence's source when the timeline has more than one, so its transcriptId can be found, and a page never splits sentences that start at the same time, so paging always moves on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A frame preview restored its timestamps with a fractional offset while the seek rounds to whole stream ticks, so a 24 fps frame on a 30 fps canvas could land one canvas frame early (21 dB at 1.391 s). It now undoes the seek in whole ticks, leaving the timestamps export's own seek gives. An outline sentence ends where the timeline skips ahead over words it plays elsewhere (clips moved out of source order), so its index range never takes in words it doesn't show. The CLI sentence lines name the source when the timeline has more than one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on a closed pipe A slow-motion frame can need the source frame before the keyframe its own time falls just after; the preview now seeks a second early as well as keeping the frames from the keyframe before (21 dB before at 0.25x). `clip timeline transcribe --sentences | head` ended with an unhandled EPIPE once head closed the pipe on a long listing; a closed pipe now just ends it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
whisper.cpp gives each pause to the word before it and runs late at sentence ends, so a short cut by the transcript kept up to 0.35 s of the next sentence. make_short now measures the source's levels around each edge (one short FFmpeg read, 10 ms RMS frames), finds the pause the transcript boundary points at by a threshold set from those levels, and cuts 0.1 s into it. An edge with no clear pause keeps the transcript rule and warns; a word whisper places past the cut keeps its caption. On the 30-sentence say recording every sentence-short is clean: no edge clipped, none keeping a neighbor's speech (before: 18 ends kept the next sentence, 3 starts were clipped). On jfk.wav every cut lands in a pause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rder Levels are measured with the transcription's aresample=async=1:first_pts=0, so a late audio start or a timestamp gap can't shift a pause; digital silence counts as quiet but is left out of the floor estimate, or room tone between sentences would read as sound. A word pulled inside a cut keeps its place after the word before it, and the speech check uses the pulled words, so a one-word range whisper places past its cut is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in order A pause that ends by the time the last word starts is the one before it, so an accurately timed last word with a long pause ahead of it is no longer cut off. A range across a source split captions its words once, and words crowded against a cut move earlier to make room instead of sharing a start, so no caption shown a word at a time is lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 16, 2026
|
🎉 This PR is included in version 0.5.2 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice 2 of 4 toward v0.5.2 (long recording → vertical shorts). Targets the integration branch
feat/shorts, after #73 (slice 1); merging it releases nothing.feat/shortsgained #72 (color validation) while this was open; it is merged in, not rebased.Docs: README before · after (one new line in the shorts quick start); SKILL.md before · after (the transcribe and shorts paragraphs under "Build a multi-clip composition").
What
Sentence outline (
src/timeline/transcript.ts)outlineSentences(words, played?)groups words into sentences. A sentence ends at sentence-ending punctuation, before a pause over 0.7 s, or after 40 words. It also ends where the timeline goes back in the source, or skips ahead over words it plays elsewhere.timelineOutline(comp, media, opts)walks the picture track's media clips in timeline order and transcribes each source once (or reads its stored transcript).startSec/endSec(timeline time),clipId,fromIndex..toIndexandtext, plusmediaIdwhen the timeline has more than one source.nextFromSec. A page never splits sentences that start at the same time, so the cursor always moves on.fromSec/toSecselect a stretch, andclipIdone clip.endsSentenceandPAUSE_SECare now exported fromcaptions.tsinstead of copied.Transcript views
timeline_transcript:clipIdis optional.view: 'words' | 'sentences'defaults to words with a clip and sentences without one.fromSec/toSecare new. The sentence view is returned as compact JSON, the form the page size is counted in.clip timeline transcribe [<clipId>] [--sentences] [--from S] [--to S]prints one line per sentence, with no page limit:headends the listing cleanly.Sentence-edge warnings
planShortwarns when a range starts right after, or ends right before, a word of the same sentence (no sentence end and no real pause between). The warning names the words on either side; the short is still made.warnhook onVerbContext:applyVerbscollects them, andtimeline_editreturnswarnings.Frame preview from a short window (
src/timeline/compile.ts)buildSegmentStepintoclipChains, unchanged byte for byte (the 90 exact-args compile tests pass as they were).trimboundary is snapped up to the canvas grid and capped at the segment's last encoded frame. Encoding to the clip's length keepsround(length × fps)frames.docs/backlog/frame-preview-short-window.md.A short's edges land in the pause the audio shows (
src/timeline/pauses.ts,src/ffmpeg/args/speech-levels.ts,snapToPausesinshorts.ts)Added after review: the user folded it in after a five-lens council on the slice 1 finding.
make_shortnow reads the source's levels around each edge: one FFmpeg read of 2.5 s, 10 ms RMS frames at 16 kHz mono. Gaps and a late start are padded witharesample=async=1:first_pts=0, the same as transcription.warningssay so. The council split on this; the majority chose this fallback. The warning is skipped when the range was cut mid-sentence anyway, which already warns.A speed clip stays on the frame grid
fpsfilter now follows the speedsetpts. Without it, retimed frames reached the encoder off the grid, and its frame dropping repeated some frames and shifted others by up to two. That made export unsteady at 2x, and a preview couldn't match it.Why
On a long recording the agent couldn't read the transcript: 60 minutes of words is about 983 KB as one result. It also couldn't look at a frame: the preview encoded the whole clip, 53.6 s for a 10-minute one. This slice adds the outline the agent picks shorts from, warnings when it cuts mid-sentence, and a frame preview whose cost doesn't grow with the clip.
How to test
Checks
pnpm lint: clean (the same 7 pre-existing warnings intests/speed.test.tsandtests/transition.test.ts).pnpm type-check: clean.pnpm test: 937 passed, 32 skipped (after mergingfeat/shorts).pnpm build: OK.New unit tests:
tests/timeline-outline.test.ts(10):clipWordsacross three silence-cut sub-clips, with a word split by a cut listed once;mediaIdper sentence there, none with one source);tests/timeline-introspect.test.ts: the media frame is one step, and its graph equalssetpts…,trim=start=0,+ export's segment chain +trim=…. Also covered:tests/timeline-shorts.test.ts: warnings at a mid-sentence start and end, none for a whole sentence, the recording's edges or after a long pause.tests/mcp-server.test.ts:timeline_transcriptwithout a clip returns the compact sentence outline;timeline_editwithmake_shortreturns its warnings.Integration,
CI=true, bundled FFmpeg 6.0: the newtests/integration/timeline-frame-window.test.tscompares the window frame with the frame of the fully encoded clip (export's own path). The test pattern turns its hue 24° a frame, so a frame off by one scores about 21 dB.fpsfilter; 3.01 s 21.3 dB; the 1.01 s clip 21.0 dB; 1.391 s 21.2 dB; slow motion 21.0 dB; the sparse source 19.5 dB; the end of the source gave no image.Pause snapping:
Unit tests:
tests/timeline-pauses.test.ts(11) covers:tests/timeline-shorts.test.tscovers:maxWords: 1;CI-gated
tests/integration/timeline-pauses.test.ts(real FFmpeg, AAC, no drawtext): the pause found lies inside the real one, by at most 40 ms (AAC smears a sharp ending about 30 ms). The same holds for audio that starts 1 s late. A steady tone has no pause.Measured through the CLI. Every sentence of the 30-sentence
sayrecording as its own short (whisper base.en):Before: 18 ends kept the next sentence (up to 0.345 s) and 3 starts were clipped (up to 0.095 s).
jfk.wav: every cut lands in a measured pause, 0.07–0.12 s from the speech. The whole speech ends at 10.48 s instead of whisper's 11.0 s.
CLI, by hand:
transcribe --sentenceson a real whisper base.en transcript of asayrecording;--from 60 --to 70;short --dry-runwarnings;| head -3underpipefail, exit 0 with nothing on stderr (an unhandledEPIPEbefore the fix).Codex review (gpt-6.1-sol): every finding was fixed, none rebutted.
cut_wordsfor sentences that can span clips;clipIdoff the picture track outlined nothing.trim=startrounded an off-grid time to the nearer frame (fixed);-noaccurate_seek);transcriptIdwith several sources (mediaIdadded);setptscould move a frame one canvas frame;-noaccurate_seek);EPIPEcrash under| head.Notes
fpsfilter after a speedsetptschanges the frames of exported speed clips. They are now steady; before, some frames repeated and others were shifted up to two frames.cut-words-pause-edges:cut_wordsandremove_retakeshave the mirror problem.retake-test-model: a whisper-gated integration test fails with base.en, onfeat/shortstoo.short-edges-snap-to-pauseis deleted.