Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,15 +161,16 @@ The same steps from the CLI:
```bash
clip timeline new
clip timeline add-media podcast.mp4 # prints the clip id, e.g. clip_ec2f047d
clip timeline transcribe clip_ec2f047d # each word with its index and time
clip timeline transcribe clip_ec2f047d # transcribe it (once per file)
clip timeline silence clip_ec2f047d # cut the pauses on main
clip timeline transcribe --sentences | head -80 # one line per sentence: time, word indices, text
clip timeline short best-1 clip_ec2f047d 1400 1478 --x 0.4 # words 1400–1478 as a 9:16 short
clip timeline --timeline best-1 frame 3 check.jpg # look at it; move --x if the speaker is off
clip timeline --timeline best-1 export best-1.mp4
clip timeline list # main and every short
```

`short` starts 0.25 s before the first word and ends with the last, fills the 9:16 frame (`--x` picks the part that shows), sets -14 LUFS and burns in captions. Give several `<clipId> <from> <to>` ranges to join moments, in the order you give them. Any command takes `--timeline <id>` to work on a short, `clip timeline --timeline best-1 undo` takes the short apart again, and `short best-1 … --replace` rebuilds it.
`short` cuts each edge inside the pause the audio shows there, keeping 0.1 s of room tone (where there's no clear pause it follows the transcript and warns), fills the 9:16 frame (`--x` picks the part that shows), sets -14 LUFS and burns in captions. Give several `<clipId> <from> <to>` ranges to join moments, in the order you give them. Any command takes `--timeline <id>` to work on a short, `clip timeline --timeline best-1 undo` takes the short apart again, and `short best-1 … --replace` rebuilds it.

---

Expand Down
4 changes: 2 additions & 2 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ To cut the pauses out of a talking clip already on the timeline, use `timeline s

After cutting, run `timeline check` before exporting: it lists every hard cut whose audio is loud right at the join (usually a clipped word or a click) and writes a contact sheet (`--sheet out.jpg`) with the outgoing clip's last frame and the incoming clip's first for each cut. Open the sheet to see the jumps. A flagged cut is a reason to trim or move that boundary, not a verdict. Export fades the audio over 8 ms on each side of every hard cut, so a cut through steady sound doesn't click; the check still flags cuts in sound, since those can clip a word.

To read what's said in a clip, run `timeline transcribe <clipId>`: it returns the clip's words with their timeline times, so you can find a sentence before cutting around it. Transcription is optional and runs a local whisper.cpp; if it isn't set up, the command says how (install whisper.cpp, download a model, set `MAKEMYCLIP_WHISPER_MODEL`). The first run per source file takes a while; later runs reuse the stored transcript. Whisper tends to leave out filler words like "um", so don't rely on the transcript to find them.
To read what's said in a clip, run `timeline transcribe <clipId>`: it returns the clip's words with their timeline times, so you can find a sentence before cutting around it. For a long recording, read `timeline transcribe --sentences` instead: one line per sentence of the whole timeline (`12:34 #1400–1413 clip_ab12 text`, the time, the word indices, the clip it starts in), and `--from S --to S` for a stretch; pipe it through `head` to read a page at a time. Transcription is optional and runs a local whisper.cpp; if it isn't set up, the command says how (install whisper.cpp, download a model, set `MAKEMYCLIP_WHISPER_MODEL`). The first run per source file takes a while; later runs reuse the stored transcript. Whisper tends to leave out filler words like "um", so don't rely on the transcript to find them.

To cut by words, use the `index` values from `timeline transcribe`: `timeline cut-words <clipId> <fromIndex> <toIndex>` removes those words (inclusive) and moves later clips on the track up. Add `--transcript <transcriptId>` from the same `timeline transcribe` output so the cut is refused, not misplaced, if the clip was transcribed again since. The transcript also lists `retakes`, phrases said twice in a row; `timeline remove-retakes <clipId>` cuts the earlier take of each (`--dry-run` lists them first). Each is one edit that `timeline undo` reverses. A cut splits the clip, so transcribe the resulting clips again before the next cut, and run `timeline check` afterwards.

Expand All @@ -333,7 +333,7 @@ To make a vertical (9:16), square (1:1) or feed (4:5) version, run `timeline ref

To even out the volume of the exported mp4, run `timeline loudness -14` (YouTube, Shorts, Reels, TikTok) or `-16` (podcasts); `off` turns it off. Export then measures the finished mp4 and normalizes it to that integrated loudness in a second pass (EBU R128, true peak at most -1.5 dBTP), and reports `loudness` with the measured before and after values. `mode: dynamic` means loudnorm couldn't apply one gain (steady or very short audio, a wide loudness range, or peaks); `skipped` means the audio was silent or too short to measure. It's one setting on the timeline that `timeline undo` reverses, it applies to the mp4 only, and it doesn't even out clips recorded at different levels.

To cut vertical shorts out of a long recording, transcribe it and make the cuts you want everywhere (pauses, retakes) on the main timeline first, then pick each moment by its words and run `timeline short <id> <clipId> <fromIndex> <toIndex>`, where `<id>` is a new name such as `best-1`. Pick whole sentences, 15–60 s in all; you choose the moments, the editor doesn't score them. The short is a timeline of its own, cut from main's clips (main's cuts carry in, main itself doesn't change): it starts 0.25 s before the first word and ends with the last (whisper's last word already covers the pause after it), fills a 9:16 frame, sets -14 LUFS and burns in captions. More `<clipId> <from> <to>` triples join several moments in the order given, so a later line can open the short as its hook. Options: `--x 0..1` to keep the speaker in frame, `--aspect 1:1|4:5`, `--words N` per caption, `--no-captions`, `--loudness <LUFS|off>`, `--source <timeline>`, `--dry-run` (prints the pieces), `--replace` to rebuild it. Every other command works on a short with `--timeline <id>`: look at it with `timeline --timeline best-1 frame 3 check.jpg`, adjust with `timeline --timeline best-1 reframe 9:16 --x 0.3`, and render it with `timeline --timeline best-1 export best-1.mp4`. `timeline list` shows every timeline, `timeline --timeline best-1 undo` takes the short apart, and `timeline delete best-1` removes it. Over MCP, it's the `make_short` verb in `timeline_edit` with `timeline` set to the new id; every timeline tool takes `timeline`, and `timeline_show` lists them all.
To cut vertical shorts out of a long recording, transcribe it and make the cuts you want everywhere (pauses, retakes) on the main timeline first, then read `timeline transcribe --sentences`, pick each moment as whole sentences and run `timeline short <id> <clipId> <fromIndex> <toIndex>` with their word indices, where `<id>` is a new name such as `best-1`. Pick whole sentences, 15–60 s in all; you choose the moments, the editor doesn't score them. The short is a timeline of its own, cut from main's clips (main's cuts carry in, main itself doesn't change): it cuts each edge inside the pause the audio shows there, keeping 0.1 s of room tone, so it never keeps the next sentence's first syllable, fills a 9:16 frame, sets -14 LUFS and burns in captions. More `<clipId> <from> <to>` triples join several moments in the order given, so a later line can open the short as its hook. Options: `--x 0..1` to keep the speaker in frame, `--aspect 1:1|4:5`, `--words N` per caption, `--no-captions`, `--loudness <LUFS|off>`, `--source <timeline>`, `--dry-run` (prints the pieces), `--replace` to rebuild it. Its `warnings` name a range that starts or ends mid-sentence (widen it to the sentence's edge unless the cut is meant) and an edge with no clear pause in the audio (music under the speech, or no break), which follows the transcript instead: look at that edge or listen to it. Every other command works on a short with `--timeline <id>`: look at it with `timeline --timeline best-1 frame 3 check.jpg`, adjust with `timeline --timeline best-1 reframe 9:16 --x 0.3`, and render it with `timeline --timeline best-1 export best-1.mp4`. `timeline list` shows every timeline, `timeline --timeline best-1 undo` takes the short apart, and `timeline delete best-1` removes it. Over MCP, it's the `make_short` verb in `timeline_edit` with `timeline` set to the new id; every timeline tool takes `timeline`, and `timeline_show` lists them all.

To hand the cut to an editor app, export the timeline as an editor project instead of a render. `timeline export cut.fcpxml` (or `--format fcpxml`) writes FCPXML 1.10 for Final Cut Pro and DaVinci Resolve; `timeline export cut.xml` (or `--format xmeml`) writes Final Cut Pro 7 XML for Premiere Pro, which Resolve reads too. Premiere doesn't import FCPXML, so use `xmeml` for it. Both reference the source files by absolute path (nothing is copied, so keep the sources where they are), place camera files by their start timecode, and keep the mp4's timing: clips, gaps, fades, volume, fade/dissolve transitions (as cross dissolves) and black or colored cards. Text over the picture becomes titles in FCPXML; in the Premiere XML it goes to `.srt` caption files next to the XML, since Premiere drops that format's titles. What the file can't hold — speed, color adjustments, chroma key, a reframe's fill, zoom or position (set the timeline to `--fit contain` and reset clip framing to carry it), rotation, wipe/slide/circle transitions, audio-only tracks, media on two tracks, a frame rate editors don't offer — is refused with the fix to make; what it holds only in part (text styling, caption boxes) is listed in the output's `notes`.

Expand Down
3 changes: 3 additions & 0 deletions docs/backlog/cut-words-pause-edges.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Cut words at the pause

`cut_words` and `remove_retakes` cut from the first word's start to the last word's end as whisper.cpp gives them (`planWordCut` in `src/timeline/transcript.ts`). whisper gives each pause to the word before it and runs late at sentence edges. On a 30-sentence test recording, cutting a sentence would also remove up to 0.35 s of the next sentence's first syllable, at 18 of 29 boundaries. Place those cut edges in the pause the audio shows, as `make_short` does (`src/timeline/pauses.ts`), through `VerbContext.speechLevels`.
3 changes: 0 additions & 3 deletions docs/backlog/frame-preview-short-window.md

This file was deleted.

3 changes: 3 additions & 0 deletions docs/backlog/retake-test-model.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Retake integration test depends on the model

`tests/integration/timeline-transcript.test.ts` ("removes a repeated take as one undoable edit") runs whenever whisper.cpp and a model are set up. With `ggml-base.en.bin` it fails on `feat/shorts` too: the model's transcript of the test's spoken phrase has no immediate repeat, so `retakes` is empty. Make the test independent of the model (assert on a stored transcript with a known repeat, as the unit tests do) or pin the model it was written for.
3 changes: 0 additions & 3 deletions docs/backlog/short-edges-snap-to-pause.md

This file was deleted.

2 changes: 1 addition & 1 deletion llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
## What v0.5.0 does

- **Remove silence:** `clip timeline silence <clipId>` cuts the pauses out of a recording (`--sensitivity`, `--dry-run`)
- **Transcribe locally:** `clip timeline transcribe <clipId>` returns each word with its timeline time, via whisper.cpp on the user's machine
- **Transcribe locally:** `clip timeline transcribe <clipId>` returns each word with its timeline time, via whisper.cpp on the user's machine; `clip timeline transcribe --sentences` lists a long recording as numbered sentences
- **Edit by transcript:** `clip timeline cut-words <clipId> <fromIndex> <toIndex>` cuts a range of words
- **Remove repeated takes:** `clip timeline remove-retakes <clipId>` cuts the earlier take of each phrase said twice in a row
- **Captions:** `clip timeline captions` burns in captions timed to the words (short phrases or one word at a time)
Expand Down
65 changes: 52 additions & 13 deletions src/cli-timeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ import {
} from './timeline/interchange.js';
import { buildMediaMap, buildMediaRefs } from './timeline/media-registry.js';
import { colorClip, mediaClip, textClip } from './timeline/ops.js';
import { measureSpeechLevels } from './timeline/pauses.js';
import {
AspectSchema,
type FramingFit,
Expand All @@ -63,6 +64,7 @@ import {
findRetakes,
planWordCut,
storedTranscriptForClip,
timelineOutline,
transcribeClip,
} from './timeline/transcript.js';
import { DEFAULT_VERB_TRACK as DEFAULT_TRACK, ensureTrack, trackEnd } from './timeline/verbs.js';
Expand All @@ -85,7 +87,7 @@ Every command works on the main timeline unless --timeline <id> names another
clip timeline remove <clipId>
clip timeline silence <clipId> [--sensitivity 0..1] [--dry-run]
clip timeline check [--sheet <output.jpg>]
clip timeline transcribe <clipId> [--model <model.bin>] [--language <code>]
clip timeline transcribe [<clipId>] [--sentences] [--from S] [--to S] [--model <model.bin>] [--language <code>]
clip timeline cut-words <clipId> <fromIndex> <toIndex> [--transcript <id>]
clip timeline remove-retakes <clipId> [--dry-run] [--transcript <id>]
clip timeline captions [<clipId>] [--words N] [--track T] [--font N --color C --anchor A --background C|none] [--dry-run]
Expand All @@ -110,7 +112,7 @@ interface Parsed {
}

/** Flags that take no value, so they don't swallow the next argument. */
const BOOLEAN_FLAGS = new Set(['dry-run', 'reset', 'no-captions', 'replace']);
const BOOLEAN_FLAGS = new Set(['dry-run', 'reset', 'no-captions', 'replace', 'sentences']);

function parseArgs(args: string[]): Parsed {
const positional: string[] = [];
Expand Down Expand Up @@ -177,6 +179,15 @@ function timelineId(value: string): string {
return parsed.data;
}

/** Timeline time as m:ss, or h:mm:ss from an hour. */
function clock(sec: number): string {
const whole = Math.floor(sec);
const s = String(whole % 60).padStart(2, '0');
const m = Math.floor(whole / 60) % 60;
const h = Math.floor(whole / 3600);
return h > 0 ? `${h}:${String(m).padStart(2, '0')}:${s}` : `${m}:${s}`;
}

function wordIndex(value: string | undefined): number {
const n = Number(value);
if (value === undefined || value.trim() === '' || !Number.isInteger(n) || n < 0) {
Expand Down Expand Up @@ -562,7 +573,15 @@ export async function runTimeline(args: string[]): Promise<void> {
flags.loudness === undefined ? undefined : loudnessValue(flags.loudness, usage),
replace: flags.replace === 'true',
},
(comp, clipId, transcriptId) => storedTranscriptForClip(comp, clipId, media, transcriptId),
{
transcript: (comp, clipId, transcriptId) =>
storedTranscriptForClip(comp, clipId, media, transcriptId),
levels: async (comp, clipId, aroundSec) => {
const clip = findClip(comp, clipId)?.clip;
const info = clip?.kind === 'media' ? media.get(clip.mediaId) : undefined;
return info?.hasAudio ? measureSpeechLevels(info.path, aroundSec) : null;
},
},
);
const report = {
timeline: target,
Expand Down Expand Up @@ -616,18 +635,38 @@ export async function runTimeline(args: string[]): Promise<void> {

case 'transcribe': {
const [clipId] = positional;
if (!clipId) {
throw new Error(
'Usage: clip timeline transcribe <clipId> [--model <model.bin>] [--language <code>]',
);
}
const comp = await readComposition(timeline);
out(
await transcribeClip(comp, clipId, await buildMediaMap(), {
model: flags.model ? resolveInput(flags.model) : undefined,
language: flags.language || undefined,
}),
const opts = {
model: flags.model ? resolveInput(flags.model) : undefined,
language: flags.language || undefined,
};
if (clipId && flags.sentences !== 'true') {
out(await transcribeClip(comp, clipId, await buildMediaMap(), opts));
return;
}
// One line per sentence, cheap to read and to `head`: no page limit here.
const outline = await timelineOutline(comp, await buildMediaMap(), {
...opts,
clipId,
fromSec: optionalNum('from', flags.from),
toSec: optionalNum('to', flags.to),
maxBytes: Number.POSITIVE_INFINITY,
});
const lines = Object.entries(outline.transcriptIds).map(
([mediaId, id]) => `transcript ${mediaId}: ${id}`,
);
for (const s of outline.sentences) {
// With more than one source, its media id, to match a transcript line.
const source = s.mediaId ? ` ${s.mediaId}` : '';
lines.push(
`${clock(s.startSec)} #${s.fromIndex}–${s.toIndex} ${s.clipId}${source} ${s.text}`,
);
}
// `| head` closes the pipe early: that ends the listing, not with an error.
process.stdout.on('error', (err: NodeJS.ErrnoException) => {
if (err.code !== 'EPIPE') throw err;
});
process.stdout.write(lines.length > 0 ? `${lines.join('\n')}\n` : '');
return;
}

Expand Down
44 changes: 44 additions & 0 deletions src/ffmpeg/args/speech-levels.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
export interface SpeechLevelsArgs {
input: string;
/** Where to start measuring, in source seconds. */
fromSec: number;
durationSec: number;
}

/** Samples per measured frame at 16 kHz: 10 ms. */
export const LEVEL_FRAME_SAMPLES = 160;
export const LEVEL_SAMPLE_RATE = 16000;

/**
* The RMS level of every 10 ms of a source's first audio stream over a window,
* printed to stderr as `pts_time:…` / `lavfi.astats.Overall.RMS_level=…` lines.
* Mono at 16 kHz is plenty to tell speech from a pause, and the window is a few
* seconds, so it runs in well under a tenth of a second.
*/
export function buildSpeechLevelsArgs({ input, fromSec, durationSec }: SpeechLevelsArgs): string[] {
return [
'-hide_banner',
'-nostats',
'-ss',
String(fromSec),
'-t',
String(durationSec),
'-i',
input,
'-map',
'0:a:0',
'-af',
[
// A late start or a timestamp gap becomes silence, as for transcription,
// so frame n is always n x 10 ms after the seek point.
`aresample=${LEVEL_SAMPLE_RATE}:async=1:first_pts=0`,
'aformat=channel_layouts=mono',
`asetnsamples=n=${LEVEL_FRAME_SAMPLES}:p=0`,
'astats=metadata=1:reset=1',
'ametadata=mode=print:key=lavfi.astats.Overall.RMS_level',
].join(','),
'-f',
'null',
'-',
];
}
13 changes: 13 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,17 @@ export {
textClip,
videoTrack,
} from './timeline/ops.js';
export {
endInPause,
findPauses,
measureSpeechLevels,
PAUSE_KEEP_SEC,
type Pause,
parseSpeechLevels,
pauseAt,
type SpeechLevels,
startInPause,
} from './timeline/pauses.js';
export {
type Aspect,
AspectSchema,
Expand Down Expand Up @@ -251,7 +262,9 @@ export {
type ShortPlan,
type ShortRange,
type ShortRequest,
type ShortSources,
type ShortTranscriptLookup,
type SpeechLevelsLookup,
} from './timeline/shorts.js';
export {
analyzeSilence,
Expand Down
Loading
Loading