Skip to content

Vertical clips with a hook from a long video, in the dashboard's chat - #86

Merged
pallaoro merged 2 commits into
mainfrom
ai-step4-clips
Nov 10, 2026
Merged

pallaoro merged 2 commits into
mainfrom
ai-step4-clips

Conversation

@pallaoro

Copy link
Copy Markdown
Member

The request we get most: one long edited interview, 5–6 vertical clips with a strong hook. This makes it one sentence in the conversation, and moves the conversation into the Clawnify dashboard's chat when the app runs there.

What changes

Clips from a long video (make_clips, src/server/clips.ts)

  • Reads the video's transcript and makes one model call that picks moments that stand on their own (about 45 s by default, at most 8). Each moment opens on its strongest line: when that line comes later in the moment, it plays first as a cold open.
  • Each clip becomes its own project: 1080×1920, captions on, with the request's brief as the project brief. The project the request was sent to is not changed.
  • Every clip is then framed on its speaker by the same Services analysis as AI reframe. Each clip is analysed on its own stretch, never the whole talk, so the 40-shots-per-part cap stops mattering. Shots the analysis is unsure of keep the whole picture, and the reply says how many.
  • Framing is written only over the project as it was made. A clip someone already edited is left alone, and the reply says so.

Framing the cut (reframe): the same analysis on the current cut, into its 9:16 version (added if missing).

Waiting inside a background job: a task is resumable state inside InstructState. A step that has to wait on the transcript or the analyses ends the delivery, and the next delivery picks it up. Project ids are chosen before anything is created, so a lost delivery never creates a project twice. The task gives up after 40 minutes and keeps the whole picture for what didn't answer. Both operations always run in the background.

Dashboard chat (@clawnify/app 0.2.1 → 0.4.5, @clawnify/db → ^0.4.3 so there is only one copy)

  • Inside Clawnify the AI button opens the dashboard's chat (openChat). The editor tells the chat which project is open (useChatContext) and adds four "/" commands.
  • When the chat writes this project, the editor takes the new cut as one undo step (useHostChanges). The chat can also open a clip it made through the app's router (useHostNavigate).
  • A job strip above the player shows background work: what it's doing, Cancel, Run/Not now, Apply anyway, and links to the clips it made.
  • The in-app panel stays as the chat when the app runs on its own, now with links to the clips it made.
  • agent.md tells the dashboard chat to send requests to /instruct (with background: true for anything that watches footage) instead of writing the document itself.

Smaller changes: requestFraming and readFraming are shared by the route and the job, with no change in behaviour. The new schema column is ask_messages.made.

Tests

  • test/clips.test.ts (12 tests) covers:
    • picking: snapping to whole lines, hook rules, overlaps
    • the clip document
    • the whole task with a slow fake media service
    • an edited clip left alone
    • the deadline
    • creating each project once
    • framing a wide cut
    • the instruct integration: a background proposal when asked inline, made in the background run
  • e2e step 17: the client's request through the real routes and queue deliveries. Two projects at 9:16 with captions; the hook comes first and is framed cover with a crop; the asking project is unchanged.
  • vitest: 305 pass. test:e2e: all pass. tsc and vite build are clean.

Not tested here: the embedded dashboard-chat path needs the dashboard. It gets checked on Circulate after release.

@pallaoro
pallaoro merged commit aec3244 into main Nov 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant