diff --git a/README.md b/README.md index 9b65935..999c93e 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,7 @@ Most editors keep a project in a format built for their own app, not for you or - **Transitions**: put a dissolve, a fade through black or white, a wipe, a slide, a blur or a pixelate on any cut, from the mark between two clips on the timeline, or on every cut at once. A transition is centred on its cut and moves no clip, so titles and music stay where you put them, and the preview draws each one frame for frame as it exports. - **Captions**: switch them on for the whole video, pick one of twelve languages and one style. The words come from each clip's transcript, so captions follow every trim, split and reorder. - **AI, in one conversation**: add your clips and say what you want beside the cut, in your own words: it builds the cut from them ("a 30 second video from my clips") or changes the one you have. It remembers the conversation, so "shorter" or "a square one too" knows what it refers to, and it asks back instead of guessing when a request could mean two things. A request that would watch a lot of footage says how much first and, on a yes, runs in the background: keep editing or close the tab, and it never lands on top of a change you made meanwhile. The model calls a fixed set of checked operations, so the edit is always a valid document. One request, one undo. +- **Clips from a long video**: "5 vertical clips with a strong hook from this interview" reads what is said, picks the moments that stand on their own, opens each on its strongest line, and makes each its own 9:16 project with captions, framed on the person speaking. Shots it is unsure how to frame keep the whole picture, and it tells you which. "Frame it for vertical" does the same for the cut you have. - **AI assist**: Auto-cut watches several clips together and assembles the strongest sequence for what the video is for, captions included; Clean up trims one clip down to its good parts. - **Export to MP4** in draft, standard or high quality, on Clawnify's managed edit service. - **Share by link**: one link per project plays an export to anyone you send it to, with no sign-in, and a Download button. The link stays on the export you shared until you move it to a newer one, so a draft never reaches viewers; turning it off ends it for everyone. diff --git a/agent.md b/agent.md index cec054c..481876e 100644 --- a/agent.md +++ b/agent.md @@ -217,6 +217,35 @@ video from my clips" works from an empty timeline: it places a video with `add_clip`, or builds the whole main track from up to 8 of them with `assemble` (Auto-cut's pass, which replaces the main track). +### Asked in the Clawnify chat + +When a person asks you, in the dashboard's chat, to change a video project or +make videos from footage, send their words to `POST +/api/projects/{id}/instruct` rather than writing the document yourself: it +watches the footage, picks the steps (cutting, assembling, clips, framing) and +applies them as one step the person can undo, never over an edit they made +meanwhile. The editor beside the chat tells you which project is open (a +`project` record in your context). Send `background: true` when the request +watches footage (clips, framing, clean-up, "make a video from my clips"), then +tell the person it is running: the editor shows its progress. Read `GET +/api/projects/{id}/ask` for how it ended, and pass on what `said` says, +including any shot it was unsure how to frame. + +### Clips from a long video, and framing for vertical + +"5 vertical clips with a strong hook from this interview" is one request: +`/instruct` reads the video's transcript, picks the moments that stand on +their own (about 45 seconds unless asked otherwise, at most 8), opens each on +its strongest line (played first when it comes later in the moment), and makes +each one **its own project**: 1080×1920 with captions on, framed on the person +speaking. The project the request was sent to is not changed. The finished +message, and the job while it runs, list them in `made: [{ id, name }]`; open +one at `/edits/{id}`. "Frame it for vertical" frames the project's own cut the +same way, in its 9:16 version (added when missing). Shots the framing is unsure +of keep the whole picture, and the message says how many: a person checks them +in the editor's AI reframe. Both need the video on the media service (an +upload from the editor or a Drive import) and always run in the background. + `/instruct` is the same conversation people see in the editor's AI panel: it reads the last few turns, so a follow-up ("shorter") works, and your requests show there beside theirs. When a request could mean two things, or would watch @@ -317,7 +346,7 @@ in line, and asks Drive again now for the clips waiting on it. `DELETE /api/proj | PUT | `/api/projects/{id}` | Update `{ name?, brief?, edl? }` — the EDL is validated on save | | POST | `/api/projects/{id}/autocut` | `{ asset_ids, prompt? }` — AI assembles the main track from several clips | | POST | `/api/projects/{id}/instruct` | `{ instruction, background? }` — change the existing cut in words → `{ edl, said, applied, asked, job? }`; `asked: true` means `said` is a question back (or, with `job`, a proposal) and nothing was changed. `background: true` → `202` with a queued `job` | -| GET | `/api/projects/{id}/ask` | The project's AI conversation, oldest first, and its recent background jobs → `{ messages: [{ role, text, author, applied, asked, failed, job_id }], jobs }` | +| GET | `/api/projects/{id}/ask` | The project's AI conversation, oldest first, and its recent background jobs → `{ messages: [{ role, text, author, applied, asked, failed, job_id, made }], jobs }` (`made`: projects a request made) | | GET | `/api/projects/{id}/ask/jobs/{job}` | One background job with `base_edl` (the cut it started from), `draft_edl` (its cut so far, while it runs) and `result_edl` | | POST | `/api/projects/{id}/ask/jobs/{job}/run` | Yes to a proposal: run it in the background → `202` | | POST | `/api/projects/{id}/ask/jobs/{job}/cancel` | Drop a proposal, stop a queued or running job (its result is not applied), or let go of a kept one | diff --git a/package.json b/package.json index 29b4e6e..864b828 100644 --- a/package.json +++ b/package.json @@ -12,9 +12,9 @@ "test:e2e": "esbuild src/server/index.ts --bundle --format=esm --platform=node --outfile=test/e2e/.server.mjs --log-level=warning && node --no-warnings test/e2e/run.mjs ./.server.mjs schema.sql && node --no-warnings test/e2e/drive-imports.mjs" }, "dependencies": { - "@clawnify/app": "^0.2.1", + "@clawnify/app": "^0.4.5", "@clawnify/connections": "^0.7.0", - "@clawnify/db": "^0.4.2", + "@clawnify/db": "^0.4.3", "@clawnify/queue": "^0.1.1", "@clawnify/services": "0.6.0", "@fontsource-variable/inter": "^5.3.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6352c14..e4191f1 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9,14 +9,14 @@ importers: .: dependencies: '@clawnify/app': - specifier: ^0.2.1 - version: 0.2.1(@cloudflare/workers-types@4.20260613.1)(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(@phosphor-icons/react@2.1.10(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(hono@4.12.25)(react@19.2.7) + specifier: ^0.4.5 + version: 0.4.5(@cloudflare/workers-types@4.20260613.1)(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(@phosphor-icons/react@2.1.10(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(hono@4.12.25)(react@19.2.7) '@clawnify/connections': specifier: ^0.7.0 version: 0.7.0 '@clawnify/db': - specifier: ^0.4.2 - version: 0.4.2(@cloudflare/workers-types@4.20260613.1) + specifier: ^0.4.3 + version: 0.4.3(@cloudflare/workers-types@4.20260613.1) '@clawnify/queue': specifier: ^0.1.1 version: 0.1.1 @@ -77,10 +77,10 @@ importers: version: 19.2.3(@types/react@19.2.17) '@vitejs/plugin-react': specifier: ^4.0.0 - version: 4.7.0(vite@6.4.3(jiti@2.7.0)(lightningcss@1.32.0)(yaml@2.9.1)) + version: 4.7.0(supports-color@10.2.2)(vite@6.4.3(jiti@2.7.0)(lightningcss@1.32.0)(yaml@2.9.1)) clawnify: specifier: ^0.1.12 - version: 0.1.17 + version: 0.1.17(supports-color@10.2.2) concurrently: specifier: ^9.0.0 version: 9.2.1 @@ -190,8 +190,8 @@ packages: resolution: {integrity: sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==} engines: {node: '>=6.9.0'} - '@clawnify/app@0.2.1': - resolution: {integrity: sha512-t6fx4rnbEAewU3fH/WGKfWhfhZgVbLp1JgnpCR0YSHaE6rYwxEmgHAYlN3q5gSjYutsWmXANFqro3wuyxWZlTA==} + '@clawnify/app@0.4.5': + resolution: {integrity: sha512-EBaeJvqUIIyZd/hV5+0Gth1NGJoZKz8W1eBOg8ShAuE7rV0QWDP1rK2n4SKiZdMJA8BPkzZcYWaKNy44DNKPbA==} peerDependencies: '@hono/zod-openapi': '>=0.18 <1' '@phosphor-icons/react': '>=2' @@ -206,8 +206,8 @@ packages: '@clawnify/connections@0.7.0': resolution: {integrity: sha512-e5N3ItrwdHuW80tbx08bmi2Ya1CAxsCX9pnMq/LaRK4j0EMKqToztYx4a+17TGAnJ44JHpHBYwZI/tv4PSz7Qg==} - '@clawnify/db@0.4.2': - resolution: {integrity: sha512-s+G/5bDVFx7Na8WVETfl8EhyPcbmgs5LIE1TsM0vGUOAtPihQGlVX1xpfmg5lsMv8+OChKaHsZIMnzpXY8pyAw==} + '@clawnify/db@0.4.3': + resolution: {integrity: sha512-7CqhOilLT3xjyKwO1+B4hfo/Cy2jpkLfpmgXaiQUpKLmvyPYtqOf+3w+QQyGo0QWy5QzjN60HCX0COjk055ldg==} '@clawnify/integrations@0.5.0': resolution: {integrity: sha512-asGXIsAbSe/W7RLAxW4lLJCuILhw/I+H4KGCRVCRyvxPqm87IuGytNkb3XdgbvFTqj+H73wm05fZJgFMdAO1sQ==} @@ -215,8 +215,8 @@ packages: '@clawnify/queue@0.1.1': resolution: {integrity: sha512-6gfaz4B3rVVFWMB0IjJ0mt3ausETXpiWodRqShMBTRwVtNyCXJ4FO4/f3qc/TYtnA9ZZBPVeBLjbzg5nTqvULw==} - '@clawnify/routes@0.2.2': - resolution: {integrity: sha512-+s7gjRyH8bmIUHQDHmmwGyUeEcV5m+IoWEYr2tLiWdVW7WKlFlgeKIuJ61vW9mc9k4TizL+B+Tg8qJ72zAbvOA==} + '@clawnify/routes@0.3.0': + resolution: {integrity: sha512-ALKVoN1ui9KXSSoybSKumEiJ7fTAe793y2HwCPGEkvXlgV29yAGXad8qVOeGYzrKe6KACJ8twFA4uheQrJSmqA==} peerDependencies: '@hono/zod-openapi': '>=0.18 <1' hono: '*' @@ -2606,20 +2606,20 @@ snapshots: '@babel/compat-data@7.29.7': {} - '@babel/core@7.29.7': + '@babel/core@7.29.7(supports-color@10.2.2)': dependencies: '@babel/code-frame': 7.29.7 '@babel/generator': 7.29.7 '@babel/helper-compilation-targets': 7.29.7 - '@babel/helper-module-transforms': 7.29.7(@babel/core@7.29.7) + '@babel/helper-module-transforms': 7.29.7(@babel/core@7.29.7(supports-color@10.2.2))(supports-color@10.2.2) '@babel/helpers': 7.29.7 '@babel/parser': 7.29.7 '@babel/template': 7.29.7 - '@babel/traverse': 7.29.7 + '@babel/traverse': 7.29.7(supports-color@10.2.2) '@babel/types': 7.29.7 '@jridgewell/remapping': 2.3.5 convert-source-map: 2.0.0 - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) gensync: 1.0.0-beta.2 json5: 2.2.3 semver: 6.3.1 @@ -2644,19 +2644,19 @@ snapshots: '@babel/helper-globals@7.29.7': {} - '@babel/helper-module-imports@7.29.7': + '@babel/helper-module-imports@7.29.7(supports-color@10.2.2)': dependencies: - '@babel/traverse': 7.29.7 + '@babel/traverse': 7.29.7(supports-color@10.2.2) '@babel/types': 7.29.7 transitivePeerDependencies: - supports-color - '@babel/helper-module-transforms@7.29.7(@babel/core@7.29.7)': + '@babel/helper-module-transforms@7.29.7(@babel/core@7.29.7(supports-color@10.2.2))(supports-color@10.2.2)': dependencies: - '@babel/core': 7.29.7 - '@babel/helper-module-imports': 7.29.7 + '@babel/core': 7.29.7(supports-color@10.2.2) + '@babel/helper-module-imports': 7.29.7(supports-color@10.2.2) '@babel/helper-validator-identifier': 7.29.7 - '@babel/traverse': 7.29.7 + '@babel/traverse': 7.29.7(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -2677,14 +2677,14 @@ snapshots: dependencies: '@babel/types': 7.29.7 - '@babel/plugin-transform-react-jsx-self@7.29.7(@babel/core@7.29.7)': + '@babel/plugin-transform-react-jsx-self@7.29.7(@babel/core@7.29.7(supports-color@10.2.2))': dependencies: - '@babel/core': 7.29.7 + '@babel/core': 7.29.7(supports-color@10.2.2) '@babel/helper-plugin-utils': 7.29.7 - '@babel/plugin-transform-react-jsx-source@7.29.7(@babel/core@7.29.7)': + '@babel/plugin-transform-react-jsx-source@7.29.7(@babel/core@7.29.7(supports-color@10.2.2))': dependencies: - '@babel/core': 7.29.7 + '@babel/core': 7.29.7(supports-color@10.2.2) '@babel/helper-plugin-utils': 7.29.7 '@babel/template@7.29.7': @@ -2693,7 +2693,7 @@ snapshots: '@babel/parser': 7.29.7 '@babel/types': 7.29.7 - '@babel/traverse@7.29.7': + '@babel/traverse@7.29.7(supports-color@10.2.2)': dependencies: '@babel/code-frame': 7.29.7 '@babel/generator': 7.29.7 @@ -2701,7 +2701,7 @@ snapshots: '@babel/parser': 7.29.7 '@babel/template': 7.29.7 '@babel/types': 7.29.7 - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) transitivePeerDependencies: - supports-color @@ -2710,10 +2710,10 @@ snapshots: '@babel/helper-string-parser': 7.29.7 '@babel/helper-validator-identifier': 7.29.7 - '@clawnify/app@0.2.1(@cloudflare/workers-types@4.20260613.1)(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(@phosphor-icons/react@2.1.10(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(hono@4.12.25)(react@19.2.7)': + '@clawnify/app@0.4.5(@cloudflare/workers-types@4.20260613.1)(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(@phosphor-icons/react@2.1.10(react-dom@19.2.7(react@19.2.7))(react@19.2.7))(hono@4.12.25)(react@19.2.7)': dependencies: - '@clawnify/db': 0.4.2(@cloudflare/workers-types@4.20260613.1) - '@clawnify/routes': 0.2.2(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(hono@4.12.25) + '@clawnify/db': 0.4.3(@cloudflare/workers-types@4.20260613.1) + '@clawnify/routes': 0.3.0(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(hono@4.12.25) '@hono/zod-openapi': 0.19.10(hono@4.12.25)(zod@3.25.76) hono: 4.12.25 optionalDependencies: @@ -2754,7 +2754,7 @@ snapshots: dependencies: '@clawnify/integrations': 0.5.0 - '@clawnify/db@0.4.2(@cloudflare/workers-types@4.20260613.1)': + '@clawnify/db@0.4.3(@cloudflare/workers-types@4.20260613.1)': dependencies: drizzle-orm: 0.45.2(@cloudflare/workers-types@4.20260613.1) transitivePeerDependencies: @@ -2792,7 +2792,7 @@ snapshots: '@clawnify/queue@0.1.1': {} - '@clawnify/routes@0.2.2(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(hono@4.12.25)': + '@clawnify/routes@0.3.0(@hono/zod-openapi@0.19.10(hono@4.12.25)(zod@3.25.76))(hono@4.12.25)': dependencies: '@hono/zod-openapi': 0.19.10(hono@4.12.25)(zod@3.25.76) hono: 4.12.25 @@ -3147,18 +3147,18 @@ snapshots: '@jridgewell/resolve-uri': 3.1.2 '@jridgewell/sourcemap-codec': 1.5.5 - '@modelcontextprotocol/sdk@1.29.0(zod@3.25.76)': + '@modelcontextprotocol/sdk@1.29.0(supports-color@10.2.2)(zod@3.25.76)': dependencies: '@hono/node-server': 1.19.14(hono@4.12.25) ajv: 8.20.0 - ajv-formats: 3.0.1(ajv@8.20.0) + ajv-formats: 3.0.1 content-type: 1.0.5 cors: 2.8.6 cross-spawn: 7.0.6 eventsource: 3.0.7 eventsource-parser: 3.1.0 - express: 5.2.1 - express-rate-limit: 8.5.2(express@5.2.1) + express: 5.2.1(supports-color@10.2.2) + express-rate-limit: 8.5.2(express@5.2.1(supports-color@10.2.2)) hono: 4.12.25 jose: 6.2.3 json-schema-typed: 8.0.2 @@ -3658,11 +3658,11 @@ snapshots: dependencies: csstype: 3.2.3 - '@vitejs/plugin-react@4.7.0(vite@6.4.3(jiti@2.7.0)(lightningcss@1.32.0)(yaml@2.9.1))': + '@vitejs/plugin-react@4.7.0(supports-color@10.2.2)(vite@6.4.3(jiti@2.7.0)(lightningcss@1.32.0)(yaml@2.9.1))': dependencies: - '@babel/core': 7.29.7 - '@babel/plugin-transform-react-jsx-self': 7.29.7(@babel/core@7.29.7) - '@babel/plugin-transform-react-jsx-source': 7.29.7(@babel/core@7.29.7) + '@babel/core': 7.29.7(supports-color@10.2.2) + '@babel/plugin-transform-react-jsx-self': 7.29.7(@babel/core@7.29.7(supports-color@10.2.2)) + '@babel/plugin-transform-react-jsx-source': 7.29.7(@babel/core@7.29.7(supports-color@10.2.2)) '@rolldown/pluginutils': 1.0.0-beta.27 '@types/babel__core': 7.20.5 react-refresh: 0.17.0 @@ -3686,8 +3686,8 @@ snapshots: mime-types: 3.0.2 negotiator: 1.0.0 - ajv-formats@3.0.1(ajv@8.20.0): - optionalDependencies: + ajv-formats@3.0.1: + dependencies: ajv: 8.20.0 ajv@8.20.0: @@ -3713,11 +3713,11 @@ snapshots: blake3-wasm@2.1.5: {} - body-parser@2.2.2: + body-parser@2.2.2(supports-color@10.2.2): dependencies: bytes: 3.1.2 content-type: 1.0.5 - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) http-errors: 2.0.1 iconv-lite: 0.7.2 on-finished: 2.4.1 @@ -3762,9 +3762,9 @@ snapshots: ansi-styles: 4.3.0 supports-color: 7.2.0 - clawnify@0.1.17: + clawnify@0.1.17(supports-color@10.2.2): dependencies: - '@modelcontextprotocol/sdk': 1.29.0(zod@3.25.76) + '@modelcontextprotocol/sdk': 1.29.0(supports-color@10.2.2)(zod@3.25.76) commander: 13.1.0 open: 10.2.0 zod: 3.25.76 @@ -3841,9 +3841,11 @@ snapshots: custom-error-instance@2.1.1: {} - debug@4.4.3: + debug@4.4.3(supports-color@10.2.2): dependencies: ms: 2.1.3 + optionalDependencies: + supports-color: 10.2.2 default-browser-id@5.0.1: {} @@ -3971,25 +3973,25 @@ snapshots: expect-type@1.4.0: {} - express-rate-limit@8.5.2(express@5.2.1): + express-rate-limit@8.5.2(express@5.2.1(supports-color@10.2.2)): dependencies: - express: 5.2.1 + express: 5.2.1(supports-color@10.2.2) ip-address: 10.2.0 - express@5.2.1: + express@5.2.1(supports-color@10.2.2): dependencies: accepts: 2.0.0 - body-parser: 2.2.2 + body-parser: 2.2.2(supports-color@10.2.2) content-disposition: 1.1.0 content-type: 1.0.5 cookie: 0.7.2 cookie-signature: 1.2.2 - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) depd: 2.0.0 encodeurl: 2.0.0 escape-html: 1.0.3 etag: 1.8.1 - finalhandler: 2.1.1 + finalhandler: 2.1.1(supports-color@10.2.2) fresh: 2.0.0 http-errors: 2.0.1 merge-descriptors: 2.0.0 @@ -4000,9 +4002,9 @@ snapshots: proxy-addr: 2.0.7 qs: 6.15.2 range-parser: 1.2.1 - router: 2.2.0 - send: 1.2.1 - serve-static: 2.2.1 + router: 2.2.0(supports-color@10.2.2) + send: 1.2.1(supports-color@10.2.2) + serve-static: 2.2.1(supports-color@10.2.2) statuses: 2.0.2 type-is: 2.1.0 vary: 1.1.2 @@ -4017,9 +4019,9 @@ snapshots: optionalDependencies: picomatch: 4.0.4 - finalhandler@2.1.1: + finalhandler@2.1.1(supports-color@10.2.2): dependencies: - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) encodeurl: 2.0.0 escape-html: 1.0.3 on-finished: 2.4.1 @@ -4404,9 +4406,9 @@ snapshots: '@rollup/rollup-win32-x64-msvc': 4.62.0 fsevents: 2.3.3 - router@2.2.0: + router@2.2.0(supports-color@10.2.2): dependencies: - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) depd: 2.0.0 is-promise: 4.0.0 parseurl: 1.3.3 @@ -4428,9 +4430,9 @@ snapshots: semver@7.8.4: {} - send@1.2.1: + send@1.2.1(supports-color@10.2.2): dependencies: - debug: 4.4.3 + debug: 4.4.3(supports-color@10.2.2) encodeurl: 2.0.0 escape-html: 1.0.3 etag: 1.8.1 @@ -4444,12 +4446,12 @@ snapshots: transitivePeerDependencies: - supports-color - serve-static@2.2.1: + serve-static@2.2.1(supports-color@10.2.2): dependencies: encodeurl: 2.0.0 escape-html: 1.0.3 parseurl: 1.3.3 - send: 1.2.1 + send: 1.2.1(supports-color@10.2.2) transitivePeerDependencies: - supports-color diff --git a/schema.sql b/schema.sql index 23dfa95..90d1258 100644 --- a/schema.sql +++ b/schema.sql @@ -466,3 +466,6 @@ CREATE INDEX IF NOT EXISTS idx_ask_jobs_project ON ask_jobs(project_id, created_ -- The background job an AI message belongs to, when it does. ALTER TABLE ask_messages ADD COLUMN job_id TEXT; + +-- Projects an AI message made (clips cut from a long video), as JSON [{id, name}]. +ALTER TABLE ask_messages ADD COLUMN made TEXT; diff --git a/src/client/app.tsx b/src/client/app.tsx index 4041b74..51805d8 100644 --- a/src/client/app.tsx +++ b/src/client/app.tsx @@ -1,6 +1,6 @@ import { useCallback, useEffect, useState } from "react"; import { ArrowLeft, Film } from "lucide-react"; -import { AppNav, embedded, reportLocation } from "@clawnify/app/client"; +import { AppNav, embedded, reportLocation, useHostNavigate } from "@clawnify/app/client"; import { EditRoute, ProjectsHome, type EditProject } from "./edit"; import { SharedHighlightsRoute } from "./library-highlights"; import { HighlightsRoute } from "./highlights"; @@ -56,6 +56,8 @@ function HostNav({ active, path, navigate }: { active: string; path: string; nav export function App() { const { path, navigate } = useRouter(); + // The dashboard's chat opens a page here (a clip it made) through this router, without a reload. + useHostNavigate(navigate); const id = decodeURIComponent(path.replace(/^\/+|\/+$/g, "")); const m = /^edits\/([^/]+)(\/highlights)?$/.exec(id); const editId = m ? m[1] : null; diff --git a/src/client/assistant.tsx b/src/client/assistant.tsx index 6061f27..11cd15c 100644 --- a/src/client/assistant.tsx +++ b/src/client/assistant.tsx @@ -6,11 +6,14 @@ // asks back instead of guessing. A request that would watch a lot of footage // says so first and, on a yes, runs in the background (src/server/ask-jobs.ts): // the editor keeps working, the tab can close, and the result never lands on -// top of a change made meanwhile. Auto-cut and AI reframe still run in their -// own dialogs, opened from here, until the conversation can run them itself. +// top of a change made meanwhile. +// +// Inside Clawnify the conversation is the dashboard's chat, which calls the +// same /instruct: the editor then shows only the background work in a strip +// (JobStrip). This panel is the chat when the app runs on its own. import { useCallback, useEffect, useRef, useState } from "react"; -import { Crop, Loader2, RotateCcw, Send, Sparkles, X } from "lucide-react"; +import { Crop, Film, Loader2, RotateCcw, Send, Sparkles, X } from "lucide-react"; import { api, type Edl } from "./edit"; import { btnGhost, btnIcon, btnPrimary, btnSecondary } from "./ui"; @@ -23,6 +26,13 @@ export interface AskMessage { asked: boolean; failed: boolean; job_id?: string | null; + /** Projects it made: clips cut from a long video. */ + made?: Made[]; +} + +export interface Made { + id: string; + name: string; } export type AskJobStatus = "proposed" | "queued" | "running" | "applied" | "stale" | "failed" | "cancelled"; @@ -30,10 +40,13 @@ export type AskJobStatus = "proposed" | "queued" | "running" | "applied" | "stal export interface AskJob { id: string; status: AskJobStatus; + instruction: string; applied: string[]; /** What it is doing now, while it runs. */ doing?: string | null; error: string | null; + /** Clip projects made so far, while it runs. */ + made?: Made[]; } const ACTIVE: AskJobStatus[] = ["queued", "running"]; @@ -41,13 +54,14 @@ const ACTIVE: AskJobStatus[] = ["queued", "running"]; /** Requests that show what it can do, sent as written. */ const SUGGESTIONS = [ "Make a 30 second video from my clips", + "Make 5 vertical clips with a strong hook from my talk", "Remove the pauses and dead air from every clip", - "Add a vertical 9:16 version for Reels", + "Frame it for vertical, keeping the speaker in frame", "Add a title for the first 3 seconds", ]; /** JSON with keys in order: the server stores a cut with its keys in another order. */ -function canonical(v: unknown): string { +export function canonical(v: unknown): string { if (Array.isArray(v)) return `[${v.map(canonical).join(",")}]`; if (v && typeof v === "object") { return `{${Object.keys(v as object) @@ -137,6 +151,103 @@ export function useAsk(projectId: string, getEdl: () => Edl, onApplied: (edl: Ed export type Ask = ReturnType; +/** Links to the projects a request made. */ +function MadeLinks({ made, onOpen }: { made: Made[]; onOpen: (id: string) => void }) { + return ( +
+ {made.map((p) => ( + + ))} +
+ ); +} + +/** + * The background work, when the conversation is the dashboard's chat: what a + * job is doing, its controls, and the clips it made. Nothing shows while + * nothing needs the person. + */ +export function JobStrip({ ask, onOpen }: { ask: Ask; onOpen: (id: string) => void }) { + const { jobs, messages, notTaken, act } = ask; + const [acting, setActing] = useState(null); + const [err, setErr] = useState(""); + const [dismissed, setDismissed] = useState>(new Set()); + const doAct = async (id: string, action: "run" | "cancel" | "apply") => { + setActing(`${id}:${action}`); + setErr(""); + try { + await act(id, action); + } catch (e) { + setErr(String((e as Error).message)); + } finally { + setActing(null); + } + }; + // The newest message of each finished job that made clips, until dismissed. + const madeBy = new Map(); + for (const m of messages ?? []) if (m.job_id && m.made?.length) madeBy.set(m.job_id, m.made); + const shown = jobs.filter( + (j) => + !dismissed.has(j.id) && + (j.status === "proposed" || ACTIVE.includes(j.status) || j.status === "stale" || notTaken.has(j.id) || (j.status === "applied" && madeBy.has(j.id))), + ); + if (!shown.length && !err) return null; + const close = (id: string) => setDismissed((d) => new Set(d).add(id)); + return ( +
+ {shown.map((j) => { + const busy = acting !== null; + return ( +
+
+ {ACTIVE.includes(j.status) ? : } + + {j.status === "proposed" + ? `Waiting for a yes: ${j.instruction}` + : j.status === "queued" + ? "Starting in the background…" + : j.status === "running" + ? `${j.doing ?? "Working"}…` + : j.status === "applied" + ? `Made ${madeBy.get(j.id)!.length} clips` + : "A change from the chat is waiting: the cut changed while it ran"} + + {j.status === "proposed" && ( + <> + + + + )} + {ACTIVE.includes(j.status) && ( + + )} + {(j.status === "stale" || notTaken.has(j.id)) && ( + <> + + {j.status === "stale" && ( + + )} + + )} + {j.status === "applied" && ( + + )} +
+ {(j.status === "applied" ? madeBy.get(j.id) : j.made)?.length ? ( + + ) : null} +
+ ); + })} + {err &&

{err}

} +
+ ); +} + export function AssistantPanel({ projectId, ask, @@ -147,6 +258,7 @@ export function AssistantPanel({ onAutocut, onReframe, onClose, + onOpenProject, }: { projectId: string; ask: Ask; @@ -158,6 +270,7 @@ export function AssistantPanel({ onAutocut: () => void; onReframe: () => void; onClose: () => void; + onOpenProject: (id: string) => void; }) { const { messages, setMessages, jobs, notTaken, loadErr, refresh, act } = ask; const [draft, setDraft] = useState(""); @@ -250,6 +363,7 @@ export function AssistantPanel({ Cancel + {job.made?.length ? : null} {job.applied.length > 0 && (
    {job.applied.map((a, i) => ( @@ -343,6 +457,7 @@ export function AssistantPanel({ ))}
)} + {m.made?.length ? : null} {job && controls(job)} ); diff --git a/src/client/edit.tsx b/src/client/edit.tsx index 1e481d8..71c4b3b 100644 --- a/src/client/edit.tsx +++ b/src/client/edit.tsx @@ -1,5 +1,6 @@ import { ReframeDialog } from "./reframe"; -import { AssistantPanel, useAsk } from "./assistant"; +import { AssistantPanel, JobStrip, canonical, useAsk } from "./assistant"; +import { embedded, openChat, useChatCommands, useChatContext, useHostChanges } from "@clawnify/app/client"; // Footage edit projects — the timeline editor over EDL documents. // // Layout (the classic four-region editor grid): left rail (Media / Audio / @@ -1089,6 +1090,7 @@ export function EditRoute({ id, navigate }: { id: string; navigate: (to: string) initialAssets={assets} initialFootage={footage} onOpenHighlights={() => navigate(`/edits/${id}/highlights`)} + onOpenProject={(other) => navigate(`/edits/${other}`)} /> ); } @@ -1098,12 +1100,15 @@ export function EditEditor({ initialAssets, initialFootage, onOpenHighlights, + onOpenProject, }: { initial: EditProject; initialAssets: Asset[]; initialFootage: FootageList; /** Opens the footage's highlights. */ onOpenHighlights?: () => void; + /** Opens another project, such as a clip the AI made. */ + onOpenProject?: (id: string) => void; }) { const [name, setName] = useState(initial.name); const [brief, setBrief] = useState(initial.brief ?? ""); @@ -1224,6 +1229,36 @@ export function EditEditor({ ); const ask = useAsk(initial.id, () => edlRef.current, applyAi); + // Inside Clawnify the conversation is the dashboard's chat. It knows which + // project is open, offers its requests in the "/" menu, and calls this app's + // /instruct: the same planner as the panel, so the work is the same. + useChatContext({ label: "Video project", record: { type: "project", id: initial.id, label: name } }); + useChatCommands( + useMemo(() => { + const here = `In the video project open beside this chat (OpenVideo project id ${initial.id}),`; + return [ + { id: "clips", label: "Vertical clips from a talk", description: "Short 9:16 clips, each opening on a hook", prompt: `${here} make 5 vertical clips of about 45 seconds from its long video, each opening on a strong hook.` }, + { id: "assemble", label: "Video from my clips", description: "A 30 second cut from the clips added", prompt: `${here} make a 30 second video from the clips I added.` }, + { id: "clean", label: "Remove the pauses", description: "Dead air, false starts and filler", prompt: `${here} remove the pauses and dead air from every clip.` }, + { id: "frame", label: "Frame for vertical", description: "Keep the speaker in a 9:16 frame", prompt: `${here} frame the cut for vertical 9:16, keeping the speaker in frame.` }, + ]; + }, [initial.id]), + ); + // The chat changed this project: take the cut it wrote as one undo step. + // A change typed here in the last moment before is not lost: undo has it. + useHostChanges((paths) => { + if (!paths.some((p) => p.startsWith(`/api/projects/${initial.id}`))) return; + void ask.refresh(); + void api + .get(`/api/projects/${initial.id}`) + .then((p) => { + if (canonical(p.edl) === canonical(saved.current.edl) || canonical(p.edl) === canonical(edlRef.current)) return; + saved.current = { ...saved.current, edl: p.edl }; + applyAi(p.edl); + }) + .catch(() => {}); + }); + /** Edit the cut as the format on screen sees it; framing lands on that format, the rest on the cut. */ const update = useCallback( (fn: (draft: Edl) => void, coalesce = false) => { @@ -1544,6 +1579,7 @@ export function EditEditor({