Repository navigation
Add media-studio plugin - #3
Merged
Merged
Conversation
Review the images, video and audio agents make, for editors who run the whole edit through Paseo agents: - preview_media agent tool and a card after each turn with the media it showed or created - streaming player (fragmented MP4 over plugin RPC) with filmstrip, waveform, in/out, loop and editor keys; long-GOP seeks bridged with a near-lossless encode to the next keyframe - per-file playback memory across tab switches, reloads and re-renders; timecode entry; A/B flip between versions; safe-area and crop guides; EBU R128 loudness and true peak - timecoded notes with the current frame, sent to busy agents without interrupting them, kept per file and pinned on the timeline - Media panel with resumable uploads and streamed downloads over the Paseo connection - files on SSH hosts (alias:/path) processed by that host's ffmpeg
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved upload-ID path traversal permits deletion outside the upload directory, while upload races and unbound download reads can silently overwrite or corrupt files.
12 open issues
Publish upload destinations atomically without replacement · New Validate upload IDs before deleting records · New Pin downloads to a stable render version · New Preserve the requested frame inspection time · New Preserve nonterminal and resumable transfers when clearing · New Include queued jobs in reload handover · New Wait for the baseline scan before starting the turn · New Use a supported framelog value for EBU R128 · New Reject loudness measurements from failed ffmpeg runs · New Reject incomplete waveforms after decode failures · New Make upload completion retries idempotent · New Make fractional-rate timecodes round-trip correctly · New
What changed in this PR
Adds media-studio to Paseo’s plugin collection so users can review agent-generated media and send timecoded feedback without leaving the workspace.
Changes:
- Adds media preview cards, streaming playback, timelines, playback memory, and review notes.
- Adds resumable uploads, streamed downloads, and processing through local or SSH-hosted ffmpeg.
- Documents installation and requirements, provides English and Chinese UI text, and adds CI typechecking.
| File | Description |
|---|---|
| README.zh-CN.md | Documents features, requirements, and installation in Chinese. |
| README.md | Documents features, requirements, and installation in English. |
| plugins/media-studio/tsconfig.json | Configures strict TypeScript checking. |
| plugins/media-studio/shared/settings.ts | Defines host-level plugin settings. |
| plugins/media-studio/shared/rpc.ts | Defines media, transfer, and notes RPC contracts. |
| plugins/media-studio/shared/media.ts | Adds media classification, path, and timecode helpers. |
| plugins/media-studio/shared/cards.ts | Defines preview-card data and tool-input parsing. |
| plugins/media-studio/server/transfer.ts | Implements resumable uploads and chunked downloads. |
| plugins/media-studio/server/stream.ts | Implements streaming, transcoding, and seek bridges. |
| plugins/media-studio/server/stills.ts | Generates frames, filmstrips, waveforms, and image previews. |
| plugins/media-studio/server/probe.ts | Extracts and caches media metadata. |
| plugins/media-studio/server/notes.ts | Persists per-file review notes. |
| plugins/media-studio/server/loudness.ts | Measures audio loudness and true peak. |
| plugins/media-studio/server/files.ts | Scans media folders and selects destination filenames. |
| plugins/media-studio/server/ffmpeg.ts | Adds process execution and concurrency limits. |
| plugins/media-studio/server/cache.ts | Caches generated previews and trims disk usage. |
| plugins/media-studio/server/agent.ts | Registers the preview tool and after-turn cards. |
| plugins/media-studio/README.md | Documents installation, architecture, and development. |
| plugins/media-studio/paseo-plugin.json | Declares plugin identity and minimum Paseo version. |
| plugins/media-studio/package.json | Declares packaging and development dependencies. |
| plugins/media-studio/OVERVIEW.md | Explains workflows, controls, setup, and limitations. |
| plugins/media-studio/index.server.ts | Wires settings, RPC handlers, and server cleanup. |
| plugins/media-studio/index.client.tsx | Registers panels, cards, settings, and commands. |
| plugins/media-studio/client/ui.tsx | Adds shared themed controls and display helpers. |
| plugins/media-studio/client/transfers.ts | Manages transfer retries, controls, and reload handover. |
| plugins/media-studio/client/timeline.tsx | Renders media lanes, markers, and scrubbing controls. |
| plugins/media-studio/client/state.ts | Stores panel requests and transfer state. |
| plugins/media-studio/client/settings-screen.tsx | Provides configuration and host-tool checks. |
| plugins/media-studio/client/review.tsx | Composes and sends timecoded notes with frames. |
| plugins/media-studio/client/queries.ts | Fetches metadata, previews, and file changes. |
| plugins/media-studio/client/playback.ts | Remembers playback state per host and file. |
| plugins/media-studio/client/notes.tsx | Displays, navigates, and deletes review notes. |
| plugins/media-studio/client/locales/zh-CN.json | Provides Chinese UI translations. |
| plugins/media-studio/client/locales/en.json | Provides English UI translations. |
| plugins/media-studio/client/i18n.ts | Registers translation resources. |
| plugins/media-studio/client/file-panel.tsx | Integrates media viewing into file tabs. |
| plugins/media-studio/client/card.tsx | Renders inline media cards and thumbnails. |
| package.json | Adds media-studio to the collection description. |
| .github/workflows/typecheck.yml | Adds media-studio to the typecheck matrix. |
Review effort: Balanced
| const stem = name.slice(0, name.length - ext.length); | ||
| for (let index = 1; index < 1000; index += 1) { | ||
| const candidate = path.posix.join(dir, index === 1 ? name : `${stem} (${index})${ext}`); | ||
| if (!(await host.stat(candidate))) return candidate; |
| } | ||
|
|
||
| function recordFile(uploadId: string): string { | ||
| return path.join(STATE_DIR, `${uploadId}.json`); |
Comment on lines
+329
to
+330
| const data = await download.location.host.readRange(download.location.path, input.offset, length); | ||
| if (data.length < length) throw new Error("The file changed while it was being downloaded."); |
Comment on lines
+91
to
+94
| /** The frame at `time`, rounded to a tenth of a second so scrubbing reuses requests. */ | ||
| export function useFrame(info: MediaInfo | undefined, time: number | null, width: number) { | ||
| const frame = useRpc(frameRpc); | ||
| const rounded = time === null ? null : Math.round(time * 10) / 10; |
| } | ||
|
|
||
| export function clearFinishedTransfers(): void { | ||
| transfers.set(transfers.get().filter((transfer) => transfer.status === "running")); |
| heavy(async () => { | ||
| const result = await host.run( | ||
| ffmpeg, | ||
| ["-hide_banner", "-nostdin", "-nostats", "-i", file, "-map", "0:a:0", "-vn", "-af", "ebur128=peak=true:framelog=quiet", "-f", "null", "-"], |
| { timeoutMs: 10 * 60_000, maxBytes: 1024 * 1024 }, | ||
| ); | ||
| const measured = parseEbur128(result.stderr); | ||
| if (result.code !== 0 && measured.integrated === null) throw new Error(result.stderr.trim().split("\n").at(-1) ?? "Loudness measurement failed"); |
| child.on("close", (code) => { | ||
| clearTimeout(timer); | ||
| if (inBlock > 0) blocks.push(blockPeak); | ||
| if (blocks.length === 0) { |
Comment on lines
+247
to
+249
| await upload.host.rename(record.partPath, target); | ||
| uploads.delete(input.uploadId); | ||
| await deleteRecord(input.uploadId); |
Comment on lines
+83
to
+87
| const whole = Math.floor(safe + 1e-6); | ||
| const hms = `${pad(Math.floor(whole / 3600))}:${pad(Math.floor(whole / 60) % 60)}:${pad(whole % 60)}`; | ||
| if (fps && fps > 0) { | ||
| const nominal = Math.round(fps); | ||
| const frame = Math.min(nominal - 1, Math.floor((safe - whole) * fps + 1e-6)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Adds
plugins/media-studio: review the images, video and audio agents make, for editors who run the whole edit through Paseo agents. Requires Paseo 0.15.3+ and ffmpeg on the daemon host.preview_mediaagent tool and a card after each turn with the media the turn showed or createdsteer), kept per file and pinned on the timeline, copyable as a listalias:/path) processed by that host's ffmpegAlso adds it to the CI typecheck matrix and both READMEs, including the LLM install prompt.
Tested end to end with headless Chrome against a 0.15.3 daemon over a direct connection (local files, a Cloudflare tunnel, and an SSH GPU host). Not yet tested on real macOS / VideoToolbox hardware.