Professional multi-track editing in the browser, powered by an asynchronous FFmpeg rendering engine and extensible multimodal AI.
Key Capabilities β’ Why FrameFlow? β’ System Architecture β’ Timeline AST Spec β’ Quickstart β’ Milestones
Modern video creation tooling is fundamentally divided between two flawed extremes:
- Legacy Desktop NLEs (Premiere, Final Cut, DaVinci): Heavy desktop installations, proprietary closed binary project files, no collaborative cloud primitives, and zero programmatic web APIs.
- Black-Box AI Video Generators: Prompt in, un-editable compressed video out. If a single word or cut is wrong, the creator is forced to re-prompt from scratch with zero granular control.
"With AI removed, FrameFlow is still a lightning-fast, frame-accurate, professional video editor. With AI added, you gain an autonomous assistant editor that drafts cuts directly onto your timeline β keeping you in full creative control."
Every edit, transition, and asset remains an editable, non-destructive node on the timeline that can be adjusted, re-arranged, or undone in a single keystroke.
- Independent Video, Audio & Text Tracks: Overlay dynamic titles, background music, sound effects, and B-roll seamlessly.
-
Frame-Accurate Trimming & Splitting: Non-destructively adjust clip in/out boundaries or slice clips at the playhead position (
Skey). - Magnetic Snapping Engine: Snaps clips dynamically to track bounds, adjacent cut points, and the playhead with sub-pixel precision.
-
Smooth Timecode Scrubbing & Zoom: Dynamic zoom levels (
$10\text{px/s} \to 160\text{px/s}$ ) with responsive playhead scrubbing and frame-accurate proxy preview.
- Edits never mutate timeline state directly. All actions are dispatched as atomic command objects:
AddClipCommand,MoveClipCommand,TrimClipCommand,SplitClipCommand,DeleteClipCommand
- Atomic Composite Batches: Multi-action operations (such as an automated AI rough cut) are batched via
CompositeCommand, allowing a 50-cut edit sequence to be rolled forward or backward in a single click (Ctrl+Z/Ctrl+Y).
- The timeline is an open, serializable JSON document (
TimelineDocument) acting as the single source of truth. - Automatic non-blocking background debounced autosave.
- Milestone Snapshots: Create named release checkpoints (e.g. "Director's Cut v1", "Client Review 2") and restore any previous version instantly.
- Resumable Chunked Multipart Uploads: Upload hours of raw 4K/1080p footage directly to MinIO in parallel 5MB chunks.
- Worker Media Extraction: Automated
ffprobeinspection extracts resolution, duration, framerate, and audio/video codecs. - Faststart Proxy Generation: Generates 720p H.264 web proxies and 100-slice visual scrubbing thumbnail strips in background BullMQ workers.
- Pure TypeScript compiler converts complex multi-track timelines into an optimized FFmpeg
filter_complexexecution graph:- Base solid canvas generated to exact project resolution and FPS
- Dynamic aspect-ratio scaling with automatic letterbox/pillarbox padding
- Overlays bound to timeline time ranges via
enable='between(t,start,end)' - Cross-platform font resolution for title/text overlays
- Sub-millisecond audio offset delay matching (
adelay) and mixing (amixwith silence fallback)
- Export Presets: Crisp 1080p (
1080p_h264, 5 Mbps) and fast 720p (720p_h264, 2.5 Mbps) with faststart container flags.
- Redis Pub/Sub backplane connected to Fastify WebSocket gateway (
/ws). - Streams real-time progress percentages (0%
$\to$ 100%) for both asset ingestion and video rendering directly to the user's browser.
FrameFlow is architected with strict decoupling: the API server manages business logic and control plane traffic, while distributed BullMQ worker processes manage compute-intensive FFmpeg data pipelines.
+----------------------------------+
| Next.js 14 Frontend Studio UI |
| (Zustand + React 18 + HTML5) |
+-----------------+----------------+
|
HTTP REST API / | WebSocket (/ws)
Multipart Upload | Live Progress
v
+----------------------------------+
| Fastify API Gateway (Node) |
| Auth | Projects | Renders API |
+--------+----------------+--------+
| |
Enqueue Job via | | Database Queries
BullMQ Queue | | (Prisma ORM)
v v
+------------+----+ +----+------------+
| Redis Engine | | PostgreSQL |
| Queue & PubSub | | Relational DB |
+------------+----+ +-----------------+
|
Worker consumes | Redis PubSub
job execution | progress broadcast
v
+------------+-------------------------+
| BullMQ Background Worker Mesh |
| - Chunked Video Ingestion & ffprobe |
| - 720p Proxy & Thumbnail Generator |
| - FFmpeg filter_complex Compiler |
| - Multimodal AI Analysis & Cuts |
+--------------------+-----------------+
|
| Streams chunks &
| exports rendered MP4
v
+------------+------------+
| MinIO Object Storage |
| (S3-compatible storage) |
+-------------------------+
frameflow/
βββ client/ # Next.js 14 Frontend (App Router)
β βββ src/app/ # /login, /register, /projects, /editor/[id]
β βββ src/components/
β β βββ timeline/ # Ruler, Tracks, Clips, Playhead, Snapping
β β βββ preview/ # Synchronized HTML5 Proxy Preview Player
β β βββ media/ # Media Bin, Chunked Uploader, Asset Cards
β β βββ export/ # Render Export Modal & Live Progress Bar
β βββ src/store/ # Zustand Timeline & Command State Store
βββ server/
β βββ api/ # Fastify REST API & WebSocket Gateway
β β βββ src/modules/ # Auth, Projects, Assets, Timeline, Renders
β β βββ src/gateway/ # WebSocket (/ws) Redis Pub/Sub gateway
β βββ worker/ # Asynchronous Media & Render Engine
β β βββ src/processors/ # media.processor.ts & render.processor.ts
β β βββ src/index.ts # Worker pool entrypoint
β βββ database/ # PostgreSQL Database & Persistence
β βββ prisma/schema.prisma# Prisma Schema (Users, Projects, Renders, etc.)
βββ packages/
β βββ shared/ # Isomorphic Shared Library
β βββ src/timeline/ # TimelineDocument AST & validation
β βββ src/commands/ # Command pattern implementations & history
β βββ src/compiler/ # Timeline-to-FFmpeg filtergraph compiler
β βββ src/events/ # Distributed job event definitions
βββ docker-compose.yml # Local infra: PostgreSQL, Redis, MinIO
βββ scripts/ # End-to-end verification and benchmark suites
The timeline is stored as a deterministic, version-controlled JSON schema:
{
"id": "tl-9a4f21",
"projectId": "proj-3810f2",
"version": 4,
"fps": 30,
"duration": 45.0,
"tracks": [
{
"id": "track-v1",
"type": "VIDEO",
"name": "Main Footage",
"locked": false,
"muted": false,
"clips": [
{
"id": "clip-101",
"assetId": "asset-raw-4k",
"name": "Interview A-Roll",
"start": 0.0,
"duration": 15.0,
"inPoint": 2.5,
"outPoint": 17.5,
"speed": 1.0,
"proxyUrl": "/media/proxies/asset-raw-4k.mp4"
}
]
},
{
"id": "track-t1",
"type": "TEXT",
"name": "Lower Thirds",
"locked": false,
"muted": false,
"clips": [
{
"id": "clip-201",
"assetId": null,
"name": "Speaker Name Title",
"start": 1.0,
"duration": 4.0,
"inPoint": 0.0,
"outPoint": 4.0,
"speed": 1.0,
"textConfig": {
"text": "Dr. Alex Vance β Lead Researcher",
"fontSize": 42,
"color": "#ffffff",
"position": { "x": 50, "y": 85 }
}
}
]
}
],
"markers": []
}- Node.js: v20.x or higher
- Docker & Docker Compose: For PostgreSQL, Redis, and MinIO
- FFmpeg & ffprobe: Installed in your system environment (
ffmpeg -version)
git clone https://github.com/yasirraeesit/frameflow.git
cd frameflowcp .env.example .envdocker compose up -dService health check:
- PostgreSQL:
localhost:5432 - Redis Engine:
localhost:6379 - MinIO Console: http://localhost:9001 (
minioadmin/minioadmin)
npm install
npm --workspace=@frameflow/database run db:migrateStart the API, Worker, and Web Studio concurrently:
# Terminal 1: Fastify REST API
npm --workspace=@frameflow/api run dev
# Terminal 2: BullMQ Media & Render Worker
npm --workspace=@frameflow/worker run dev
# Terminal 3: Next.js Studio Frontend
npm --workspace=@frameflow/web run devNavigate to http://localhost:3000 to open the FrameFlow Studio.
FrameFlow enforces strict type safety (zero any) and comprehensive test coverage across every package:
# Run unit & schema tests across all packages
npm --workspace=@frameflow/shared test
npm --workspace=@frameflow/api test
npm --workspace=@frameflow/worker test
# Run the complete Phase 3 End-to-End Rendering Pipeline verification suite
npx tsx --env-file=.env scripts/test-phase3-e2e.tsThe E2E suite validates:
- User registration and project creation
- Real-time WebSocket gateway connection and channel subscription
- Parallel multipart chunked video ingestion
- Background ffprobe extraction and 720p proxy generation
- Multi-track timeline assembly (Video + Audio + Text overlays)
- Asynchronous 1080p render job dispatch
- BullMQ worker execution and live progress broadcasting (0%
$\to$ 100%) - Direct inspection of the rendered MP4 via
ffprobe(dimensions, codecs, bitrates, audio channels) - Mid-flight render cancellation and cleanup
| Milestone | Scope | Status | Tag |
|---|---|---|---|
| Phase 1: Foundation | Monorepo scaffolding, JWT auth, projects CRUD, chunked video uploads, worker ffprobe metadata, 720p proxies, thumbnail strips, WebSocket progress. | β Complete | phase-1-complete |
| Phase 2: Timeline Editor | Multi-track timeline UI, serializable JSON AST, non-destructive Command Pattern (unlimited undo/redo), PostgreSQL autosave & version snapshots, synchronized proxy player. | β Complete | phase-2-complete |
| Phase 3: Rendering Pipeline | Timeline-to-FFmpeg filter_complex compiler, BullMQ render workers, export presets (1080p/720p), job cancellation, client export modal with live download. |
β Complete | phase-3-complete |
| Phase 4: AI Analysis | FFmpeg scene detection, audio extraction & Whisper transcription, swappable multimodal AI adapter (Gemini / Claude / Mock), importance scoring, natural language search. | π In Progress | main |
| Phase 5: AI-Assisted Assembly | Natural language timeline prompts ("create a 60s highlight reel"), atomic command emission, diff preview with accept/reject, track locking preservation. | β³ Planned | β |
| Phase 6: Production Hardening | Rate limiting, signed URLs, structured logging, health checks, queue latency metrics, benchmarks documentation (docs/benchmarks.md). |
β³ Planned | β |
We welcome contributions from developers, video creators, and AI researchers!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'feat: Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE for more information.