StoryMind

AI Storyboard Director — From Story to Screen

Tell StoryMind what film you want to make. It plans every shot like a professional director, generates each scene with AI, and assembles the final video automatically.

Most AI video tools generate a clip from a prompt and hope it holds together.
StoryMind plans the shot list first — then generates, and stays consistent across the whole film.

Quick Start  ·  Watch Full Demos  ·  How It Works  ·  Pipelines  ·  Providers  ·  Agent Guide

---

Demo

Two films, fully planned and generated by StoryMind end to end. Click either preview to watch in full — the demo site has real seekable players, not looping previews.

"THE LAST SIGNAL" · 30s sci-fi short A lone astrophysicist in 2157 decodes an alien signal — a three-million-year-old farewell from a dead civilization. 6 shots Doubao Seedance 2.0 Piper TTS cold-blue cinematic grade

"THE NOODLE STALL" 面摊有只猫 · 2min animated fable A three-legged stray cat finds belonging at a rain-soaked noodle stall — and repays it the night a thief comes. 24 shots Ghibli-style 2D animation character consistency held across full runtime

▶ linhao-city.github.io/StoryMind — watch both, start to finish

---

What Is StoryMind?

Most AI video tools take a prompt and return a clip. StoryMind works the way a real film production does — it thinks before it shoots.

The core idea: before generating a single frame, StoryMind runs an LLM-powered Storyboard Director that breaks your script into a structured shot-by-shot plan. Each shot gets a specific scale (close-up, wide, aerial), a camera movement (dolly-in, crane, handheld), a lighting design, and exact character descriptions that carry through every scene. Only then does video generation begin — with prompts that actually communicate cinematography.

StoryMind is also a complete end-to-end production system. It can make a real video video — not just Ken Burns over stills. The agent builds a corpus from free stock footage and open archives, retrieves actual motion clips, cuts them into a timeline, and renders a finished piece. Or it generates every frame from scratch with AI video models. Or it edits your own talking-head footage. All from a plain-language prompt.

Your prompt: "A scientist decodes an alien farewell signal from 3 million years ago"
     ↓
StoryboardPlanner (Claude):
  Shot 1 — WS · static · cold-blue lab, scientist silhouetted at console
  Shot 2 — ECU · dolly-in · her eyes reflecting cascading alien symbols
  Shot 3 — MCU · handheld · hands manipulating holographic equations
  Shot 4 — LS · slow pull-back · figure alone in vast dark control room
  Shot 5 — CU · static · single tear, expression shifting to acceptance
  Shot 6 — EWS · crane-up · starfield, distant nebula, silence
     ↓
Doubao Seedance generates each shot with the exact cinematography prompt
     ↓
FFmpeg assembles, mixes narration + music, burns titles
     ↓
Final 30-second film

---

Key Features

Our Core Innovation — Storyboard Planning

• LLM Shot Decomposition — StoryboardPlanner uses Claude to turn a treatment into a structured JSON shot plan: shot scale, camera movement, lighting, emotional beat, and character anchors per shot
• Character Consistency — CharacterSheet locks each character's visual description and injects it verbatim into every shot prompt featuring that character — no drift across clips
• Cross-Shot Visual Anchors — SceneConsistencyTracker registers the color grade and lighting style from Shot 1 and propagates them across all subsequent shots
• Professional Prompt Engine — PromptEnhancer translates vague adjectives ("epic", "dramatic") into specific cinematography instructions (low-angle, slow dolly-in, deep shadows)

Production Capabilities

• 12 complete pipelines — cinematic trailers, animated explainers, documentary montages, talking heads, screen demos, character animation, podcast repurposing, dubbing, and more
• 55+ production tools — video generation, image creation, TTS, music, audio mixing, subtitles, color grading, face enhancement, scene detection, and analysis
• Real-footage documentary — builds a semantically-indexed corpus from free stock sources (Pexels, Pixabay, Unsplash) and cuts actual motion footage into a finished timeline
• Web research built in — before writing a word, the agent runs 15+ searches across news, academic sources, and video platforms to ground every video in real, current data
• No vendor lock-in — every capability has multiple provider options; a 7-dimension scoring engine picks the best match automatically
• Budget governance — cost estimate before execution, configurable spend caps, per-action approval thresholds

---

Quick Start

Prerequisites

Tool	Version	Install
Python	3.10+	python.org
Node.js	18+	nodejs.org
FFmpeg	any	see below
AI assistant	—	Claude Code, Cursor, etc.

FFmpeg by platform:

# Windows
winget install --id Gyan.FFmpeg --source winget

# macOS
brew install ffmpeg

# Ubuntu / Debian
sudo apt install ffmpeg

Install

git clone https://github.com/LinHao-city/StoryMind.git
cd StoryMind

# Linux / macOS
make setup

# Windows (no make)
pip install -r requirements.txt
cd remotion-composer && npm install && cd ..
pip install piper-tts
cp .env.example .env

Verify

python -c "
from tools.tool_registry import ToolRegistry
reg = ToolRegistry(); reg.discover()
avail = [n for n in reg.list_all() if reg.get(n).get_status().value == 'available']
print(f'{len(avail)} tools ready')
"

Configure API Keys

# .env — all optional, add what you have

# NetEase Leihuo Gateway (Doubao video/image + Mimo TTS):
LEIHUO_API_KEY=your-key
LEIHUO_BASE_URL=https://ai.leihuo.netease.com/v1
ANTHROPIC_BASE_URL=https://ai.leihuo.netease.com/
ANTHROPIC_AUTH_TOKEN=your-key

# Free stock media (sign up, no credit card):
PEXELS_API_KEY=your-key        # pexels.com/api
PIXABAY_API_KEY=your-key       # pixabay.com/api/docs
UNSPLASH_ACCESS_KEY=your-key   # unsplash.com/developers

# International providers (optional):
FAL_KEY=your-key               # fal.ai — Kling, Veo, FLUX
ELEVENLABS_API_KEY=your-key    # Premium TTS + music
OPENAI_API_KEY=your-key        # GPT Image, OpenAI TTS
HEYGEN_API_KEY=your-key        # Multi-model video gateway
RUNWAY_API_KEY=your-key        # Runway Gen-4
SUNO_API_KEY=your-key          # AI music generation

Local GPU — free video generation

pip install -r requirements-gpu.txt

# Add to .env:
VIDEO_GEN_LOCAL_ENABLED=true
VIDEO_GEN_LOCAL_MODEL=wan2.1-1.3b
# Options: wan2.1-1.3b, wan2.1-14b, hunyuan-1.5, cogvideo-5b

---

How It Works

You describe a film idea
         ↓
StoryboardPlanner (LLM)
  → structured shot plan: scale, movement, lighting, character anchors
         ↓
CharacterSheet + SceneConsistencyTracker
  → inject character descriptions + style anchors into every prompt
         ↓
PromptEnhancer
  → translate emotional language into precise cinematography instructions
         ↓
Video generation (Doubao Seedance / Kling / Veo / local GPU)
  → one clip per shot, with consistent character and color
         ↓
Audio (TTS narration + music search/generation)
         ↓
Composition (Remotion or FFmpeg)
  → titles, transitions, audio mix
         ↓
Post-review (duration, audio levels, frame check)
         ↓
Final video

Every stage is driven by a director skill — a Markdown instruction file that the agent reads before executing. You stay in control: the agent presents its plan and asks for approval at each creative decision point.

---

Try These Prompts

Open the project in your AI coding assistant and try:

"Make a 30-second cinematic sci-fi short: a lone astronaut finds an alien artifact on Mars"

"Create a 45-second product teaser for a fictional AR glasses brand called LensX"

"Make a 60-second documentary about deep-sea bioluminescent creatures, real footage only"

"Create a 30-second anime-style animation of a samurai standing in a cherry blossom storm"

"Make a 90-second explainer about how transformer models work, with narration and visuals"

---

Pipelines

Pipeline	Output	Best For
Cinematic	Trailers, mood films, teasers	Brand films, sci-fi shorts, art projects
Animated Explainer	Narrated explainer with visuals	Education, tutorials, product demos
Documentary Montage	Footage-cut montage	Video essays, stock-footage narratives
Animation	Motion graphics, kinetic titles	Social media, abstract concepts
Avatar Spokesperson	Presenter-driven video	Corporate, training, announcements
Clip Factory	Batch short-form clips	Repurposing long content
Hybrid	Source footage + AI inserts	Enhancing existing material
Screen Demo	Polished screen recording	Product walkthroughs, tutorials
Podcast Repurpose	Audiogram-style video	Podcast marketing
Talking Head	Speaker video	Presentations, interviews
Localization & Dub	Translated/dubbed video	Multi-language distribution

---

Providers

Video Generation

Provider	Type	Access
Doubao Seedance 2.0	Cloud API	Leihuo gateway
Kling	Cloud API	fal.ai
Google Veo 3	Cloud API	fal.ai / HeyGen
Runway Gen-4	Cloud API	Direct / HeyGen
MiniMax	Cloud API	fal.ai
HeyGen	Cloud API	Multi-model gateway
WAN 2.1	Local GPU	Free
Hunyuan	Local GPU	Free
CogVideo	Local GPU	Free
Pexels	Stock	Free key
Pixabay	Stock	Free key

Image Generation

Provider	Type	Access
Doubao SeeDream 5.0	Cloud API	Leihuo gateway
FLUX	Cloud API	fal.ai
gpt-image-2 / DALL-E 3	Cloud API	OpenAI / Leihuo
Google Imagen	Cloud API	fal.ai
Recraft	Cloud API	fal.ai
Pexels / Pixabay / Unsplash	Stock	Free key
Local Diffusion	Local GPU	Free

TTS & Audio

Provider	Type	Cost
Mimo TTS	Cloud API	Free (Leihuo)
Piper	Local	Free, offline
ElevenLabs	Cloud API	Paid
OpenAI TTS	Cloud API	Paid
MiniMax Speech	Cloud API	Leihuo gateway

Music: Pixabay (free search), Suno AI, ElevenLabs Music

---

Upgrade Roadmap

See UPGRADE_PROGRESS.md for the full roadmap.

Phase	Status	What
Phase 1	✅ Done	LLM storyboard planning, character sheet, cinematography director skill
Phase 2	✅ Done	Cross-shot consistency tracker, prompt enhancer
Phase 3	🔲 Planned	InstantID face-lock post-processor (GPU required)

---

Architecture

StoryMind/
├── tools/
│   ├── planning/       # StoryboardPlanner, CharacterSheet, SceneConsistencyTracker, PromptEnhancer
│   ├── video/          # Video generation (Doubao Seedance, Kling, Pexels, etc.)
│   ├── audio/          # TTS (Mimo, Piper), music, mixing
│   ├── graphics/       # Image generation (SeeDream, FLUX, Pexels)
│   ├── enhancement/    # Color grade, face enhance, InstantID (Phase 3)
│   └── analysis/       # Scene detect, frame sampling, transcription
├── pipeline_defs/      # YAML pipeline manifests (v3.0)
├── skills/
│   ├── core/           # cinematography-director, remotion, color-grading
│   └── pipelines/      # Per-pipeline stage director skills
├── remotion-composer/  # React/Remotion video composition engine
├── schemas/            # JSON schema validation
└── UPGRADE_PROGRESS.md

---

Contributing

1. Add a tool: create a Python file in tools/, inherit from BaseTool — auto-discovered, no registration needed
2. Add a pipeline: create a YAML manifest in pipeline_defs/ + stage skills in skills/pipelines/
3. See AGENT_GUIDE.md for the full agent contract

---

Testing

# Linux / macOS
make test-contracts
make test

# Windows
python -m pytest tests/contracts/
python -m pytest tests/

---

License

GNU AGPLv3

---

Contact

Linhao · City University of Hong Kong (Dongguan)
📧 72510916@cityu-dg.edu.cn
🐙 github.com/LinHao-city

---

StoryMind — Plan the shot. Generate the scene. Tell the story.