feat(video): opt-in TwelveLabs Pegasus scene analysis for generated clips - #38
Open
mohit-twelvelabs wants to merge 1 commit into
Open
feat(video): opt-in TwelveLabs Pegasus scene analysis for generated clips#38mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Augment the storyboard/synthesis pipeline with an optional natural-language
scene description of each generated video clip via TwelveLabs Pegasus 1.5.
- New PegasusSceneAnalyzer adapter (src/models/twelvelabs.py) following the
existing model-adapter conventions: lazy env credentials
(TWELVELABS_API_KEY), endpoint via the shared {PROVIDER}_BASE_URL registry,
no new dependency (uses requests).
- Opt-in hook in VideoGenerator.generate_clip: when a key is configured and
the clip has a public (OSS-signed) URL, attach the description to a new
StoryboardFrame.scene_analysis field. Fully non-breaking and no-op without
a key; any error is swallowed so analysis never affects generation.
- Tests (no-network unit + key-gated live), .env.example + README config-mode
row (EN + 中文).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
中文摘要
本 PR 为 Studio 分镜/合成流程新增可选的 TwelveLabs Pegasus 1.5 场景分析:当配置了
TWELVELABS_API_KEY时,每个生成的视频片段会被 Pegasus 描述(场景 / 氛围 / 关键动作),结果写入新增的StoryboardFrame.scene_analysis字段,可用于质检("画面是否符合提示词?")与自动字幕。src/models/qwen_vl.py):env 懒加载凭证、{PROVIDER}_BASE_URL端点注册表、不新增依赖(复用requests)。可在 https://twelvelabs.io 免费获取 API key,有慷慨的免费额度。
What this adds (English)
An opt-in TwelveLabs Pegasus 1.5 scene-analysis step in the Studio storyboard/synthesis pipeline. When
TWELVELABS_API_KEYis configured, each generated I2V clip is described by Pegasus (setting / mood / key action) and the result is attached to a newStoryboardFrame.scene_analysisfield — useful for QA ("did the shot match the prompt?") and auto-captioning, closing the loop on the generation step.Why it helps this project
The pipeline already produces a clip per storyboard frame (
src/apps/comic_gen/video.py) but has no way to verify what was actually rendered. Pegasus turns the rendered clip back into text, giving reviewers a fast, model-agnostic QA signal and a caption source without changing any default behavior.Opt-in & non-breaking
TWELVELABS_API_KEYis set — behavior is bit-for-bit unchanged otherwise.{PROVIDER}_BASE_URLendpoint registry, and no new dependency (uses the already-requiredrequests).sign_url_for_api); local-only clips are skipped with a log line.Changes
src/models/twelvelabs.py— newPegasusSceneAnalyzeradapter (start async/analyzetask → poll → returndatatext).src/apps/comic_gen/video.py— opt-in hook ingenerate_clip+_analyze_scenehelper.src/apps/comic_gen/models.py— new optionalStoryboardFrame.scene_analysisfield.src/utils/endpoints.py—TWELVELABSdefault base URL..env.example,README.md,README_EN.md— optional config-mode row + env var.tests/test_twelvelabs_scene_analysis.py— no-network unit tests + aTWELVELABS_API_KEY-gated live test.How it was tested
/analyzetask → poll → a non-empty scene description came back (e.g. "a lush green forest with a large tree… serene and natural… sunlight filtering through the leaves"). This confirms the route/auth (x-api-key) and thedataresponse shape the adapter implements.black --checkandflake8are clean on the added files (the analyzer follows the repo's 100-col style).I ran the repo's Python tests for the new module, but did not run the full
build_mac.sh/frontend build (no UI change here; the field is purely additive). Happy to follow the/lumenx-model-onboardingworkflow or adjust placement if maintainers prefer the hook live elsewhere in the pipeline.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.