Skip to content

feat(video): opt-in TwelveLabs Pegasus scene analysis for generated clips - #38

Open
mohit-twelvelabs wants to merge 1 commit into
alibaba:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat(video): opt-in TwelveLabs Pegasus scene analysis for generated clips#38
mohit-twelvelabs wants to merge 1 commit into
alibaba:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

中文摘要

本 PR 为 Studio 分镜/合成流程新增可选的 TwelveLabs Pegasus 1.5 场景分析:当配置了 TWELVELABS_API_KEY 时,每个生成的视频片段会被 Pegasus 描述(场景 / 氛围 / 关键动作),结果写入新增的 StoryboardFrame.scene_analysis 字段,可用于质检("画面是否符合提示词?")与自动字幕。

  • 完全可选、不改变默认行为:未配置 key 时为空操作,流程与之前逐位一致;任何错误都会被吞掉,绝不影响视频生成或 frame 状态。
  • 遵循现有 provider/adapter 约定(参考 src/models/qwen_vl.py):env 懒加载凭证、{PROVIDER}_BASE_URL 端点注册表、不新增依赖(复用 requests)。
  • Pegasus 需要公网 URL,因此仅在片段已镜像到 OSS(object key → 签名 URL)时运行。

可在 https://twelvelabs.io 免费获取 API key,有慷慨的免费额度。


What this adds (English)

An opt-in TwelveLabs Pegasus 1.5 scene-analysis step in the Studio storyboard/synthesis pipeline. When TWELVELABS_API_KEY is configured, each generated I2V clip is described by Pegasus (setting / mood / key action) and the result is attached to a new StoryboardFrame.scene_analysis field — useful for QA ("did the shot match the prompt?") and auto-captioning, closing the loop on the generation step.

Why it helps this project

The pipeline already produces a clip per storyboard frame (src/apps/comic_gen/video.py) but has no way to verify what was actually rendered. Pegasus turns the rendered clip back into text, giving reviewers a fast, model-agnostic QA signal and a caption source without changing any default behavior.

Opt-in & non-breaking

  • No-op unless TWELVELABS_API_KEY is set — behavior is bit-for-bit unchanged otherwise.
  • Constructed cheaply; does no network/credential work until invoked.
  • The whole analysis path is wrapped so any failure is logged and swallowed — it can never fail or alter video generation or frame status.
  • Follows existing conventions: lazy env credentials, the shared {PROVIDER}_BASE_URL endpoint registry, and no new dependency (uses the already-required requests).
  • Pegasus needs a public URL, so analysis only runs when the clip was mirrored to OSS (object key → signed URL via the existing sign_url_for_api); local-only clips are skipped with a log line.

Changes

  • src/models/twelvelabs.py — new PegasusSceneAnalyzer adapter (start async /analyze task → poll → return data text).
  • src/apps/comic_gen/video.py — opt-in hook in generate_clip + _analyze_scene helper.
  • src/apps/comic_gen/models.py — new optional StoryboardFrame.scene_analysis field.
  • src/utils/endpoints.pyTWELVELABS default base URL.
  • .env.example, README.md, README_EN.md — optional config-mode row + env var.
  • tests/test_twelvelabs_scene_analysis.py — no-network unit tests + a TWELVELABS_API_KEY-gated live test.

How it was tested

  • Focused suite passes locally (Python 3.12): 5 passed, 1 skipped offline; the live test passes with a real key.
  • Live-verified the Pegasus 1.5 contract end-to-end against the real API: a real clip → async /analyze task → poll → a non-empty scene description came back (e.g. "a lush green forest with a large tree… serene and natural… sunlight filtering through the leaves"). This confirms the route/auth (x-api-key) and the data response shape the adapter implements.
  • black --check and flake8 are clean on the added files (the analyzer follows the repo's 100-col style).

I ran the repo's Python tests for the new module, but did not run the full build_mac.sh/frontend build (no UI change here; the field is purely additive). Happy to follow the /lumenx-model-onboarding workflow or adjust placement if maintainers prefer the hook live elsewhere in the pipeline.

Note on contribution process: this targets an Alibaba-owned repo, which may require a CLA — happy to sign whatever's needed. Flagging it honestly so it's not a surprise during review.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Augment the storyboard/synthesis pipeline with an optional natural-language
scene description of each generated video clip via TwelveLabs Pegasus 1.5.

- New PegasusSceneAnalyzer adapter (src/models/twelvelabs.py) following the
  existing model-adapter conventions: lazy env credentials
  (TWELVELABS_API_KEY), endpoint via the shared {PROVIDER}_BASE_URL registry,
  no new dependency (uses requests).
- Opt-in hook in VideoGenerator.generate_clip: when a key is configured and
  the clip has a public (OSS-signed) URL, attach the description to a new
  StoryboardFrame.scene_analysis field. Fully non-breaking and no-op without
  a key; any error is swallowed so analysis never affects generation.
- Tests (no-network unit + key-gated live), .env.example + README config-mode
  row (EN + 中文).
@CLAassistant

CLAassistant commented Jun 27, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants