Skip to content

feat(gpt): 接入 TwelveLabs Pegasus 视频理解后端(可选)#418

Open
mohit-twelvelabs wants to merge 2 commits into
JefferyHcool:developfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat(gpt): 接入 TwelveLabs Pegasus 视频理解后端(可选)#418
mohit-twelvelabs wants to merge 2 commits into
JefferyHcool:developfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

中文摘要

为 BiliNote 新增一个可选的 GPT 后端:当供应商 typetwelvelabs 时,用 TwelveLabsPegasus 视频理解模型直接「看」视频,从画面 + 语音两路信息生成笔记,而不只是总结转写文本。对演示、操作录屏、图表讲解类视频尤其有用——画面里常常藏着转写拿不到的信息。

  • 完全可选、不改默认行为GPTFactory.from_config 仅在 type == "twelvelabs" 时路由到新后端,其余供应商一字未改;不配置 TwelveLabs 供应商时整条链路与原来完全一致。
  • API Key 从供应商配置读取,不硬编码、不入库到仓库
  • 关键文件:backend/app/gpt/twelvelabs_gpt.py(新后端)、backend/app/gpt/gpt_factory.py(路由)、backend/app/models/gpt_model.pyGPTSource 加可选 video_url)、backend/app/services/note.py(把 URL 传给 source)、backend/requirements.txtdoc/twelvelabs.md
  • 已知限制:Pegasus 需要可公开访问的视频 URL;本地文件/纯音频任务会优雅报错并提示改用常规文本模型。详见 doc/twelvelabs.md

可在 https://twelvelabs.io 免费申请 API Key —— 有较慷慨的免费额度。


English

This adds an optional GPT backend that uses TwelveLabs Pegasus video understanding to generate notes directly from a video's visual + audio content, instead of summarizing only the transcript. Great for demos, screencasts, and slide/diagram-heavy videos where the screen carries information the transcript misses.

What it adds

  • A new TwelveLabsGPT backend in backend/app/gpt/twelvelabs_gpt.py implementing the existing GPT.summarize interface.
  • GPTFactory.from_config routes to it only when a provider's type == "twelvelabs". Every other provider path is untouched.
  • GPTSource gains an optional video_url (additive, defaults None); NoteGenerator passes the source URL through. Pegasus watches the video URL directly.
  • Reuses the existing note prompt (generate_base_prompt) so style/format/tag options still apply.

Why it helps this project

BiliNote already markets "多模态视频理解"; this gives users a true video-understanding path: the model sees the video rather than relying on transcription quality, which matters for visual-heavy content.

Opt-in / non-breaking

If no TwelveLabs provider is configured, behavior is identical to before. No defaults change.

How it was tested

  • No-network unit tests (backend/tests/test_twelvelabs_gpt.py, stub-based to match the repo's existing test style): verify the factory routing, that summarize calls Pegasus with the video URL and max_tokens >= 512, and that a missing URL / empty key raises clearly. Run with the repo's existing pytest; existing GPT tests still pass (no regression).
  • Live contract verification against the real TwelveLabs API: confirmed analyze(model_name="pegasus1.5"/"pegasus1.2", video=VideoContext_Url(...) | videoId, prompt=..., max_tokens=...) returns Markdown notes synthesized from both visual and audio tracks. Also confirmed the API's max_tokens >= 512 constraint (a 400 parameter_invalid otherwise), which the implementation enforces via _MIN_MAX_TOKENS.
  • python -m py_compile passes on all changed backend files.

Known limitations (stated honestly)

  • Pegasus needs a publicly reachable video URL; local-file / audio-only tasks have no URL and the backend raises a clear message asking the user to use a regular text model. (Some sample-host MP4s are rejected by Pegasus as video_file_broken due to container/codec; real platform CDN URLs from Bilibili/YouTube are fine.)
  • max_tokens floor of 512; analyzed window must be ≥ 4s.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Note on branch target

Per CONTRIBUTING.md, features target develop. This PR is opened against develop. Happy to adjust scope/conventions per maintainer preference.

huangjianwu and others added 2 commits June 23, 2026 11:46
新增可选 GPT 后端,供应商 type=twelvelabs 时用 Pegasus 直接观看视频,
从画面+语音生成笔记,而非仅总结转写文本。未配置时链路行为不变。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant