feat(gpt): 接入 TwelveLabs Pegasus 视频理解后端(可选)#418
Open
mohit-twelvelabs wants to merge 2 commits into
Open
Conversation
新增可选 GPT 后端,供应商 type=twelvelabs 时用 Pegasus 直接观看视频, 从画面+语音生成笔记,而非仅总结转写文本。未配置时链路行为不变。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
中文摘要
为 BiliNote 新增一个可选的 GPT 后端:当供应商
type为twelvelabs时,用 TwelveLabs 的 Pegasus 视频理解模型直接「看」视频,从画面 + 语音两路信息生成笔记,而不只是总结转写文本。对演示、操作录屏、图表讲解类视频尤其有用——画面里常常藏着转写拿不到的信息。GPTFactory.from_config仅在type == "twelvelabs"时路由到新后端,其余供应商一字未改;不配置 TwelveLabs 供应商时整条链路与原来完全一致。backend/app/gpt/twelvelabs_gpt.py(新后端)、backend/app/gpt/gpt_factory.py(路由)、backend/app/models/gpt_model.py(GPTSource加可选video_url)、backend/app/services/note.py(把 URL 传给 source)、backend/requirements.txt、doc/twelvelabs.md。可在 https://twelvelabs.io 免费申请 API Key —— 有较慷慨的免费额度。
English
This adds an optional GPT backend that uses TwelveLabs Pegasus video understanding to generate notes directly from a video's visual + audio content, instead of summarizing only the transcript. Great for demos, screencasts, and slide/diagram-heavy videos where the screen carries information the transcript misses.
What it adds
TwelveLabsGPTbackend inbackend/app/gpt/twelvelabs_gpt.pyimplementing the existingGPT.summarizeinterface.GPTFactory.from_configroutes to it only when a provider'stype == "twelvelabs". Every other provider path is untouched.GPTSourcegains an optionalvideo_url(additive, defaultsNone);NoteGeneratorpasses the source URL through. Pegasus watches the video URL directly.generate_base_prompt) so style/format/tag options still apply.Why it helps this project
BiliNote already markets "多模态视频理解"; this gives users a true video-understanding path: the model sees the video rather than relying on transcription quality, which matters for visual-heavy content.
Opt-in / non-breaking
If no TwelveLabs provider is configured, behavior is identical to before. No defaults change.
How it was tested
backend/tests/test_twelvelabs_gpt.py, stub-based to match the repo's existing test style): verify the factory routing, thatsummarizecalls Pegasus with the video URL andmax_tokens >= 512, and that a missing URL / empty key raises clearly. Run with the repo's existingpytest; existing GPT tests still pass (no regression).analyze(model_name="pegasus1.5"/"pegasus1.2", video=VideoContext_Url(...) | videoId, prompt=..., max_tokens=...)returns Markdown notes synthesized from both visual and audio tracks. Also confirmed the API'smax_tokens >= 512constraint (a 400parameter_invalidotherwise), which the implementation enforces via_MIN_MAX_TOKENS.python -m py_compilepasses on all changed backend files.Known limitations (stated honestly)
video_file_brokendue to container/codec; real platform CDN URLs from Bilibili/YouTube are fine.)max_tokensfloor of 512; analyzed window must be ≥ 4s.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.
Note on branch target
Per
CONTRIBUTING.md, features targetdevelop. This PR is opened againstdevelop. Happy to adjust scope/conventions per maintainer preference.