Repository navigation
feat: 新增可选的 TwelveLabs 供应商(Pegasus 视频理解 + Marengo 多模态向量)/ Add opt-in TwelveLabs provider - #15
Open
mohit-twelvelabs wants to merge 1 commit into
Open
mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
- 新增 TwelveLabsProvider(httpx,与 VectorEngineProvider 同风格) - analyze: Pegasus 视频内容理解(URL / asset_id) - embed_text / embed_video: Marengo 512 维多模态向量 - 接入 ProviderFactory、provider 校验白名单与枚举、get_models 列表 - 前端 API 密钥管理新增 twelvelabs 选项 - 单元测试(无网络)+ 受 TWELVELABS_API_KEY 控制的集成测试 - 完全可选,不配置该 Key 时系统行为不变
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
简介(中文)
为项目新增一个 完全可选 的 TwelveLabs 视频理解供应商,沿用现有的 provider 工厂模式,不改变任何默认行为——不配置 TwelveLabs API Key 时系统一切照旧。
配置方式与其它供应商一致:在“API 密钥管理”页面新增一条
provider = twelvelabs的密钥即可。免费额度可在 https://twelvelabs.io 获取。What this adds (English)
A new opt-in TwelveLabs provider that plugs into the existing
ProviderFactorypattern, mirroringVectorEngineProvider(httpx-based, samelog_provider_calldecorator and Chinese docstrings/logging style):backend/src/services/provider/twelvelabs_provider.pyanalyze(...)— Pegasus video understanding (accepts a publicvideo_urlor an uploadedasset_id; Pegasus 1.5 does not accept a barevideo_id).embed_text(...)/embed_video(...)— Marengo 512-dim multimodal embeddings.ProviderFactory.create("twelvelabs", ...), the provider validation whitelist +APIKeyProviderenum, andAPIKeyService.get_models(pegasus1.5for video,marengo3.0for embeddings).twelvelabsoption added to the API Key management dropdown / label / icon maps.Why it helps this project
AICON is a node-based AI video creation workflow. Pegasus gives the canvas a way to understand generated/imported clips (auto shot descriptions, summaries, moderation), and Marengo gives semantic asset retrieval/dedup across canvas nodes — both natural extensions of the existing storyboard/asset pipeline.
Opt-in / non-breaking
This is purely additive. No defaults change; if no TwelveLabs key is configured, behavior is identical to before. It reads its own API key from the repo's existing per-user, DB-backed API Key management (encrypted) — no key is hardcoded.
How it was tested
embedencoding, the Pegasus payload shape, and the embedding extraction (backend/tests/unit/test_twelvelabs_provider.py) — 6 passed.TWELVELABS_API_KEY(skipped when unset). I ran it against the real API and confirmed a Marengo text embedding returns a 512-dim vector — passed.api.twelvelabs.io/v1.3: Marengomarengo3.0requiresmultipart/form-data(even for text-only; raw vector key isfloat), and Pegasuspegasus1.5acceptsvideo: {type, url}withmax_tokens >= 512.black/isorton the new files only (kept the diff minimal — existing files were edited surgically, not reformatted).Note: my local macOS sandbox couldn't load
libmagic(a pre-existing dependency of an unrelated conftest import chain), so I ran the focused TwelveLabs tests with a tinymagicstub injected; they pass cleanly. The full suite should run normally in your CI/Linux env where libmagic is present.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.