Skip to content

feat: 新增可选的 TwelveLabs 供应商(Pegasus 视频理解 + Marengo 多模态向量)/ Add opt-in TwelveLabs provider - #15

Open
mohit-twelvelabs wants to merge 1 commit into
869413421:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

mohit-twelvelabs wants to merge 1 commit into
869413421:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

简介(中文)

为项目新增一个 完全可选 的 TwelveLabs 视频理解供应商,沿用现有的 provider 工厂模式,不改变任何默认行为——不配置 TwelveLabs API Key 时系统一切照旧。

  • Pegasus(视频理解 / analyze):对成片或素材做内容分析,可用于自动生成分镜描述、镜头摘要、内容审核等创作辅助。
  • Marengo(多模态向量 / embed):为视频与文本生成 512 维向量,可用于素材库语义检索、画布节点之间的素材匹配与去重。

配置方式与其它供应商一致:在“API 密钥管理”页面新增一条 provider = twelvelabs 的密钥即可。免费额度可在 https://twelvelabs.io 获取。

What this adds (English)

A new opt-in TwelveLabs provider that plugs into the existing ProviderFactory pattern, mirroring VectorEngineProvider (httpx-based, same log_provider_call decorator and Chinese docstrings/logging style):

  • backend/src/services/provider/twelvelabs_provider.py
    • analyze(...) — Pegasus video understanding (accepts a public video_url or an uploaded asset_id; Pegasus 1.5 does not accept a bare video_id).
    • embed_text(...) / embed_video(...) — Marengo 512-dim multimodal embeddings.
  • Wired into ProviderFactory.create("twelvelabs", ...), the provider validation whitelist + APIKeyProvider enum, and APIKeyService.get_models (pegasus1.5 for video, marengo3.0 for embeddings).
  • Frontend: twelvelabs option added to the API Key management dropdown / label / icon maps.

Why it helps this project

AICON is a node-based AI video creation workflow. Pegasus gives the canvas a way to understand generated/imported clips (auto shot descriptions, summaries, moderation), and Marengo gives semantic asset retrieval/dedup across canvas nodes — both natural extensions of the existing storyboard/asset pipeline.

Opt-in / non-breaking

This is purely additive. No defaults change; if no TwelveLabs key is configured, behavior is identical to before. It reads its own API key from the repo's existing per-user, DB-backed API Key management (encrypted) — no key is hardcoded.

How it was tested

  • No-network unit tests for the factory wiring, the multipart embed encoding, the Pegasus payload shape, and the embedding extraction (backend/tests/unit/test_twelvelabs_provider.py) — 6 passed.
  • A live integration test gated on TWELVELABS_API_KEY (skipped when unset). I ran it against the real API and confirmed a Marengo text embedding returns a 512-dim vector — passed.
  • I live-verified both REST contracts against api.twelvelabs.io/v1.3: Marengo marengo3.0 requires multipart/form-data (even for text-only; raw vector key is float), and Pegasus pegasus1.5 accepts video: {type, url} with max_tokens >= 512.
  • Ran black / isort on the new files only (kept the diff minimal — existing files were edited surgically, not reformatted).

Note: my local macOS sandbox couldn't load libmagic (a pre-existing dependency of an unrelated conftest import chain), so I ran the focused TwelveLabs tests with a tiny magic stub injected; they pass cleanly. The full suite should run normally in your CI/Linux env where libmagic is present.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

- 新增 TwelveLabsProvider(httpx,与 VectorEngineProvider 同风格)
  - analyze: Pegasus 视频内容理解(URL / asset_id)
  - embed_text / embed_video: Marengo 512 维多模态向量
- 接入 ProviderFactory、provider 校验白名单与枚举、get_models 列表
- 前端 API 密钥管理新增 twelvelabs 选项
- 单元测试(无网络)+ 受 TWELVELABS_API_KEY 控制的集成测试
- 完全可选,不配置该 Key 时系统行为不变
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant