A FastMCP stdio server for video extraction and understanding with MiniCPM-V.

Supports:

• local inference (same host runs model)
• remote inference (MCP host uploads video to a remote HTTP worker with GPU)
• segmented video understanding for long videos (default 60s shards)
• ModelScope-first model download in CN with fallback to HuggingFace
• Linux+CUDA optional vLLM acceleration (OpenAI-compatible endpoint)
• overload / OOM guardrails

• Video metadata extraction via ffprobe
• Keyframe sampling (interval, scene-cut, explicit timestamps)
• Clip extraction with optional re-encode fallback
• Video understanding powered by MiniCPM-V-4_5
• Native video-input understanding (MiniCPM-V) with automatic frame fallback
• Auto-priority Q4 quantization on CUDA (MINICPMV_QUANT_MODE=auto)
• Backend mode: auto | local | remote
• Segment-mode analysis for long videos + post-segment global synthesis
• Key moments with timestamps, keyframe export, and optional key clips
• Capacity check (GPU/VRAM) before inference
• Best-effort OOM cleanup + degraded retry
• Designed for local MCP clients and automation workflows

• video_detect_backend(model_id='', device='auto', min_vram_gb=18.0, allow_cpu_fallback=false)
• video_probe(path)
• video_detect_scenes(path, scene_threshold=0.35, min_gap_sec=0.8, max_scenes=40)
• video_extract_frames(path, mode='scene', interval_sec=2.0, timestamps_csv='', scene_threshold=0.35, min_gap_sec=0.8, max_frames=12, output_dir='', prefix='frame')
• video_extract_clip(path, start_sec, end_sec, output_path='', reencode=False)
• video_understand(path, question='Summarize this video', sample_mode='scene', ..., backend_mode='auto', remote_url='', segment_sec=60, prefer_native_video=True, integrate_segments=True, keyframe_count=6, export_key_clips=False, key_clip_sec=6.0, dry_run=False)

• Python 3.10+
• uv
• ffmpeg and ffprobe in PATH
• GPU recommended for MiniCPM-V inference

uv sync

For ModelScope download path, install modelscope:

uv pip install modelscope

uv run minicpmv-video-mcp

Run a remote HTTP worker on your Ubuntu GPU server:

uv run minicpmv-video-remote-http

Default endpoint:

• POST /v1/video/understand
• GET /health

Example (server env):

export MINICPMV_DEVICE=cuda
export MINICPMV_MIN_VRAM_GB=18
export MINICPMV_REMOTE_MAX_CONCURRENCY=1
export MINICPMV_REMOTE_TOKEN='your-strong-token'
uv run minicpmv-video-remote-http

Then on MCP host, call:

• video_understand(..., backend_mode='remote', remote_url='http://192.168.31.18:8788')

Serve MiniCPM-V with vLLM on the inference host (recommended Q4/AWQ model):

vllm serve openbmb/MiniCPM-V-4_5-AWQ \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192

Then set MCP-side env:

export MINICPMV_VLLM_MODE=auto
export MINICPMV_VLLM_BASE_URL=http://127.0.0.1:8000/v1
export MINICPMV_VLLM_MODEL_ID=openbmb/MiniCPM-V-4_5-AWQ

When MINICPMV_VLLM_MODE=auto, the server uses vLLM only if current host is Linux and CUDA is available; otherwise it falls back to local MiniCPM inference.

[mcp_servers.minicpmv_video]
command = "uv"
args = ["run", "--project", "D:/path/to/mcp-minicpmv-video", "minicpmv-video-mcp"]
enabled = true
startup_timeout_sec = 60
tool_timeout_sec = 300

• MINICPMV_MODEL_ID (default: openbmb/MiniCPM-V-4_5)
• MINICPMV_Q4_MODEL_ID (default: openbmb/MiniCPM-V-4_5-AWQ)
• MINICPMV_DEVICE (auto|cuda|mps|cpu)
• MINICPMV_DTYPE (auto|bfloat16|float16|float32)
• MINICPMV_ATTN_IMPL (default: sdpa)
• MINICPMV_QUANT_MODE (auto|none|4bit|8bit, default: auto, CUDA-only)
• MINICPMV_PREFER_Q4 (default: true)
• MINICPMV_MODEL_SOURCE (auto|modelscope|huggingface, default: auto)
• MINICPMV_PREFER_MODELSCOPE (default: true)
• MINICPMV_MODELSCOPE_MODEL_ID (optional override)
• MINICPMV_MODELSCOPE_Q4_MODEL_ID (optional override)
• MINICPMV_MODELSCOPE_CACHE_DIR (optional)
• MINICPMV_MIN_VRAM_GB (default: 18.0)
• MINICPMV_ALLOW_CPU_FALLBACK (false by default)
• MINICPMV_MAX_CONCURRENT_INFERENCES (default: 1)
• MINICPMV_MAX_VIDEO_MB (default: 2048)
• MINICPMV_MAX_VIDEO_SEC (default: 7200)
• MINICPMV_MAX_FRAMES_CAP (default: 24)
• MINICPMV_MAX_SEGMENTS_CAP (default: 180)
• MINICPMV_MAX_NEW_TOKENS_CAP (default: 1536)
• MINICPMV_MAX_SLICE_NUMS_CAP (default: 3)
• MINICPMV_MAX_KEYFRAMES_CAP (default: 20)
• MINICPMV_MAX_KEY_CLIP_SEC (default: 15)

Remote client/server:

• MINICPMV_REMOTE_URL (e.g. http://192.168.31.18:8788)
• MINICPMV_REMOTE_TIMEOUT_SEC (default: 300)
• MINICPMV_REMOTE_RETRIES (default: 2)
• MINICPMV_REMOTE_TOKEN (optional bearer auth)
• MINICPMV_REMOTE_MAX_CONCURRENCY (worker-side, default 1)
• MINICPMV_REMOTE_HOST (default 0.0.0.0)
• MINICPMV_REMOTE_PORT (default 8788)
• MINICPMV_REMOTE_LOG_LEVEL (default info)

vLLM:

• MINICPMV_VLLM_MODE (auto|on|off, default: auto)
• MINICPMV_VLLM_BASE_URL (e.g. http://127.0.0.1:8000/v1)
• MINICPMV_VLLM_MODEL_ID (default fallback to MINICPMV_Q4_MODEL_ID)
• MINICPMV_VLLM_API_KEY (optional bearer auth)
• MINICPMV_VLLM_TIMEOUT_SEC (default: 300)

• video_understand(..., dry_run=True) skips model inference and only returns sampled frames.
• segment_sec defaults to 60; set segment_sec=0 to disable segmentation.
• Segmented runs will synthesize a global timeline summary after per-segment understanding (integrate_segments=True).
• key_moments include timestamps and extracted keyframes; set export_key_clips=True to also output short reference clips.
• video_detect_backend now includes model source/vLLM auto-selection details for diagnostics.
• For stability, keep concurrency at 1 unless you validate VRAM headroom.
• 4bit/8bit typically requires bitsandbytes on Linux CUDA environments.