keyframe

Stream-friendly keyframe extraction and LLM video understanding.

keyframe turns a video, a webcam, or an RTSP stream into a short, ordered set of representative frames and a natural-language description of what happens across the clip. It uses one algorithm and one CLI for both offline files and live streams: the only thing that changes between the two is the iterator that yields frames.

   video in  →  sample  →  embed  →  segment  →  select  →  caption out
                  │          │          │          │           │
              1 frame/s    YOLO /     online      sharpness   chat.completions
              by default   phash /    EMA shot    + repr      multi-image
                           hsv /      detection   + spacing   request
                           SAM        + anti-     + diversity
                                      flicker     filter

A single video can have tens of thousands of frames. Sending all of them to a multimodal LLM is wasteful and slow. Sending one frame per second is what the v1 prototype did, and the output suffered from motion blur and near-duplicate frames.

The fix is a four-stage funnel that runs identically on a file or a live stream:

1. Sample the input at a fixed wall-clock interval (default 1 frame/sec). File sources seek directly to each sample point through ffmpeg or cv2.CAP_PROP_POS_FRAMES, so the decoder never wastes work on frames the pipeline is going to throw away.
2. Embed each sampled frame into a vector. Cheap baselines (phash, hsv) cost a millisecond on CPU. Neural backbones (YOLOv8 / YOLO11 / YOLO26 / MobileSAM) run on MPS or CUDA when available.
3. Segment the embedding stream online. The shot mean is an exponential moving average; a new shot opens when the next frame's cosine similarity falls below --sim-threshold. A --min-shot-sec guard prevents sub-second flicker (auto-focus, hand shake) from producing throwaway segments.
4. Select the best frames inside each segment. Each candidate is scored by a weighted z-score of (Laplacian sharpness, similarity to the segment centroid). The top frames are picked greedily under a minimum temporal spacing. Long segments get more keyframes (one per --seconds-per-keyframe, capped at --max-per-segment). A cross-segment diversity filter drops any keyframe that is nearly identical to one already kept, with a min-keyframes floor for short videos.
5. Caption every selected keyframe in a single multi-image chat completion request. The system prompt forbids guessing locations, brands, or motivations that are not visible.

Because the only piece that distinguishes offline and online cases is the VideoSource iterator, the pipeline only ever calls for frame in source: and decides one frame at a time. The same code path that captions a recorded clip also tails a live RTSP stream and writes keyframes to disk as they happen.

Requires Python 3.10+, optionally ffmpeg on the $PATH (for the fast decode path).

# core install
python3 -m pip install -e .

# add the Gradio web UI
python3 -m pip install -e '.[web]'

# add CLIP / SigLIP / MobileCLIP embedders (open_clip + Pillow)
python3 -m pip install -e '.[clip]'

# add the dev tooling (pytest, ruff)
python3 -m pip install -e '.[dev]'

Copy .env.example to .env and fill in your OPENAI_API_KEY (and OPENAI_BASE_URL if you are using a proxy such as OpenRouter). The caption stage is optional — pass --no-caption to skip the LLM call entirely.

# offline: a video file in, an explanation out
python3 -m keyframe run video/video.MOV

# online: webcam stream, keyframes emitted live to logs/stream_events.jsonl
python3 -m keyframe run 0

# RTSP camera, lower threshold so subtle scene changes register
python3 -m keyframe run rtsp://192.168.1.10/stream1 --sim-threshold 0.92

# replay a file at native fps to prove the pipeline keeps up in real time
python3 -m keyframe run video/video.MOV --realtime

# skip captioning when you only want keyframes + visualisations
python3 -m keyframe run video/video.MOV --no-caption

# launch the Gradio web UI
python3 -m keyframe web --port 7860

# benchmark several embedders on the same clip and write a comparison report
python3 -m keyframe benchmark video/video.MOV --embedders yolov8n phash hsv

run writes everything for one source into outputs/<source_stem>_<YYYYMMDD_HHMMSS>/. Each run is self-contained, never overwrites a previous run, and ships a README.md plus summary.json you can read without exploring the rest of the tree.

from keyframe import PipelineConfig, run_pipeline

result = run_pipeline("video/video.MOV", PipelineConfig())
print(result.run_dir, result.keyframes, result.caption_text)

For UIs and notebooks that want live updates, iterate the pipeline directly:

from keyframe import iter_pipeline, PipelineConfig

for event in iter_pipeline("video/video.MOV", PipelineConfig()):
    if event.stage == "frame":
        print(event.message)
    if event.final_result is not None:
        print("done:", event.final_result.run_dir)

iter_pipeline is a generator of ProgressEvent objects. It is what the CLI and the Gradio app consume internally.

.
├── pyproject.toml                 installable package + entry point
├── requirements.txt               runtime deps (plain pip)
├── README.md                      this file
├── .env.example                   template for OPENAI_API_KEY / OPENAI_BASE_URL
├── video/                         input videos (gitignored)
├── outputs/                       one folder per run (gitignored)
├── tests/                         pytest suite (unit + smoke tests)
└── src/keyframe/
    ├── __main__.py                python3 -m keyframe
    ├── cli.py                     argparse, sub-commands
    ├── pipeline.py                streaming orchestrator + ProgressEvent
    ├── source.py                  VideoSource: file, webcam, or RTSP via cv2
    ├── ffmpeg_source.py           hardware-accelerated decode for files
    ├── embedders.py               YOLO, MobileSAM, phash, hsv (plugin registry)
    ├── segmenter.py               online EMA shot detector with anti-flicker
    ├── selector.py                per-segment + cross-segment keyframe picker
    ├── captioner.py               multi-image chat.completions caller
    ├── visualize.py               timeline, similarity curve, grid, film strip
    ├── benchmark.py               cross-embedder report
    ├── webapp.py                  Gradio UI with live progress
    ├── config.py                  dataclass configs, dotenv loader
    ├── io_utils.py                unicode-safe image IO + math helpers
    └── logging_setup.py           rich logger

outputs/<source_stem>_<YYYYMMDD_HHMMSS>/

YOLO weights are downloaded on first use by ultralytics and cached in the project root. CLIP family weights live in ~/.cache/clip and are pulled by open_clip on first call. The two CPU baselines (phash, hsv) exist so the benchmark mode can answer "is the heavy model worth it?".

Cluster scenes by layout with the YOLO family; cluster by semantic content (people doing things, objects, settings) with the CLIP family. On most real footage, mobileclip-s0 is the sweet spot for live use and siglip-b16 is the sweet spot for offline quality.

python3 -m keyframe list-embedders

Two benchmark modes ship out of the box.

Embedder benchmark — runs the full pipeline once per embedder and compares latency, segment count, keyframe count, and Jaccard agreement of keyframe times against the first (reference) embedder.

python3 -m keyframe benchmark video/video.MOV \
    --embedders yolov8n mobileclip-s0 siglip-b16 phash hsv \
    --sample-interval 1.0 --sim-threshold 0.94

Output: outputs/benchmarks/benchmark.md plus per-embedder run folders.

Captioner benchmark — runs the keyframe pass once, then sends the same keyframe set to every requested LLM. Useful for picking which model to default to.

python3 -m keyframe caption-bench video/video.MOV \
    --models gpt-5.4 google/gemini-2.5-flash anthropic/claude-3.5-haiku \
            qwen/qwen2.5-vl-72b-instruct

Output: outputs/caption-bench/caption_benchmark.md (and .json) with latency, character / word counts, and the full caption text per model.

When OPENAI_API_KEY is set, the pipeline now captions every shot as it closes in a background worker, instead of waiting until the whole video has been processed. The Gradio UI's Live narration card fills in line by line as the LLM responses come back; a stronger model then writes the final synthesis paragraph at the end.

Toggle and tune via CaptionerConfig:

PipelineConfig(
    captioner=CaptionerConfig(
        model="gpt-5.4",                 # final synthesis model
        segment_model="google/gemini-2.5-flash",  # fast per-shot model
        segment_max_frames=2,            # frames per shot caption
        stream_per_segment=True,         # default; set False to disable
    ),
)

Per-run artefacts:

If OPENAI_API_KEY is missing or --no-caption is passed, streaming captions are silently disabled and the keyframe pipeline runs as before.

# webcam, save keyframes as they happen, no LLM
python3 -m keyframe run 0 --no-caption --sim-threshold 0.92

# replay a file at native fps so you can prove the pipeline keeps up
python3 -m keyframe run video/video.MOV --realtime

The pipeline emits one line of logs/stream_events.jsonl per analyzed frame, in real time. Each "event": "frame" line carries the timestamp, similarity, and is_shot_start flag. Downstream systems (queue producer, webhook, secondary model) can tail that file or be wired directly into StreamingSegmenter.ingest.

python3 -m keyframe web --port 7860

The Gradio app has two tabs:

• Run — upload a video, watch the film strip build as each frame is analyzed, see the keyframes gallery fill in, and read the final LLM caption. Streams ProgressEvents straight from iter_pipeline.
• Compare models — run the same clip through several embedders and stack their film strips. A good embedder shows clear segment colour blocks aligned with real shot cuts; a bad one shows random colour noise.

# run the unit tests (selector, segmenter, config, embedders, CLI, IO)
python3 -m pytest tests -q

# lint (optional, requires the [dev] extra)
ruff check src tests

The pytest suite covers the deterministic core (segmenter, selector, config, CPU embedders, IO helpers, CLI parser). It does not download neural weights or hit the network, so it runs in well under a second on a laptop.

• The captioner sends a single chat completion request. For very long videos (hundreds of keyframes) you should split into a windowed map-reduce. Not implemented yet.
• The frame cache writes every sampled frame to disk at cache_thumb_width. For multi-hour streams this will grow without bound. Add a ring-buffer policy for true 24/7 operation.
• YOLO backbone embeddings are a side-effect of a detector, not a general-purpose visual encoder. They cluster scenes by gross layout, not by semantic content. For finer-grained understanding, register a new Embedder subclass that wraps CLIP or SigLIP.