Video LLM inference

Video LLMs, built for speed.

We build the inference engines, and the models that run on them, for video language models that understand live and long video in real time.

A pixel globe. The animation follows a camera and OpenVector mark into a pallet inside a warehouse, highlighting its position in the pedestrian route.

Distillation

Same answer.
About 30× sooner.

We distilled Qwen3-VL-8B into a small model fine-tuned for a computer vision task. It matches the 8B model’s accuracy and returns each answer in about 16 ms.¹

Latency per video window, batch 1, one A100

Qwen3-VL-8Breads every frame513 ms

OpenVector distilled modelsame accuracy16 ms

~30×

faster than Qwen3-VL-8B on computer vision tasks, at the same accuracy¹

+59%

throughput on Qwen3-VL-32B from our 4-bit quantization²

The problem

Video breaks LLM serving.

A video LLM turns every frame into hundreds of tokens. On a long or live stream, the context keeps growing, prefill dominates the bill, and latency climbs out of reach.

Faster kernels help at the edges. The bigger gains come from changing the model itself: how it reads video, how much of it it keeps, and how small it can be for the job.

We work on the whole stack.

Architecture

Hybrid models

A Qwen decoder with linear, O(n) attention encodes the video, so cost grows with the length of the stream, not its square. A bidirectional head reads the whole encoding at once and answers zero-shot.

Training

Fine-tuning and distillation

We distil large video LLMs into small students built for one job. On our computer vision tasks, they match their teacher at a fraction of the cost.

Serving

Inference engine

4-bit quantization, prefix and KV-cache reuse, speculative decoding and continuous batching, all tuned for video tokens rather than text.

API

One API for live and long video.

Send a stream or a recording, ask in plain language, and get structured answers as the video plays. We are working towards video that runs for hours, at the same cost per minute as the first.

const answer = await openvector.watch({
  stream: "rtsp://camera-04.local/live",
  ask: "Is anyone on the loading dock without a hi-vis vest?",
  format: { type: "boolean", evidence: true },
});

Illustrative. The API is in private development.

CVPR 2025 Highlight

Grounded in research.

Our co-founder’s CVPR 2025 paper with Shimon Ullman and Daniel Harari showed that vision models keep up to 80% of their capability using 3% of the pixels.³ We bring the same idea to video LLMs: spend compute only where the answer is.

Read the research

Internal benchmark on a workplace computer vision task: a distilled model on person crops (15.5–16.8 ms per window) against Qwen3-VL-8B decoding all frames (513 ms), batch 1 on an NVIDIA A100. Agreement with the 8B model on the same crops is within 0.02 κ.
Internal benchmark: our NVFP4 quantization against the official FP8 checkpoint, 16 concurrent requests on an RTX PRO 6000 Blackwell with vLLM. Benchmark accuracy changes by 1–5 points depending on model size.
Gizdov, Ullman and Harari, “Seeing More with Less: Human-like Representations in Vision Models”, CVPR 2025.