VLX-VR

English | 中文

_{Overview video: VLX-VR for agentic-aware video reasoning}

📺 HD version: Watch on YouTube

VLX-VR is an agentic-aware video reasoning model trained inside a video reasoning framework defined by a Think–Memory–Observation loop. It targets real-world video analysis where visual, audio, textual, and temporal evidence is scattered across short and long videos, and where fixed-context, single-pass VideoQA is not enough.

Instead of locking the video context before reasoning begins, VLX-VR learns to decide what evidence is needed, invoke read_memory or write_memory, incorporate the returned Observation, and decide whether to continue or produce the task output.

Join the VLX community to connect with developers, explore applications, share feedback, and shape the future of multimodal AI.

Official WeChat	Discord Community

For technical support, partnerships, and community inquiries, contact us at marketing@hzlh.com.

• [2026-09-22] 🔥🔥🔥 GitHub package published: README (EN/ZH), overview assets, and a notebook.
• [2026-09] Paper released: VLX-VR: An Agentic-Aware Video Reasoning Model (arXiv:2609.09985).
• [2026-09] Overview video published: Watch on YouTube.

Figure 1. Framework-defined Think–Memory–Observation loop used to train and run VLX-VR.

Modern video large language models are strong at describing clips and answering Visual QA from a fixed sampled context. Real-world event analysis is harder:

• Adaptive evidence acquisition: revisit earlier moments, seek missing cues, or resolve conflicts after an initial observation.
• Multimodal grounding: combine frames, audio, OCR/text overlays, and timestamps rather than relying on a single modality.
• State maintenance: keep intermediate hypotheses and observations across multi-step reasoning.
• Termination control: stop when evidence is sufficient, instead of looping forever or answering too early.

Many pipelines still follow:

video + question -> fixed sampled context -> single-pass answer

VLX-VR changes the task to:

video + instruction -> Think -> Memory(read/write) -> Observation -> continue or answer

This makes evidence acquisition, memory use, and termination part of the model's learned reasoning policy, not only an external prompt wrapper.

• Agentic-aware video reasoning: trained to operate inside a Think–Memory–Observation loop rather than only sitting behind an external agent controller.
• Direct multimodal memory access: Memory exposes read_memory / write_memory; VLX-VR selects the operation from the current reasoning state.
• Strong MINERVA result: 78.79% accuracy among models in our comparison.
• Stable across duration bins: 76.70% / 78.73% / 80.92% under MINERVA's original three duration groups; CDAV = 2.97 pp².
• Checkable process: on correctly answered samples, 96.20% of reasoning traces agree with MINERVA reference traces and the evidence they describe; about 75.80% of all samples satisfy both answer correctness and this evidence-grounded criterion.

People understand videos by locating relevant moments, comparing states before and after an event, reading text in a scene, and relating spoken content to visual actions. Supporting such reasoning requires more than recognizing objects or isolated frames.

In many standard VideoQA pipelines, models such as Video-LLaVA, Qwen3-VL, and VideoLLaMA 3 receive sampled frames or clips as a fixed input and produce an output in one pass. Because evidence is determined before reasoning begins, the model cannot adaptively acquire new evidence after detecting missing, ambiguous, or conflicting information.

Existing video agents move closer through iterative gathering, temporal localization, and memory queries. However, placing a general-purpose VLM inside an external loop without training it for that role can still fail: the model may request redundant evidence, forget intermediate states, or terminate too early.

VLX-VR focuses on a core question:

Can a video reasoning model be trained not merely to participate in an agentic pipeline, but to learn evidence acquisition, memory use, and termination as integral parts of its reasoning process?

VLX-VR's answer is:

Train an agentic-aware model inside a Think–Memory–Observation loop with direct multimodal-memory access.

At reasoning step t, VLX-VR maintains a state s_t that includes the task instruction, observed evidence, multimodal content stored in memory, current output hypotheses, and unresolved uncertainty.

The loop has three stages:

1. Think: interpret the task, propose the next evidence need, and estimate whether current evidence is sufficient.
2. Memory: invoke read_memory or write_memory to retrieve multimodal evidence or retain intermediate state.
3. Observation: return the result of the Memory operation to VLX-VR for the next decision.

When evidence is sufficient, VLX-VR produces the final output with supporting evidence. When evidence is insufficient or conflicting, it returns to Think and starts another Memory step.

s_0 = initialize(video, instruction)
while not stop(s_t):
    think_t, call_t = VLX-VR(s_t)
    obs_t = execute(call_t)          # call_t ∈ {read_memory, write_memory}
    s_{t+1} = update(s_t, obs_t)
output = VLX-VR(s_t)

Multimodal memory is an external and inspectable reasoning state rather than only a cache of textual summaries. Video, audio, supporting evidence, Observations, and intermediate states can be read, written, recorded, and reused across steps.

We evaluate primarily on MINERVA because it combines broad temporal coverage with human-annotated reasoning traces. Each question includes a video, a question, five choices, and a reference reasoning trace, making it suitable for scoring both answer accuracy and evidence-grounded process quality.

Figure 2. Accuracy across MINERVA duration groups. VLX-VR stays strong as duration increases under the original three-bin grouping.

Under the original MINERVA grouping, VLX-VR has mean accuracy 78.78%, CDAV 2.97 pp², std 1.72 pp, and range 4.22 pp. Public baselines decline with duration; VLX-VR does not under this grouping.

A refined split of the long-video bin shows the Above 15 min result is driven mainly by 15–30 min (83.40%), while Above 30 min is 74.75%. Performance remains broadly stable rather than collapsing on longer videos.

VLX-VR is strongest on reading, situational awareness, temporal reasoning, and numerical reasoning. Counting, state changes, causal reasoning, and spatial perception remain challenging.

On correctly answered samples, 96.20% of VLX-VR reasoning traces are consistent with MINERVA reference traces and the evidence they describe. Across the full set:

78.79% × 96.20% ≈ 75.80%

samples satisfy both answer correctness and this evidence-grounded trace criterion. This joint rate is still higher than Seed2.1 Pro's answer-only accuracy (70.70%), though the criteria differ.

Three real side-by-side comparisons of VLX-VR and Gemini 3.1 Pro on the same user prompts and public web footage, focusing on how each model grounds evidence, timestamps, and final answers.

Case 1 · Counterfactual score If the player in green had made his shot on the goal at 02:22 in the video, what would the score have been? (0) 1-0. (1) 2-0. (2) 3-0. (3) 2-1. (4) 2-2. ▶️ Watch MP4

Case 2 · Temporal counting How many turns does white play between the first time they are put in check and the end of the game? (0) 4. (1) 2. (2) 0. (3) 3. (4) 1. ▶️ Watch MP4

Case 3 · Anomaly + timestamps 看看视频里有哪些和我们日常操作反常的内容，时间点发我下 ▶️ Watch MP4

Files live under assets/demos/. Test clips are public-web footage.

See notebooks/vlx_vr.ipynb — Coming soon.

• Compared with a fixed-context VideoQA model, VLX-VR can adaptively acquire and reassess evidence during reasoning.
• Compared with an untrained VLM inside an external agent loop, VLX-VR learns evidence acquisition, memory use, and termination as policy behaviors.
• Compared with answer-only VideoQA evaluation, MINERVA-style traces let us check whether correct answers are also evidence-grounded.
• Compared with public long-video curves that drop with duration, VLX-VR remains strong under MINERVA's original three duration groups.

Our team has spent years building multimodal perception and reasoning systems, with open-source projects such as OmDet, VLM-R1, VLX-Seek, VLX-Flow, and VLX-Go. VLX-VR extends this line from perception and streaming understanding toward agentic-aware video reasoning with explicit multimodal memory control.

@article{li2026vlx,
  title={VLX-VR: An Agentic-Aware Video Reasoning Model},
  author={Li, Sheng and Liu, Peng and Zhang, Qianqian and Zhao, Tiancheng},
  journal={arXiv preprint arXiv:2609.09985},
  year={2026}
}

This project is released under the Apache License 2.0. See the license file for terms covering code and model weights.