TRACE

TRACE connects causal video history, adapter execution, model or system responses, telemetry and scoring in one auditable workflow.

TRACE evaluation design — the Core → Adapter → model/system → scoring path, with QA and Proactive timelines.

Execution conditions separate visual state, response triggering and the evaluation boundary. They are comparison dimensions rather than capability labels.

Execution modes — native versus prefix-input visual state, autonomous versus polling response triggering, and model/Adapter versus complete-system boundaries.

TRACE evaluates two complementary behaviors:

The standard benchmark cohort contains 833 QA records and 407 Proactive records (1,270 windows) across 517 videos. The checked-in release also contains the full population: 1,248 records, 522 videos and 1,338 Proactive windows. Eight sampling-stress records are excluded from the standard cohort, so --subset full covers a different population.

Source videos are downloaded separately from the upstream datasets: StreamingBench and OVO-Bench. TRACE ships the v1.1.0 annotations, release manifest and validation metadata, not the source videos or model weights. Read the dataset card and data terms before using them. StreamingBench authors permit redistribution of the modified annotation files; StreamingBench-derived TRACE additions are under CC BY-NC-SA 4.0 within the rights held by TRACE contributors. OVO-Bench annotation terms remain unresolved.

The figures show how quality, timing and response behavior vary across configurations. Use the scorecards and metric definitions below when comparing results across execution conditions.

LiveCC and MOSS-Preview are near 65% QA accuracy, yet completion, response latency and recorded output-token totals differ. MiniCPM-O is shown as a diagnostic because its text-interface invalid-output rate is 93.52%.

MOSS-VL and AURA reach 8.05% and 7.92% In-window Accuracy. MOSS-VL has the lower False-alarm Rate (41.1% versus 59.1%) and a shorter observed median delay.

Explore and filter the complete results in the offline results explorer, or read the full benchmark results and measurement limits. The machine-readable result snapshot is available at

Tested with Python 3.10–3.12. Clone or download this repository, then enter its root directory:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
trace --help

The osb command remains installed as a compatibility alias of trace. OSB_VLM_JUDGE_ environment variables are still read as fallbacks for the TRACE_VLM_JUDGE_ names.

trace data validate --release data/releases/v1.1.0

Keep the source checkout: the wheel alone does not include annotations or videos. Model dependencies belong in the model's own environment.

This creates a synthetic video, runs QA and Proactive, validates bundles, rescores them and checks the expected answers. No GPU or judge service is required. The test double deliberately reads GT; this is a software check, not a model score.

set -euo pipefail
python scripts/make_smoke_fixture.py --output output/smoke
trace data validate --release output/smoke/release
for task in qa proactive; do
  trace run --task "$task" --release output/smoke/release --subset tiny \
    --adapter trace_bench.adapters:TestDoubleAdapter \
    --video-root output/smoke --output "output/smoke/$task" \
    --synthetic --judge-mode exact --checkpoint-every 5
  trace bundle validate "output/smoke/$task"
  trace score "output/smoke/$task" --judge-mode exact
done
python scripts/check_smoke_results.py --output output/smoke

Expect both answer checks to pass, QA accuracy and Proactive window accuracy to equal 1.0, zero failures, and synthetic: true / official_eligible: false. The 12-second fixture has an answer window aligned to default 5-second polling. Missing model telemetry is expected here. A valid bundle alone does not prove that an answer was produced.

Results stay in ignored output/. The generator refuses an existing fixture directory; use a new path consistently for another smoke run. Do not delete prior experiment results to rerun a tutorial.

1. Prepare source videos from StreamingBench and OVO-Bench.
2. Follow the LiveCC walkthrough, or the experimental ThinkStream guide.
3. For another model, implement the adapter interface; the adapter can live in its own repository.
4. Run tiny, inspect predictions, failures and telemetry, then select full. See scoring and resume for semantic judging.

Core delivers timestamped RGB uint8 arrays at 1 FPS through OpenCV. QA receives the legal prefix and identical user content across models. Proactive uses 5s/10s event windows or annotated state intervals, and evidence ends at the final strict-window boundary, clamped to source duration. Native/non-native visual state and autonomous/polling triggering are separate comparison dimensions; see the evaluation protocol.

• Data and limitations
• Release files and fields
• Examples and support status
• Evaluation protocol · scoring and results
• Release/version policy · validation
• Third-party notices · contributing · security · code of conduct
• Citation · changelog

TRACE code is released under MIT. Dataset rights and upstream attribution are described in DATA_TERMS.md. The repository does not host an online evaluator, and real model results require the relevant runtime, model weights and source videos.

The two Git-based checks below require a Git checkout and must run from its root. ZIP users can still install TRACE, run evaluations, tests and distribution checks; clone the repository to run these maintainer checks. CI covers Python 3.10–3.13.

mkdir -p output
python -m pytest -q --basetemp=output/pytest
ruff check .
python scripts/check_docs.py
python scripts/check_public_files.py
python -m build --outdir output/dist
python scripts/check_distribution.py --dist output/dist --output output/distribution-check

Use fresh build/check directories when repeating distribution checks. CI verifies synthetic answers and unpacked-source tests. GPU inference and live semantic-judge validation are separate checks.