Official release code for GarlTTC, an RGB-event object time-to-contact estimation benchmark. This repository contains training, inference, evaluation, dataset conversion, and CodaBench helper code for reproducing the released GarlTTC results.
| Asset | Repository | Contents |
|---|---|---|
| 📦 eAP media dataset | NAIL-HNU/eAP-dataset | RGB/event media shared by eAP and GarlTTC |
| 📄 GarlTTC annotations | NAIL-HNU/GarlTTC-dataset | Public train labels plus test metadata |
| 🧠 GarlTTC checkpoints | NAIL-HNU/GarlTTC-model | Full model and ablation checkpoints |
Download the eAP dataset first. The GarlTTC dataset references eAP RGB/event media and does not duplicate those files.
| Page | Link |
|---|---|
| 🌐 eAP dataset project page | https://nail-hnu.github.io/eAP_dataset/ |
| 🏆 eAP Benchmark | https://www.codabench.org/competitions/17291/ |
| 🏆 GarlTTC Benchmark | https://www.codabench.org/competitions/17289/ |
The public test annotations do not include ground-truth TTC values. Submit predictions to the CodaBench benchmark for official evaluation.
Run the setup script below from the repository root. It creates this layout:
GarlTTC_release/
data/
eAP-dataset/ # HF NAIL-HNU/eAP-dataset
GarlTTC-dataset/ # HF NAIL-HNU/GarlTTC-dataset
GarlTTC-model/ # HF NAIL-HNU/GarlTTC-model snapshot
checkpoints/
paper_ours_full.pth
paper_visual_only_lhr.pth
paper_event_only_lhr.pth
...
outputs/
The default release config is:
configs/garl_ttc_eventdecoder.yaml
It is the full RGB+event model config. The final checkpoint is:
checkpoints/paper_ours_full.pth
Python 3.8 is recommended. The project is packaged with uv.
python -m pip install uv
uv syncOne-command setup:
bash scripts/setup_release_assets.shIf Hugging Face requires authentication, export a token first:
export HF_TOKEN=hf_xxx
bash scripts/setup_release_assets.shUseful environment overrides:
DATA_DIR=/path/to/assets bash scripts/setup_release_assets.sh
SKIP_UV_SYNC=1 bash scripts/setup_release_assets.shBy default, DATA_DIR is ./data.
Generate a CodaBench JSON submission on the public test split:
uv run python tools/infer.py \
--config configs/garl_ttc_eventdecoder.yaml \
--checkpoint checkpoints/paper_ours_full.pth \
--data-root data/eAP-dataset \
--garlttc-annotation-root data/GarlTTC-dataset \
--output-json outputs/garlttc_test_submission.jsonThe output schema is:
{
"meta": {"format": "garlttc_prediction_v1"},
"results": {
"<sample_token>": {"ttc": 1.0}
}
}Zip this JSON as root-level submission.json before submitting to CodaBench:
uv run python - <<'PY'
from pathlib import Path
import zipfile
src = Path("outputs/garlttc_test_submission.json")
dst = Path("outputs/garlttc_test_submission.zip")
with zipfile.ZipFile(dst, "w", compression=zipfile.ZIP_DEFLATED, compresslevel=6) as zf:
zf.write(src, "submission.json")
print(dst)
PYThe public HF dataset does not include test TTC ground truth. If you have the private CodaBench reference labels locally, run:
uv run python tools/eval.py \
--config configs/garl_ttc_eventdecoder.yaml \
--checkpoint checkpoints/paper_ours_full.pth \
--data-root data/eAP-dataset \
--garlttc-annotation-root data/GarlTTC-dataset \
--test-labels-parquet /path/to/private_labels/test/labels.parquet \
--asset-file configs/splits/test.txt \
--output-dir outputs/eval_test12Expected paper test12 row:
MiDc/MiDs/MiDl/MiDn: 53.1 / 37.6 / 40.6 / 31.3
FRc/FRs/FRl/FRn: 0.0 / 0.0 / 0.0 / 0.0
num_samples: 6762
Train with the public HF-style annotations:
uv run python tools/train.py \
--config configs/garl_ttc_eventdecoder.yaml \
--data-root data/eAP-dataset \
--garlttc-annotation-root data/GarlTTC-dataset \
--output-dir outputs/train_fullFor a quick loader smoke test:
uv run python tools/train.py \
--config configs/garl_ttc_eventdecoder.yaml \
--data-root data/eAP-dataset \
--garlttc-annotation-root data/GarlTTC-dataset \
--epochs 1 \
--max-batches 1 \
--batch-size 1 \
--num-workers 0 \
--output-dir outputs/train_smokeThe full model uses paper_visual_only_lhr.pth and paper_event_only_lhr.pth
as branch pretraining checkpoints. The setup script downloads them into
checkpoints/.
The model repo also contains ablation checkpoints and matching configs:
configs/ablation/*.yaml
checkpoints/paper_*_baseline.pth
checkpoints/paper_*_lhr*.pth
Evaluate one ablation by swapping --config and --checkpoint, for example:
uv run python tools/eval.py \
--config configs/ablation/visual_lhr.yaml \
--checkpoint checkpoints/paper_visual_only_lhr.pth \
--data-root data/eAP-dataset \
--garlttc-annotation-root data/GarlTTC-dataset \
--test-labels-parquet /path/to/private_labels/test/labels.parquet \
--asset-file configs/splits/test.txt \
--output-dir outputs/eval_visual_lhrBuild the GarlTTC HF staging tree from local source annotations:
uv run python -m garl_ttc_benchmark.build_garlttc_dataset \
--dataset-info configs/dataset_info.json \
--garlttc-annotation-root dataset/annotations \
--eap-public-root data/eAP-dataset \
--output-root outputs/GarlTTC-dataset-staging \
--overwriteBuild and test the CodaBench bundle:
uv run python -m garl_ttc_benchmark.codabench.build_codabench_bundle \
--garlttc-output-root outputs/GarlTTC-dataset-staging \
--output-dir outputs/GarlTTC-codabench \
--overwrite \
--run-local-testsStage the model repository:
uv run python -m garl_ttc_benchmark.stage_model_repo \
--output-root outputs/GarlTTC-model-staging \
--overwrite