Cheng-You Lu1*, Peisen Zhou1*, Angela Xing1*, Chandradeep Pokhariya2, Arnab Dey3, Ishaan N Shah2, Rugved Mavidipalli1, Dylan Hu1, Andrew Comport3, Kefan Chen1, Srinath Sridhar1
1Brown University 2CVIT, IIIT Hyderabad 3I3S-CNRS/Université Côte d’Azur
*Corresponding author
Projectpage · Paper · Video
We propose DiVa360, a real-world 360° dynamic visual dataset containing 21 object-centric sequences categorized by different motion types, 25 intricate hand-object interaction sequences, and 8 long-duration sequences for 17.4M frames.
diva360.mp4
Build conda environment using environment.yml, then follow instructions in each benchmark method to install required packages.
conda env create --file environment.yml
conda activate diva360_venv
Example Directory Structure
.
├── assets
│ ├── calib_long
│ │ ├── image
│ │ └── segmented_ngp
│ ├── calib_short
│ │ ├── image
│ │ └── segmented_ngp
│ └── objects
│ ├── blue_car
│ │ ├── calib
│ │ ├── dynamic_data
│ │ │ ├── 150
│ │ │ ├── I-NGP
│ │ │ │ ├── test
│ │ │ │ ├── train
│ │ │ │ └── traj
│ │ │ ├── frames_1
│ │ │ ├── kplanes
│ │ │ │ ├── blue_car_00
│ │ │ │ │ └── test_images
│ │ │ │ ├── blue_car_00_IST
│ │ │ │ ├── blue_car_01
│ │ │ │ ├── blue_car_01_IST
│ │ │ │ ├── test
│ │ │ │ └── traj
│ │ │ ├── mixvoxels
│ │ │ │ ├── imgs_spiral_hr_all
│ │ │ │ └── imgs_test_all
│ │ │ └── stds_1
│ │ ├── image
│ │ ├── segmented_ngp
│ │ └── undist
│ └── …
├── generate_scripts
├── metadata
├── objects_scripts
│ ├── battery
│ ├── blue_car
│ ├── bunny
│ ├── chess
│ ├── chess_long
│ ├── clock
│ ├── crochet
│ ├── dog
│ ├── drum
│ ├── flip_book
│ ├── horse
│ ├── hour_glass
│ ├── jenga
│ ├── jenga_long
│ ├── k1_double_punch
│ ├── k1_hand_stand
│ ├── k1_push_up
│ ├── keyboard_mouse
│ ├── kindle
│ ├── legos
│ ├── maracas
│ ├── music_box
│ ├── origami
│ ├── painting
│ ├── pan
│ ├── peel_apple
│ ├── penguin
│ ├── piano
│ ├── plasma_ball
│ ├── plasma_ball_clip
│ ├── poker
│ ├── pour_salt
│ ├── pour_tea
│ ├── put_candy
│ ├── put_fruit
│ ├── puzzle
│ ├── red_car
│ ├── rubiks_cube
│ ├── scissor
│ ├── slice_apple
│ ├── soda
│ ├── stirling
│ ├── tambourine
│ ├── tea
│ ├── tornado
│ ├── trex
│ ├── truck
│ ├── unlock
│ ├── wall_e
│ ├── wolf
│ ├── world_globe
│ ├── writing_1
│ ├── writing_2
│ └── xylophone
├── src
└── utils
You can download the data here. Please note that the dataset is large. Here is a breakdown of required storage:
- Raw Data: 1.4 TB
- Processed Data: 1.8 TB
- Trained Models: 6 TB
- Rendered Videos: 63.1 GB
Here are a list of sequences
- battery
- blue_car
- bunny
- chess
- clock
- dog
- drum
- flip_book
- horse
- hour_glass
- jenga
- k1_double_punch
- k1_hand_stand
- k1_push_up
- keyboard_mouse
- kindle
- maracas
- music_box
- pan
- peel_apple
- penguin
- piano
- plasma_ball
- plasma_ball_clip
- poker
- pour_salt
- pour_tea
- put_candy
- put_fruit
- red_car
- scissor
- slice_apple
- soda
- stirling
- tambourine
- tea
- tornado
- trex
- truck
- unlock
- wall_e
- wolf
- world_globe
- writing_1
- writing_2
- xylophone
Here are a list of long sequences
- chess_long
- crochet
- jenga_long
- legos
- origami
- painting
- puzzle
- rubiks_cube
Please consider citing these methods if you think they are helpful! Below are methods modified for DiVa360 benchmarks.
# Download processed data
# Please refer to the directory structure section
cp -r frames_1 ../code/DiVa360/assets/objects/blue_car/
cp transforms_* ../code/DiVa360/assets/objects/blue_car/
Please install the methods from in the Benchmark Methods section
# Train
sh objects_scripts/blue_car/train_ingp.sh
# Test and Benchmark
sh objects_scripts/blue_car/test_ingp.sh
# Render
sh objects_scripts/blue_car/traj_ingp_hr.sh
# Train
sh objects_scripts/blue_car/train_mixvoxels.sh
# Test and Benchmark
sh objects_scripts/blue_car/eval_mixvoxels.sh
# Render
sh objects_scripts/blue_car/render_mixvoxels.sh
# Train
sh objects_scripts/blue_car/train_kplanes.sh
# Test
sh objects_scripts/blue_car/test_kplanes.sh
# Render
sh objects_scripts/blue_car/render_kplanes.sh
# Benchmark
python utils/move_kplanes_test.py --root assets/objects/ --name blue_car
python utils/benchmark.py --root assets/objects/blue_car/ --start 0 --num_frames 150 --target_path kplanes/test --wh_bg
You can also preprocess raw data by yourself.
Download raw data:
# Download raw data
# Decompress file
gzip -d 2023-05-02_session_blue_car_synced.tar.gz
tar -xf 2023-05-02_session_blue_car_synced.tar
# Extract frames from the video
object_scripts/blue_car/move.sh <data path>/2023-05-02_session_blue_car_synced/synced
Camera pose estimation
# Download data for pose estimation
tar -xf 2023-04-29_session_calibration_2.tar
# If you are using long-duration object, download and untar this file instead
gzip -d 2023-10-21_session_calib.tar.gz
tar -xf 2023-10-21_session_calib.tar
# Run pose estimation with colmap version 3.8
python src/colmap_calib.py -r <data path>/2023-04-29_session_calibration_2
# Please refer to the directory structure section
mv <data path>/2023-04-29_session_calibration_2/params.txt assets/calib_short/
Camera pose refinement through I-NGP (please compile the I-NGP from the benchmark method)
# Manually segment one frame and put it in calib_short
python src/refine_params.py --root_dir assets/calib_short/ --optimize_params --network ../models/instant-ngp/configs/nerf/base.json --roi 0.5 0.45 0.5 --n_steps 10000 --aabb_scale 4 --face_to_cam --gui
# optim_param.txt to transform.json
python utils/params2nerf.py --root assets/calib_short/ --use_kp
Segmentation
sh objects_scripts/blue_car/segment_frame.sh
Undistortion
sh objects_scripts/blue_car/undistortion.sh
Q: What is the coordination system of the transform.json?
A: It follows the original NeRF coordinate system. The transform.json follows the I-NGP style, which can be used like the fox example. Please check nerfstudio for the details.
Q: How can I use it on other methods?
A: We suggest using llff or blender dataloader if the codebase supports it. The codebase should support the intrinsic matrix per camera.
Q: Why does the I-NGP segmentation not work well?
A: This is common situation. Tuning the parameter can solve the problem most of the time. We point out the limitations in the paper. hour_glass, plasma_ball, and plasma_ball_clip are segmented manually.
Q: Do I need to manually segment each object's first frame?
A: No, you can reuse the refined pose for other objects captured on the same date.
- add Gaussian Splatting to the benchmark
If you find this dataset useful for your research, please consider citing:
@inproceedings{diva360,
title={DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields},
author={Cheng-You Lu and Peisen Zhou and Angela Xing and Chandradeep Pokhariya and Arnab Dey and Ishaan N Shah and Rugved Mavidipalli and Dylan Hu and Andrew Comport and Kefan Chen and Srinath Sridhar},
booktitle = {Conference on Computer Vision and Pattern Recognition 2024},
year={2024}
}