Sim4D is a dataset for 4D-SLAM, introduced in the paper 4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians (CVPR 2025). Leveraging large-scale animated 3D meshes available on the internet, Sim4D generates diverse dynamic scenes with ground truth camera poses, time-varying geometry, and color — facilitating scientific evaluation and opening up modern 4D-SLAM research. In this repository, we provide both the dataset and its rendering script to help you generate your own customized dataset.

To quickly download the dataset we used for 4DTAM, visit this google drive. The v1_eval_scenes folder contains the sequences used for 4DTAM evaluation. v1 folder contains more examples of rendered dataset.

Start by downloading and installing Blender. Please note that this repo assumes Blender 4.3.2. After downloading, set the BLENDER_PATH, for example by

export BLENDER_PATH=/path/to/download/blender-4.3.2-linux-x64/blender 

Clone this repo and install the required dependencies.

conda env create --file environment.yaml
conda activate sim4d

In order to render your own dataset, you need to download (1) animated 3D meshes and (2) texture maps. To download the meshes and textures we used to create Sim4D, visit this google drive and skip this step.

• We recommend using Sketchfab as it contains a large number of animated 3D meshes that are under CC-BY license.
• Visit the website, select the CC-BY license, click Animated, and download the meshes you wish to render.

• We set the background to be a cube with some texture added onto it. To get a photorealistic render, you need to download high quality texture maps. We recommend using Poly Haven as all their assets are under CC0 license.

• Go to rendering/__init__.py and change the values of DATASET_DIR and PROJECT_DIR.
• Your folder structure should look like following:

glbs
ㄴobjects             # object meshes
  ㄴarquivo.glb
  ㄴastronaut.glb
  ㄴ...
ㄴscenes              # scene meshes (not needed for v1)
  ㄴabandoned_gallery.glb
  ㄴabandoned_warehouse.glb
  ㄴ...

textures            # textures downloaded from Poly Haven
ㄴasphalt_02_4k.blend
  ㄴasphalt_02_4k.blend
  ㄴtextures
    ㄴasphalt_02_diff_4k.jpg
    ㄴasphalt_02_disp_4k.png
    ㄴasphalt_02_nor_gl_4k.exr
    ㄴasphalt_02_rough_4k.jpg
ㄴbrick_pavement_02_4k.blend
ㄴ...

v1
ㄴarquivo
  ㄴconfig.txt
  ㄴtrain
    ㄴdepth
      ㄴ0000.exr        # depth map in .exr format (float32)
      ㄴ0000.png        # depth map in quantized .png format (uint16)
      ㄴ0000_vis.jpg    # depth map visualized in jet colormap
      ㄴ...
    ㄴintrins
      ㄴ0000.npy        # camera intrinsics as 3x3 numpy array
      ㄴ...
    ㄴnormal
      ㄴ0000.exr        # normal map in .exr format (float32)
      ㄴ0000.png        # normal map in quantized .png format (uint8)
      ㄴ...
    ㄴpose
      ㄴ0000.npy        # camera pose (world2cam) as 4x4 numpy array
      ㄴ...
    ㄴrgb
      ㄴ0000.png        # rendered image
      ㄴ...
    ㄴrgb_masked
      ㄴ0000.png        # rendered image, masked using foreground mask
      ㄴ...
  ㄴtest
    ㄴdepth
      ㄴ0010_view0.exr  # for each timestamp (first 4 digits),
      ㄴ...             # we render 8 test views
      ㄴ0010_view7.exr  # we do this every 20th frame, starting from the 10th
    ㄴintrins
    ㄴnormal
    ㄴpose
    ㄴrgb
    ㄴrgb_masked
ㄴastronaut
ㄴ...

The file rendering/v1/render_args.json contain the arguments for the render. For example, it will look like this:

{
"arquivo": {
    "mesh_id": "arquivo",       # file name of the mesh
    "url": "https://...",       # link to the mesh
    "creator": "https://...",   # see NOTE-1 below
    "license": "https://...",   # see NOTE-2 below
    "NoAI": false,              # see NOTE-3 below
    "modified": false,          # see NOTE-4 below
    
    "anim_length": [1, 240],    # keyframes between which the animation is defined
    "num_frames": 540,          # number of frames to render
    "correct_num_vertices": 42941,  # correct number of vertices, used to check whether the mesh is loaded correctly
    "ignore_meshes": [],        # some objects in the mesh are ignored when resizing and centerizing it
    "delete_meshes": [],        # some objects in the mesh are deleted

    "object_motion": false,     # if true, the object is re-centerized every frame
    "center_xyz": [0,0,1],      # centerization is applied only to the axes with "1"
    "frame_rate": 60,           # desired frame rate
    "yaw_offset": 0.0,          # offset in the yaw angle
    "elev_offset": 0.0,         # offset in the elevation angle
    "object_scale": 1.0,        # object scaling factor
    "xyz_offset": [0,0,0.2],    # xyz offset applied to the object
    "shade_smooth": true        # if true, the smooth shading is applied
    },
}

• NOTE-1: To comply with the CC-BY license, we give credit to the creator by providing the link to their Sketchfab page.
• NOTE-2: To comply with the CC-BY license, we provide a link to the license.
• NOTE-3: If this is set to True, it means the mesh may not be used in datasets for, in the development of, or as inputs to generative AI programs. This is requested by the creator, so please comply with this.
• NOTE-4: To comply with the CC-BY license, we indicate whether changes were made to the mesh.

cd rendering/v1
python render.py

The generated dataset will be stored under DATASET_DIR/v1/.

cd rendering/v1
python postprocess.py

Run the above to convert the depth/normal maps from .exr format to quantized .png format. This helps reduce the file size.

Running the above code will give you the images and their corresponding ground truth labels: depth / surface normals / foreground mask / camera pose / camera intrinsics.

With the provided render script, you can easily change the frame rate (the animation will be re-interpolated accordingly), add/remove motion blur, change object materials, etc.

Please also note that it is straightforward to render more challenging sequences by using scene meshes with more complex shapes, adding motion blur, defining more complex camera trajectories, change camera intrinsics within the sequence, etc.

Below are some examples showcasing renders with more complex object layouts, dynamic motion, and varied camera movements.

We thank Ollie Boyne, the author of BlenderSynth for comments and feedback. BlenderSynth is a Python library for generating large scale synthetic datasets using Blender and is a vital component of our rendering script.

If you use Sim4D in your work, please cite 4DTAM in which the dataset was introduced.

@inproceedings{4DTAM,
    title={4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians},
    author={Matsuki, Hidenobu and Bae, Gwangbin and Davison, Andrew},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2025}
}