Bi-JCR: Bi-Manual Joint Camera Calibration and Scene Representation

This repository is the official implementation of the paper Bi-Manual Joint Camera Calibration and Scene Representation. Bi-JCR enables multiple robot manipulators, each with a wrist-mounted RGB camera, to recover their hand-eye calibrations, the relative pose between manipulator bases, and a unified scale-consistent 3D scene representation, all from the same set of captured images and without any calibration markers. By leveraging 3D foundation models for dense, marker-free multi-view correspondence, Bi-JCR jointly estimates these transformations, producing a representation that supports downstream bimanual manipulation tasks such as collaborative grasping and object hand-overs.

Project page: https://tomtang502.github.io/bijcr_web/

Please cite our work via the bibtex:

@article{tang2025bijcr,
   title={Bi-Manual Joint Camera Calibration and Scene Representation},
   author={Tang, Haozhan and Zhang, Tianyi and Johnson-Roberson, Matthew and Zhi, Weiming},
   journal={arXiv preprint arXiv:2505.24819},
   year={2025}
}

It is recommended to use a package manager like conda to create a Python environment for installation. Our code is tested on Ubuntu 22.04 LTS with CUDA 12.4 and PyTorch 2.5.

git clone --recursive https://github.com/tomtang502/bijcr.git bijcr
cd bijcr

# create conda env
conda create -n bijcr python=3.10 cmake=3.14.0 -y
conda activate bijcr

# install bi-JCR (also installs vendored DUSt3R requirements)
sudo chmod +x installation.sh
./installation.sh

If you cloned without --recursive, fetch the DUSt3R submodule with:

git submodule update --init --recursive

The DUSt3R foundation model checkpoint and a small set of bi-manual sample trajectories are downloaded via:

sudo chmod +x download_ckpts.sh download_sample_data.sh
./download_ckpts.sh
./download_sample_data.sh

These populate ckpt/ and data/ respectively.

Each experiment lives in a single directory data/{exp_name}/ containing per-frame images and saved end-effector poses for both arms. The expected layout is:

data/
└── {exp_name}/
    ├── can_left_0.png         # primary-manipulator wrist-camera frame 0
    ├── can_left_1.png
    ├── ...
    ├── can_left_{N-1}.png
    ├── can_right_0.png        # secondary-manipulator wrist-camera frame 0
    ├── can_right_1.png
    ├── ...
    ├── can_right_{N-1}.png
    ├── can_left_poses.pt      # torch tensor of shape (N, 6): [roll, pitch, yaw, x, y, z]
    └── can_right_poses.pt     # torch tensor of shape (N, 6): [roll, pitch, yaw, x, y, z]

Each pose row is (roll, pitch, yaw, x, y, z) in the manipulator's base frame, with the translation in millimetres (Bi-JCR converts to metres internally via factor=1000). RPY angles follow Rz · Ry · Rx Euler convention. Image filenames must sort numerically — i.e., zero-pad if you have more than 10 frames per arm.

If you would like to additionally validate the recovered hand-eye transformations against an AprilTag ground truth, place 10 paired AprilTag captures under data/apriltag_data/ following the same naming convention, then run with --validate_with_apriltag.

gen_jcr.py walks through how to use Bi-JCR. Given a directory of images captured by each manipulator's wrist-mounted camera together with the corresponding end-effector poses, it recovers each arm's camera-to-end-effector transformation, the inter-arm relative pose, the scale factor, and a metric-scale 3D point cloud expressed in the primary manipulator's base frame.

Run the bundled sample experiment:

python gen_jcr.py --exp_name sample --n_imgs_per_side 7

Available CLI arguments:

• --exp_name (default sample): name of the experiment directory under --data_root.
• --n_imgs_per_side (default 7): number of images per manipulator (the paper uses 4, 7, or 9).
• --data_root (default data): root directory containing experiment subdirectories.
• --out_dir (default output): directory where the calibrated point cloud, camera poses, and recovered transformations are saved.
• --no_sgd: skip the SGD refinement on SO(3) and use only the closed-form solver. With SGD refinement (the default), the closed-form initialization is further refined via gradient descent on the rotation manifold.
• --validate_with_apriltag: additionally compare the recovered hand-eye transforms against an AprilTag-derived ground truth. Requires AprilTag captures under data/apriltag_data/.

tag_val.py provides the AprilTag-based ground-truth comparator used by --validate_with_apriltag. It reports the rotational and translational residuals between the recovered hand-eye transformations and the AprilTag-derived ones.

The paper experiments are performed on a pair of AgileX Piper 6-DoF manipulators, each carrying a low-cost USB webcam mounted on its gripper. Bi-JCR itself does not depend on the specific arm or camera — any pair of robot arms with wrist-mounted RGB cameras and forward-kinematic end-effector pose estimates can be used.

This repository includes a module licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Module: DUSt3R

Original Authors: Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, Jérôme Revaud

Source: https://github.com/naver/dust3r

License: CC BY-NC-SA 4.0

CC BY-NC-SA 4.0