Overview • Install • Usage • Pretrained Weights for MCCNN networks • Related • References

Pytorch implementation of [MCCNN] neural network which computes a similarity measure on pair of small image patches.

MCCNN is available on Pypi and can be installed by:

pip install MCCNN

After cloning source code from repository, do a local pip install in a virtualenv through MCCNN Makefile:

make install

PyTorch is an optional dependency. If your work requires it (e.g. training or running neural networks, or loading .pt weight files), install the torch extras group:

make install-torch

or

pip install .[torch]

Note: If you do not need PyTorch, the base make install is sufficient.

Documentation explains how to train and use MCCNN convolutional neural network. To generate it, please execute the following commands:

make docs

Let's see pandora_plugin_mccnn for real life example.

Pretrained weights for mc-cnn fast and mc-cnn accurate neural networks are available in the weights directory :

• mc_cnn_fast_mb_weights.pt and mc_cnn_accurate_mb_weights.pt are the PyTorch weights of the pretrained networks on the Middlebury dataset [Middlebury]
• mc_cnn_fast_data_fusion_contest.pt and mc_cnn_accurate_data_fusion_contest.pt are the PyTorch weights of the pretrained networks on the Data Fusion Contest dataset [DFC]
• mc_cnn_fast_int8_excl_01.onnx and mc_cnn_fast_dw.onnx are the ONNX weights of the pretrained networks on the Middlebury dataset [Middlebury]

To download the pretrained weights:

wget https://raw.githubusercontent.com/CNES/Pandora_MCCNN/master/src/mc_cnn/weights/mc_cnn_fast_mb_weights.pt
wget https://raw.githubusercontent.com/CNES/Pandora_MCCNN/master/src/mc_cnn/weights/mc_cnn_fast_data_fusion_contest.pt
wget https://raw.githubusercontent.com/CNES/Pandora_MCCNN/master/src/mc_cnn/weights/mc_cnn_accurate_mb_weights.pt
wget https://raw.githubusercontent.com/CNES/Pandora_MCCNN/master/src/mc_cnn/weights/mc_cnn_accurate_data_fusion_contest.pt
wget https://raw.githubusercontent.com/CNES/Pandora_MCCNN/master/src/mc_cnn/weights/mc_cnn_fast_int8_excl_01.onnx
wget https://raw.githubusercontent.com/CNES/Pandora_MCCNN/master/src/mc_cnn/weights/mc_cnn_fast_dw.onnx

Pretrained weights are stored into the pip package and downloaded for any installation of mc_cnn pip package. To access it, use the weights submodule :

from mc_cnn.weights import get_weights, get_onnx
mc_cnn_fast_mb_weights_path = get_weights(arch="fast", training_dataset="middlebury")
mc_cnn_fast_data_fusion_contest_path = get_weights(arch="fast", training_dataset="dfc")
mc_cnn_accurate_mb_weights_path = get_weights(arch="accurate", training_dataset="middlebury")
mc_cnn_accurate_data_fusion_contest = get_weights(arch="accurate", training_dataset="dfc")
onnx_mc_cnn_fast_int_8_mb_weights_path = get_onnx(arch="onnx_fast_int8", training_dataset="middlebury")
onnx_mc_cnn_fast_dw_mb_weights_path = get_onnx(arch="onnx_fast_dw", training_dataset="middlebury")

Please cite the following paper when using MCCNN:

Defonte, V., Dumas, L., Cournet, M., & Sarrazin, E. (2021, July). Evaluation of MC-CNN Based Stereo Matching Pipeline for the CO3D Earth Observation Program. In 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS (pp. 7670-7673). IEEE.

Cournet, M., Sarrazin, E., Dumas, L., Michel, J., Guinet, J., Youssefi, D., Defonte, V., Fardet, Q., 2020. Ground-truth generation and disparity estimation for optical satellite imagery. ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[MCCNN] Zbontar, J., & LeCun, Y. (2016). Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res., 17(1), 2287-2318.

[Middlebury] Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., & Westling, P. (2014, September). High-resolution stereo datasets with subpixel-accurate ground truth. In German conference on pattern recognition (pp. 31-42). Springer, Cham.

[DFC] Bosch, M., Foster, K., Christie, G., Wang, S., Hager, G. D., & Brown, M. (2019, January). Semantic stereo for incidental satellite images. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 1524-1532). IEEE.