Official code for the paper SHARCS: Shared Concept Space for Explainable Multimodal Learning

Requirement: Python 3.9+

Installing SHARCS

cd src
python -m venv SHARCS
source  SHARCS/bin/activate
pip install -r requirements.txt

XOR-AND-XOR: created inside the src/data_utils.py file

MNIST+SUPERPIXELS: downloaded original dataset and merged inside the src/data_utils.py file

HalfMNIST: downloaded original dataset and merged inside the src/data_utils.py file

CLEVR: follow the instruction in the official repository, constrining the generation to have only one object. In the question process you need to use the src/clevr_data/questions.json file as template. However, we provide a smaller dataset (100 questions) to try the models.

Requirement: Wandb already configured (Section 1 and 2)

wandb sweep --project SAHRCS ./config/[DATASET_NAME]_sweep.yaml

where [DATASET_NAME] could be clevr, xor, halfmnist, mnist+superpixels.

The previous command gives you the [AGENT_ID] and the full command to run the wandb agent. It is similar to the following one:

wandb agent [WANDB_ID]/SHARCS/[AGENT_ID]

Additionally, we set up src/clevr.ipynb, a Jupyter Notebook where it is possible to run step by step an experiments on CLEVR (it is set up with the smaller dataset included in the repository, therefore the results are not comparable with the one trained on the full dataset)