DistroFair is an automated test generation framework that identifies class-level fairness violations. DistroFair automatically learns the distribution of images in the dataset and generates images that are outside the distribution and finds class-level fairness violations in the generated images.

Link: DOI

• torch
• detectron2 (Note: A GPU with CUDA is highly recommended)
• numpy
• cv2
• PIL
• scipy

Note: Authentication keys for each of the individual subject programs are required to run the notebooks. You will also require a GPU and additional packages such as LaMa.

• from google.cloud import vision

Link: Credentials

• azure.cognitiveservices.vision.computervision

Link: Credentials

• boto3

Link: Credentials

We also use LaMa to manipulate images during the generation process.

The code consists of 4 files.

• CLM_Submit.ipynb
• RecSubj3.ipynb
• submitFairnessErrGroupFinder.py
• submitFairnessErrCounter.py

CLM_Submit.ipynb is primarily concerned with the generation of OOD images. In particular, it seeks to find the bounds for the images in each cluster and generates images accordingly. It is able to generate images associated with insertion. It also generates the intermediate images for the deletion and rotation operations. LaMa is then used to further manipulate the image.

RecSubj3.ipynb is primarily concerned with querying the subject programs and writing the results to a text file.

submitFairnessErrGroupFinder.py is used to identify classes that violate our fairness criteria.

submitFairnessErrCounter.py is used to find the number of erroneous images among the generated images.

The original images used for testing are included at the link below.

Link: OriginalImages

Note: Generated Images are included in a separate zip file due to the large file size. It is approximately 70 GB.

Link: GeneratedImages

To execute these notebooks, you will need authentication keys for each of the individual subject programs. You will also require a GPU and additional packages such as LaMa.

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

@article{rajan2024distribution,
  title={Distribution-aware fairness test generation},
  author={Rajan, Sai Sathiesh and Soremekun, Ezekiel and Le Traon, Yves and Chattopadhyay, Sudipta},
  journal={Journal of Systems and Software},
  volume={215},
  pages={112090},
  year={2024},
  publisher={Elsevier}
}