The Large Language Models Course Project

These projects are part of the Large Language Models at the Isfahan University of Technology developed under the supervision of Dr. Mina Amiri.

• Pouria Sameti
• Mahsa Alipour

This repository focuses on applying Vision-Language Models (VLMs) to medical data. We study a variety of learning paradigms, including zero-shot learning, few-shot learning, and model fine-tuning. In addition, we investigate image processing techniques to enhance the quality of medical images, improve data representation, and ultimately boost model performance. The objective is to evaluate different methodologies on established medical benchmarks and investigate how various training strategies and image enhancement techniques impact model performance.

Dataset
Part 1: Zero-Shot Classification
Part 2: Fine-Tuning
Part 3: CLIP & BLIP
Part 4: CLIP & MediBLIP

This project uses the MedMNIST benchmark, a large-scale collection of standardized biomedical image datasets designed for evaluating machine learning and deep learning models in medical imaging. MedMNIST provides multiple datasets from different medical domains, each consisting of labeled images for classification tasks.

The datasets used in this project are:

• DermaMNIST:A dataset of dermatoscopic images for skin lesion classification.
• TissueMNIST: A histopathology dataset consisting of microscopic tissue images.
• OCTMNIST: A dataset of Optical Coherence Tomography (OCT) retinal images.
• OrganMNIST: A dataset of computed tomography (CT) images containing slices from different human organs.

You can find more information about the MedMNIST datasets, download links, and documentation here

In the first stage of this project, we evaluate the zero-shot classification capabilities of Vision-Language Models on medical imaging datasets. We use the CLIP ViT-B/32 (CLIP-Base-Patch32) model without any task-specific training or fine-tuning.

Before inference, we apply a set of image preprocessing and enhancement techniques to improve image quality and emphasize medically relevant features. These preprocessing steps are intended to improve the model's ability to extract meaningful visual representations.

Since prompt engineering plays a crucial role in zero-shot performance, we design dataset-specific and class-specific prompt templates. Rather than using a single generic prompt, we tailor the prompts according to the characteristics of each dataset and the corresponding disease or anatomical class. This allows the model to better align visual features with their textual descriptions.

The table below summarizes the zero-shot classification performance of CLIP ViT-B/32 on the four MedMNIST datasets. The model was evaluated using our dataset-specific prompt templates together with the proposed image preprocessing pipeline. Performance is reported in terms of Accuracy and Area Under the ROC Curve (AUC).

In the second stage of this project, we fine-tune CLIP ViT-B/16 (CLIP-Base-Patch16) for medical image classification. Instead of performing zero-shot inference, we use the CLIP image encoder as a feature extractor and attach a Multi-Layer Perceptron (MLP) classification head. A separate classification head is trained for each MedMNIST dataset according to its number of classes.

To leverage the pre-trained knowledge of CLIP while adapting it to the medical domain, we fine-tune the last layers of the image encoder together with the MLP classifier. This enables the model to learn domain-specific visual representations without retraining the entire network.

To improve generalization and reduce overfitting, we employ several training strategies, including:

• Early Stopping to prevent unnecessary training once validation performance stops improving.
• Weighted Cross-Entropy Loss to mitigate the effects of class imbalance.
• Different learning rates for the CLIP encoder and the MLP classifier, allowing stable fine-tuning of the pre-trained backbone while enabling faster learning in the classification head.
• L2 Regularization (Weight Decay) to reduce overfitting by penalizing large model weights.
• Additional optimization and regularization techniques to improve training stability and model performance.

The objective of this stage is to evaluate how supervised fine-tuning improves classification performance compared to the zero-shot baseline and to establish a stronger benchmark for medical image classification using Vision-Language Models.

The table below summarizes the Fine-Tuning performance of CLIP ViT-B/16 on the four MedMNIST datasets. Performance is reported in terms of Accuracy and Area Under the ROC Curve (AUC).

The table summarizes the supervised fine-tuning performance of CLIP ViT-B/16 on the four MedMNIST datasets. Compared to the zero-shot approaches, fine-tuning significantly improves performance across all datasets, demonstrating the effectiveness of adapting the pre-trained CLIP image encoder to medical imaging tasks.

• OCTMNIST achieved the best overall performance, with an Accuracy of 89.20% and an AUC of 99.12%. These results indicate that the fine-tuned model is highly effective at learning discriminative features for retinal OCT image classification.

• DermaMNIST also achieved strong performance, reaching an Accuracy of 83.94% and an AUC of 97.28%. This suggests that fine-tuning enables the model to capture the subtle visual characteristics of different skin lesions.

• OrganMNIST obtained an Accuracy of 81.54% and an AUC of 98.18%, demonstrating that the model can effectively distinguish between different anatomical structures in CT images after supervised adaptation.

• TissueMNIST was the most challenging dataset, achieving an Accuracy of 67.43% and an AUC of 93.39%. Although its accuracy is lower than the other datasets, the high AUC indicates that the model still learns meaningful representations and has strong class discrimination capability.

Overall, fine-tuning substantially outperforms the zero-shot and BLIP-assisted zero-shot approaches. The combination of supervised learning, selective fine-tuning of the CLIP image encoder, and regularization techniques—including early stopping, weighted loss, differential learning rates, and L2 regularization—allows the model to effectively adapt to the medical domain while maintaining strong generalization performance. These results demonstrate that task-specific fine-tuning is essential for achieving high-performance medical image classification with Vision-Language Models.

In the third stage of this project, we combine the strengths of BLIP-Base and CLIP ViT-B/32 to improve zero-shot medical image classification.

First, the BLIP model generates a natural language caption describing the input medical image. This caption is then used as the textual input to the CLIP text encoder, while the original image is simultaneously processed by the CLIP vision encoder. The image and text embeddings are compared to perform zero-shot classification.

As in the previous experiments, we apply image preprocessing and enhancement techniques before inference to improve image quality and emphasize important medical features. We also evaluate different prompt formulations and combine them with the BLIP-generated captions to obtain more informative textual representations.

The overall pipeline is as follows:

1. Apply image preprocessing and enhancement.
2. Generate an image caption using BLIP-Base.
3. Encode the generated caption using the CLIP ViT-B/32 text encoder.
4. Encode the corresponding image using the CLIP ViT-B/32 vision encoder.
5. Compute image-text similarities and perform zero-shot classification.

The table presents the zero-shot classification performance of the BLIP-Base + CLIP ViT-B/32 pipeline on the four MedMNIST datasets. The results indicate that incorporating BLIP-generated captions does not consistently improve zero-shot performance across all datasets.

In the fourth stage of this project, we replace the general-purpose BLIP-Base captioning model with MediBLIP-Base, a vision-language model that has been fine-tuned on medical datasets. The motivation is to generate more accurate and domain-specific image captions that better describe medical findings and anatomical structures.

The overall pipeline is similar to Part 3. The pipeline consists of the following steps:

1. Apply image preprocessing and enhancement.
2. Generate a medical image caption using MediBLIP-Base.
3. Encode the generated caption using the CLIP ViT-B/32 text encoder.
4. Encode the corresponding image using the CLIP ViT-B/32 vision encoder.
5. Compute image-text similarities and perform zero-shot classification.

Unlike the previous stage, which relies on captions generated by a general-domain model, this approach leverages MediBLIP, whose captions are tailored to medical images. The objective is to evaluate whether domain-specific captions provide richer semantic information and improve zero-shot classification performance compared to both manually designed prompts and the general-purpose BLIP-based approach.

The table presents the zero-shot classification performance of the BLIP-Base + CLIP ViT-B/32 pipeline on the four MedMNIST datasets. The results indicate that incorporating BLIP-generated captions does not consistently improve zero-shot performance across all datasets.

• DermaMNIST achieved the highest classification accuracy (45.58%) among the evaluated datasets. However, its performance is noticeably lower than the zero-shot CLIP baseline, suggesting that the automatically generated captions do not provide sufficiently discriminative information for skin lesion classification.

• TissueMNIST obtained the highest AUC (0.6427) despite having a relatively low accuracy (18.70%). This indicates that the model has a moderate ability to distinguish between tissue classes, even though its final classification accuracy remains limited.

• OCTMNIST achieved an accuracy of 33.10% with an AUC of 0.5865, demonstrating moderate performance but still falling short of the CLIP-only zero-shot baseline.

• OrganMNIST was the most challenging dataset, achieving the lowest accuracy (10.88%) and an AUC of 0.4875, which is close to random performance.