Deep Learning for Brain Tumor Classification:

CNN Models with Saliency Map Insights

Abstract—This paper aims to classify brain tumors using Con- ”why” behind a prediction is critical. A robust solution must
volutional Neural Networks (CNNs) applied to MRI images. We therefore balance high classification accuracy with explainabil-
employed four models: a custom CNN, VGG16, ResNet101, and ity, providing insights into the model’s rationale that clinicians
a hybrid model combining VGG16 and ResNet101, with saliency
maps used for model interpretability. The custom CNN achieved can rely upon.
the highest accuracy of 98.01%, followed by VGG16 (96.33%),
ResNet101 (90%), and the hybrid model (97.10%). Saliency maps B. Research Goal
provided insights into the features driving predictions, enhancing This study aims to develop a reliable and interpretable
model transparency. Our findings demonstrate the effectiveness model for brain tumor classification that leverages CNN archi-
of CNNs for brain tumor classification, with the custom CNN
performing the best, and suggest further research into model tectures and XAI techniques. To achieve this, we will imple-
explainability, data augmentation, and hybrid techniques to ment four distinct models—a custom CNN, a VGG16-based
improve accuracy and generalizability. model, a ResNet101-based model, and a hybrid of ResNet101
Index Terms—Brain tumor classification, Convolutional Neu- and VGG16—and evaluate their performance on classification
ral Networks (CNNs), MRI imaging, deep learning, VGG16, accuracy and interpretability. Saliency maps will be employed
ResNet101, Hybrid model, Saliency Map.
to enhance model transparency by highlighting the specific
areas in MRI images that contribute most significantly to clas-
I. I NTRODUCTION
sification decisions. This study intends to identify the model
Brain tumor classification using medical images, particu- that best balances accuracy and explainability, contributing to
larly MRI scans, has become a significant area of research, the broader goal of advancing AI-driven diagnostic tools in
leveraging deep learning models for enhanced diagnostic ac- medical imaging.
curacy. The use of Convolutional Neural Networks (CNNs)
has proven effective in distinguishing between tumor and II. RELATED WORK
non-tumor regions, providing automated systems that assist Brain tumor classification using deep learning has gained
medical professionals in faster diagnosis [1]. However, de- significant attention due to its ability to assist radiologists
spite their accuracy, CNNs often operate as ”black boxes,” in diagnosing and predicting tumor types from medical im-
lacking transparency, which hinders their practical application ages. Recent advancements in Convolutional Neural Networks
in clinical settings where understanding the reasoning behind (CNNs) and hybrid models have shown promising results
a model’s decision is crucial [2]. in accurately classifying brain tumors, such as gliomas and
This study investigates four models: a custom CNN, a meningiomas. However, one of the major challenges in ap-
VGG16-based model, a ResNet101-based model, and a hybrid plying deep learning models in the medical field is the lack
model combining ResNet101 and VGG16. By incorporating of interpretability and explainability of the model’s decision-
Explainable AI (XAI) techniques, particularly saliency maps, making process. This has led to the integration of Explainable
the research aims to provide visual explanations for model AI (XAI) techniques like saliency maps to enhance model
predictions, thus improving interpretability. This not only aids transparency and improve trustworthiness.
in achieving high accuracy but also ensures that clinicians can The study by Abiwinanda et al. [4] presents a Convolutional
trust and understand the model’s decision-making process [3]. Neural Network (CNN) for brain tumor classification using
MRI scans, achieving a high accuracy of 94%. The model
A. Problem Statement categorizes tumors into glioma, meningioma, and pituitary
The classification of brain tumors from MRI images is tumor classes, showcasing CNN’s ability to automatically
fraught with challenges stemming from the diversity in tu- extract features and classify tumors effectively. However, the
mor characteristics, including variations in size, location, study does not address model interpretability, which is crucial
and morphology. Traditional diagnostic methods are resource- in clinical settings. Incorporating explainable AI techniques,
intensive, requiring expert radiologists to spend significant such as saliency maps, could enhance understanding of the
time analyzing complex MRI data. The manual nature of these model’s decision-making process. Despite the challenge posed
processes often limits throughput and can lead to inconsis- by tumor appearance variations, the paper demonstrates CNN’s
tencies across practitioners. Although deep learning models potential in medical imaging.
can provide a faster alternative, models lacking interpretability The authors [5] proposes the use of Convolutional Neural
hinder adoption in clinical settings where understanding the Networks (CNNs) for brain tumor classification. The authors
highlighted the potential of CNNs in classifying MRI images TABLE I
of brain tumors into categories such as glioma, meningioma, DATASET D ISTRIBUTION
and pituitary tumor. Their method uses several pre-processing Class Training Data Test Data Total Data
techniques, including noise removal and feature enhancement, Pituitary 1,457 300 1,757
before applying the CNN model. The experimental results No Tumor 1,595 405 2,000
Meningioma 1,339 306 1,645
showed promising accuracy, especially when tested with MRI Glioma 1,321 300 1,621
datasets, proving CNN’s effectiveness in automating brain Total 5,712 1,311 7,023
tumor detection. This approach could potentially assist doctors
in early diagnosis, improving patient outcomes.
The paper [6] combines CNNs with MRI imaging for brain images, leveraging their unique strengths in feature extraction.
tumor classification. It highlights the use of pre-processed MRI The architecture also incorporates a saliency map technique
scans and feature extraction techniques, showing that CNNs for model interpretability, highlighting the important regions
significantly outperform traditional methods in detecting var- of the MRI images used for classification. The models are
ious types of brain tumors. This study underscores the value evaluated using standard metrics such as accuracy, sensitivity,
of MRI scans in aiding clinical diagnosis. and specificity to ensure their effectiveness.
The study [7] examines different CNN architectures for The overall process is illustrated in Figure 1, which outlines
classifying brain tumors from MRI images. It focuses on the complete architecture
pre-processing methods like data augmentation and noise
reduction, and the fine-tuning of hyperparameters for optimal
CNN performance. The paper also addresses challenges such
as class imbalance and overfitting, providing insights into
CNN optimization for better diagnosis.
The paper [8] presents an automatic system for classifying
brain tumors using CNNs. The end-to-end system processes
raw MRI data, extracts features, and classifies tumors as
benign or malignant. Evaluated on multiple datasets, the
model outperforms traditional machine learning algorithms,
demonstrating CNNs’ potential for automating tumor detection
and improving diagnostic accuracy.
The study [9] explores the use of deep learning mod-
els, particularly CNNs, for brain tumor classification from
MRI images. It highlights the importance of preprocessing
techniques like image normalization and contrast adjustment
for improving model performance. The study also addresses
challenges such as data scarcity and the need for larger
Fig. 1. Proposed Methodology
datasets, concluding that deep learning shows promise for real-
time tumor diagnosis.
The study [10] investigates a hybrid model combining B. Dataset
CNNs with other machine learning techniques like Random The dataset [11] used for this project comprises a total of
Forest for improved brain tumor classification. Evaluated on 7,023 brain MRI images, drawn from three different sources:
datasets from multiple medical institutions, the model demon- the Figshare dataset [12], the SARTAJ dataset [13], and the
strates strong generalization across diverse data. The authors Br35H dataset [14]. The images are categorized into four
stress the importance of integrating deep learning with clinical classes—glioma, meningioma, pituitary tumors, and no tumor.
practices for early tumor detection, providing valuable insights Notably, the ”no tumor” class specifically contains images
for advancing AI-based diagnostic tools. sourced from the Br35H dataset. This diverse collection sup-
III. RESEARCH METHODOLOGY ports robust model training and evaluation across distinct brain
tumor types as well as non-tumor cases. In table 1 we can see
A. System Architecture of Proposed Model the distribution of images. In Figure 2, we present sample
The system architecture for the proposed model follows images from each category, showcasing the diversity of the
a structured flow, starting with data collection, where MRI dataset across different brain tumor types and non-tumor cases.
images are processed as inputs. The data undergoes prepro-
cessing, involving steps such as resizing, normalization, and
augmentation to ensure robustness. Integrates four distinct C. Preprocessing
approaches: a standard CNN model, a VGG16-based model, In order to enhance the performance and accuracy of the
a ResNet101-based model, and a hybrid model combining models, several data preprocessing techniques were applied to
ResNet101 and VGG16. Each model is trained to classify MRI the dataset. These techniques aim to standardize the images
 E. Custom CNN
The custom CNN model designed for brain tumor classi-
fication comprises a sequence of convolutional, pooling, and
dense layers that facilitate hierarchical feature extraction and
classification. The architecture begins with a convolutional
Fig. 2. Sample images from each class in the brain MRI dataset.
layer utilizing 32 filters of size 3×3 with ReLU activation,
enabling the network to learn initial low-level features such
as edges and textures. Subsequent layers increase the filter
count to 64 and then to 128, enhancing the model’s capacity
to capture more complex and abstract features in deeper layers,
which is particularly useful for distinguishing nuanced patterns
in tumor images.
Fig. 3. Sample images after preprocessing
Each convolutional layer performs a discrete convolution
operation, defined by:
and increase the model’s robustness by creating diverse rep- M −1 N −1
!
(k)
X X
resentations of the training data. fi,j = σ (k)
xi+m,j+n · wm,n + b(k) (1)
• Resizing Images: Each image is resized to 224x224 m=0 n=0
pixels. This ensures that all input images have consistent (k)
dimensions, which is important for feeding them into a where fi,j represents the feature map at location (i, j) for the
neural network. Consistent image size also reduces the k-th filter, x is the input feature map, w is the filter of size
computational complexity. M ×N , b is the bias term, and σ denotes the activation function
• Grayscale Conversion: The images are converted to (ReLU in this case). This operation enables each convolutional
grayscale to reduce the computational complexity and layer to focus on localized features across the input images,
focus on the structural features of the banknotes, as color allowing the model to learn patterns critical for brain tumor
information might not be critical for distinguishing real detection.
from fake notes. Each convolutional layer is followed by a max-pooling
• Normalization: The pixel values of the images are nor- layer, which reduces the spatial dimensions of the feature maps
malized to a range of [0, 1] by dividing by 255. This helps while retaining important features and discarding redundant
in speeding up the convergence of the training process ones. This not only reduces computational cost but also helps
and stabilizes the learning rate. prevent overfitting, making the model more robust to dataset
• Data Augmentation: We applied several augmentation variations. After the final convolutional block, the feature maps
techniques to artificially expand the dataset and improve are flattened into a one-dimensional vector for processing in
the robustness of the model. The following augmentations the fully connected layers.
were applied: A dense layer with 512 neurons and ReLU activation is
– Random rotation by a range of [-10, 10] degrees. applied to refine the features, followed by a dropout layer (0.5
– Random horizontal flip with a 50 rate) to reduce overfitting by randomly deactivating neurons.
– Random zoom within a range of [0.9, 1.1]. The model ends with a softmax output layer for multi-class
classification. Using categorical cross-entropy loss and the
These augmentations allow the model to learn better
Adam optimizer ensures efficient learning and convergence.
generalizations by exposing it to different variations of
This architecture captures both low- and high-level features,
the images.
enhancing classification performance and generalization for
In Fig. 3, we show some images after applying the prepro- tumor detection.
cessing steps mentioned above, including resizing, grayscale
conversion, and data augmentation. F. VGG16 Based Model
D. Classification Model The VGG16-based model starts with the pre-trained VGG16
We focus on the various deep learning architectures imple- architecture, originally trained on the ImageNet dataset. This
mented to perform accurate classification. For this study, we approach leverages pre-trained weights to utilize the model’s
utilized four distinct models: a custom Convolutional Neural ability to recognize essential low-level features such as edges
Network (CNN) tailored specifically for our dataset, a VGG16- and textures, which are critical for distinguishing different
based model, a ResNet101-based model, and a hybrid model types of brain tumors. By using the base VGG16 model
that combines VGG16 and ResNet101 layers. Each model was without the top classification layers, we can build a custom
chosen for its unique architecture, with the aim of evaluating classification head tailored to our specific task, enhancing its
and comparing their strengths in terms of feature extraction accuracy and relevance for brain tumor classification.
and classification accuracy across our specific classification To further optimize the model, we freeze the weights of
task. the initial layers of the VGG16 network, except for the last
15 layers. This strategy preserves the feature extraction capa- Where FVGG and FResNet are the feature vectors from the two
bilities learned from the extensive ImageNet dataset, reducing base models. By combining these features, the model benefits
computational load and preventing overfitting by keeping these from the distinct strengths of both networks—VGG16’s ability
layers fixed during training. By fine-tuning only the last few to capture finer low-level features and ResNet101’s ability to
layers, the model can adapt to the specific characteristics of learn more complex, hierarchical patterns. After concatenating
brain tumor images more effectively and efficiently. We add a the feature vectors from VGG16 and ResNet101, several fully
custom classification head to the network, starting with a Glob- connected (dense) layers are added with 512, 256, and 128
alAveragePooling2D layer to reduce the spatial dimensions of neurons. These layers refine the extracted features by learning
the base model’s output while retaining crucial information. complex combinations and reducing the number of neurons
This simplifies the model and helps prevent overfitting. Next, helps regularize the model, preventing overfitting while main-
dense layers with progressively fewer neurons (256, 128, 64) taining sufficient capacity. Dropout layers with a rate of 0.5
and higher dropout rates (0.6, 0.5, 0.4) are added to regularize further mitigate overfitting by randomly deactivating neurons
the model, promoting reliance on diverse neuron combinations. during training, encouraging better generalization.
The output layer uses a softmax activation function to predict This hybrid model combines the strengths of both VGG16
the probability of each class (glioma, meningioma, pituitary and ResNet101 for improved feature extraction, helping the
tumors, or no tumor), ensuring accurate multi-class classifica- model generalize better and achieve higher classification accu-
tion. This approach combines transfer learning, regularization, racy, particularly for brain tumor classification, which requires
and fine-tuning to improve brain tumor classification. both low-level and high-level features for accurate diagnosis.
The final output layer uses a softmax activation function
G. ResNet101 Based Model
to provide class probabilities for multi-class classification,
The ResNet101-based model utilizes transfer learning, start- ensuring the sum of the predicted probabilities equals one.
ing with pre-trained weights from ImageNet to leverage the The model is compiled with the Adam optimizer, using a low
feature extraction capabilities of the pre-trained layers. By learning rate of 1 × 10−5 , ensuring smooth convergence and
freezing the majority of the convolutional layers and only preventing overshooting.
training the last 30 layers, the model retains learned features
from ImageNet and focuses on adapting the higher-level layers I. Saliency Map (XAI)
to the specific brain tumor classification task. This approach To better understand the explainability of the model’s pre-
saves computational resources and prevents overfitting, espe- dictions, we applied saliency maps. These maps highlight the
cially when working with a smaller dataset. regions in the input images that are most important for the
A custom classification head is added with Global Average model’s decision-making process [15]. By emphasizing the
Pooling, reducing the feature map dimensions while retain- crucial areas, saliency maps help elucidate the features the
ing the most critical information. The dense layer with 256 model focuses on for each prediction. The saliency map is
neurons and ReLU activation helps refine the features, while generated by computing the gradients of the predicted class
a dropout rate of 0.7 is applied to minimize overfitting. This score with respect to the input image pixels, revealing the
dropout helps the model generalize better by randomly deacti- pixels with the most significant impact on the model’s output.
vating neurons during training. The final softmax output layer
provides the predicted class probabilities. The Adam optimizer IV. RESULT AND ANALYSIS
with a low learning rate (1e-5) ensures stable convergence and A. Performance Evaluation
adaptive learning, contributing to improved performance and Our model was validated by the available training and test
generalization for classifying brain tumors. subsets that were predefined. The training subset was em-
H. Hybrid Model ployed to optimize the model’s parameters during the training
The hybrid model combines two powerful pre-trained ar- phase, while the testing subset remained separate and was
chitectures, VGG16 and ResNet101, both initialized with exclusively reserved for evaluating the model’s performance
ImageNet weights. To prevent overfitting and improve gener- post-training. Multiple performance metrics including accu-
alization, the convolutional layers of both models are frozen, racy, precision, and recall, were used to determine the model’s
reducing the number of trainable parameters and speeding performance These metrics give insight into various aspects of
up the training process. The outputs from both models are the model’s predictive capabilities, and with their help, we are
passed through Global Average Pooling (GAP), which reduces able to perform a detailed assessment of the model’s overall
spatial dimensions while preserving relevant information, es- performance. We analyze the performance of the trained
pecially useful for smaller datasets. The pooled features from deep-learning models based on the aforementioned evaluation
VGG16 and ResNet101 are then concatenated, combining their metrics. We interpret the results to identify trends, strengths,
strengths to capture diverse feature representations for brain and areas for improvement in each model’s performance.
tumor classification. Mathematically, the concatenation can be B. Model performance
expressed as:
The custom CNN model achieved the highest accuracy of
Fhybrid = Concatenate (FVGG , FResNet ) (2) 98.01%, thanks to its architecture specifically tailored for this
 Fig. 4. Confusion Matrix of CNN Model Fig. 6. Confusion Matrix of ResNet101 Based Model

Fig. 5. Training and Validation Performance Curves of VGG16 Based Model

dataset. This design enabled it to capture unique patterns

in brain tumor images, resulting in fewer misclassifications,
as seen in the confusion matrix in Figure 4. Its specialized Fig. 7. Confusion Matrix of Hybrid Model
approach minimized overfitting and offered superior task-
specific optimization.
The VGG16-based model followed with 96.33% accuracy. C. Explainable AI (Saliency Map)
While it captured fine-grained details and low-level features Fig. 8 shows a saliency map visualization that helps us
well, its general-purpose architecture lacked the task-specific understand the areas of a brain scan that are most important
focus of the custom CNN. The performance curves in Figure 5 to the model’s prediction. The image consists of three panels,
show effective learning with minimal overfitting, demonstrat- each providing different views of the brain scan. The first
ing its generalization ability. panel displays the original grayscale brain scan, likely an MRI,
The ResNet101 model achieved 90% accuracy, the lowest which provides detailed structural information of the brain.
of the four. Although its deeper architecture aids feature The second panel shows the raw saliency map, where the
extraction, it led to overfitting due to the smaller dataset, as image is predominantly blue, with brighter regions in greenish-
shown in Figure 6’s confusion matrix. Its depth introduced yellow indicating areas that are more significant to the model.
noise, reducing its effectiveness for this task. These brighter areas highlight the parts of the scan that the
The hybrid model, combining VGG16 and ResNet101, model focuses on when making its predictions.
reached 97.10% accuracy. While it benefited from both mod- The final panel combines the original scan with the saliency
els’ strengths, the combination created redundant features, map, highlighting the key regions of interest in green. The
preventing it from surpassing the custom CNN. Figure 7 shows brightest greenish-yellow regions represent the areas the model
balanced performance across classes, but the hybrid approach considers most important, while the darker green and blue
lacked the focused optimization of the custom model. regions are less significant. This overlay helps visualize the
Table 2 summarizes the performance metrics of all models, model’s focus areas, making it clear which parts of the brain
highlighting their comparative strengths and limitations in are most relevant for the model’s decision-making.
brain tumor classification. By using saliency maps, we can gain insights into the
model’s reasoning and understand which areas of the brain

TABLE II
P ERFORMANCE C OMPARISON OF D IFFERENT M ODELS

Model Accuracy (%) Precision Recall F1-Score

Custom CNN 98.01 0.9807 0.9775 0.9790
VGG16-based Model 96.33 0.965 0.964 0.963
ResNet101-based Model 90.00 0.901 0.891 0.894
Hybrid Model 97.10 0.9696 0.9694 0.9691 Fig. 8. Saliency Map
scan are most influential in its predictions. This technique [10] P. S. Anand, A. K. Pandey, and D. P. Gupta, ”Deep Learning-Based Brain
enhances interpretability and can be particularly useful in Tumor Classification,” International Journal of Medical Engineering and
Informatics, vol. 8, no. 4, pp. 134-145, 2021.
medical imaging for identifying crucial features relevant to [11] M. Nickparvar, ”Brain Tumor MRI Dataset,” Kaggle, 2020. [Online].
diagnosis. Available: https://www.kaggle.com/datasets/masoudnickparvar/brain-
I see, thank you for clarifying. Here’s the revised conclusion tumor-mri-dataset
[12] ”Brain Tumor Dataset,” Figshare, 2015. [Online]. Available:
based on your work on brain tumor classification using four https://figshare.com/articles/dataset/brain tumor dataset/1512427
models: [13] S. Bhuvaji, ”Brain Tumor Classification (MRI),” Kaggle, 2021. [On-
line]. Available: https://www.kaggle.com/datasets/sartajbhuvaji/brain-
V. C ONCLUSION tumor-classification-mri
[14] A. Hamada, ”Brain Tumor Detection,” Kaggle, 2021. [Online].
This study demonstrates the effectiveness of convolutional Available: https://www.kaggle.com/datasets/ahmedhamada0/brain-
neural networks (CNNs) for brain tumor classification, with tumor-detection
our custom CNN model achieving the highest accuracy among [15] K. Simonyan, A. Vedaldi, and A. Zisserman, ”Deep Inside Convolu-
tionalNetworks: Visualising Image Classification Models and Saliency
the four models tested. Our results show that deep learning Maps,”arXiv preprint arXiv:1312.6034, 2013.
can significantly aid in medical imaging, particularly in the
accurate diagnosis of brain tumors.
We incorporated saliency maps into our evaluation, provid-
ing valuable insights into the features driving model predic-
tions. This enhances interpretability and trust in AI applica-
tions, crucial for clinical adoption. The use of explainable AI
methods like saliency maps adds transparency, ensuring the
reliability of the models in sensitive healthcare contexts.
Future research could explore multimodal approaches com-
bining imaging data with other medical information, as well
as transfer learning techniques for broader applicability. Per-
sonalized systems tailored to individual patients’ needs could
further improve classification accuracy and patient care.
In summary, our work highlights the potential of AI in brain
tumor classification, demonstrating both high performance and
the importance of interpretability in medical AI. Ethical con-
siderations must remain a priority to ensure AI technologies
benefit all patients equitably.
R EFERENCES
[1] S. R. Sahu, A. M. Goh, and C. A. H. A. Chan, ”Deep learning for brain
tumor classification in MRI images: A review,” IEEE Access, vol. 8, pp.
105071-105086, June 2020.
[2] X. Zhang, Y. Wang, and J. Chen, ”Interpretable deep learning for medical
image analysis: A survey,” IEEE Transactions on Neural Networks and
Learning Systems, vol. 32, no. 12, pp. 5594-5610, Dec. 2021.
[3] A. K. Gupta, N. Agrawal, and S. Shah, ”Explainable AI techniques
for interpreting brain tumor classification models: A case study with
saliency maps,” IEEE Journal of Biomedical and Health Informatics,
vol. 25, no. 4, pp. 1102-1110, April 2021.
[4] N. Abiwinanda, M. Hanif, S. T. Hesaputra, A. Handayani, and T.
R. Mengko, ”Brain Tumor Classification Using Convolutional Neural
Network,” in World Congress on Medical Physics and Biomedical
Engineering 2018, L. Lhotska, L. Sukupova, I. Lacković, and G. S.
Ibbott, Eds., vol. 68/1. Singapore: Springer, 2019, pp. 155–158.
[5] M. A. Raza, S. M. S. Rehman, and M. S. Alam, ”Brain Tumor
Classification Using Convolutional Neural Network,” in Proceedings
of the International Conference on Advanced Computer Science &
Applications, pp. 71-76, 2019.
[6] H. Ali, A. M. S. Ali, and S. Zubair, ”Brain Tumor Classification Using
CNN and MRI Imaging,” Int. J. Computer Science and Information
Security, vol. 18, no. 3, pp. 98-105, 2020..
[7] S. Kumar, S. K. Gupta, and R. P. Verma, ”Convolutional Neural Network
for Brain Tumor Classification: A Review,” International Journal of
Engineering and Technology, vol. 7, no. 1, pp. 56-63, 2021.
[8] M. T. Alam, M. F. Anwar, and M. S. Mollah, ”Automatic Brain Tumor
Classification Using CNN,” Journal of Healthcare Engineering, vol.
2020, pp. 1-12, 2020.
[9] S. S. Roy, A. K. Gupta, and P. Bandyopadhyay, ”MRI Brain Tumor
Classification Using Deep Learning Models,” Journal of Medical Imag-
ing and Health Informatics, vol. 10, no. 5, pp. 1059-1066, 2020.