MEDICINE RECOMENDATION SYSTEM:THE FUTURE HEALTH CARE

A PROJECT WORK REPORT

Submitted by

GAUTHAMKUMAR.J (113121UG03027)

ARUNRAJ.K (113121UG03013)

NAVEEN.R (113121UG03074)

In partial fulfilment for the award of the degree of

BACHELOR OF ENGINEERING
IN
COMPUTER SCIENCE AND ENGINEERING
VEL TECH MULTI TECH Dr. RANGARAJAN Dr. SAKUNTHALA
ENGINEERING COLLEGE, ALAMATHI ROAD, AVADI, CHENNAI-62

(AN AUTONOMOUS INSTITUTION)

ANNA UNIVERSITY, CHENNAI 600 025.

JULY 2024
 ANNA UNIVERSITY, CHENNAI 600 025
BONAFIDE CERTIFICATE

Certified that this project report of title “MEDICINE RECOMENDATION

SYSTEM:THE FUTURE HEALTH CARE” is the bonafide work of
GAUTHAMKUMAR.J(113121UG03027),ARUNRAJ.K(113121UG03013),NAVEEN.R
(113121UG03074)who carried out the project work under my supervision.

SIGNATURE SIGNATURE
HEAD OF THE DEPARTMENT SUPERVISOR
Dr.R.Saravanan, B.E, M.E(CSE)., Ph.D. Mrs.V.Helen Sathiya,MCA,M.Tech(CSE).
PROFESSOR, ASSISTANT PROFESSOR,
Department of Computer Science and Department of Computer Science and
Engineering, Engineering,
Vel Tech Multi Tech Dr. Rangarajan Vel Tech Multi Tech Dr. Rangarajan
Dr. Sakunthala Engineering College, Dr. Sakunthala Engineering College,
Avadi, Chennai-600 062 Avadi, Chennai-600 062
 CERTIFICATE FOR EVALUATION

This is to certify that the project entitled “MEDICINE RECOMMENDATION

SYSTEM: THE FUTURE HEALTH CARE” is the bonafide record of work done
by following students to carry out the project work under our guidance during the
year 2023-2024 in partial fulfilmentfor the award of Bachelor of Engineering
degree in Computer Science and Engineering conducted by Anna University,
Chennai.

GAUTHAMKUMAR.J (113121UG03027)

ARUNRAJ. (113121UG03013)

NAVEEN.R (113121UG03074)

This project report was submitted for viva voce held on

At Vel Tech Multi Tech Dr. Rangarajan and Dr.Sakunthala Engineering College.

INTERNAL EXAMINER EXTERNAL EXAMINER

 ACKNOWLEDGEMENT
We wish to express our sincere thanks to Almighty and the people who extended their
help during the course of our work. We are greatly and profoundly thankful to our
honourable Chairman, Col. Prof.Vel. Shri Dr.R.Rangarajan B.E.(ELEC), B.E.
(MECH), M.S.(AUTO)., D.Sc., &Vice Chairman, Dr.Mrs.Sakunthala Rangarajan
M.B.B.S., for facilitating us with this opportunity. We also record our sincere thanks
to our honorable Principal, Dr.V.Rajamani M.E.,Ph.D., for his kind support to take up
this project and complete it successfully. We would like to express our special thanks
to our Head of the Department, Dr.R.Saravanan, B.E, M.E(CSE)., Ph.D. Department
of Computer Science and Engineering and our project supervisor Mrs.V.Helen
Sathiya,MCA,M.Tech(CSE). for their moral support by taking keen interest on our
project work and guided us all along, till the completion of our project work and also
by providing with all the necessary information required for developing a good system
with successful completion of the same.Further, the acknowledgement would be
incomplete if we would not mention a word of thanks to our most beloved Parents for
their continuous support and encouragement all the way through the course that has
led us to pursue the degree and confidently complete the project work.

(GAUTHAMKUMAR.J) (ARUNRAJ.K) (NAVEEN.R)

 ABSTRACT

“ICU PATIENT RISK LEVEL MONITORING SYSTEMUSING SUPERVISED

LEARNING APPROACHES” is a project that aims to develop Modern Intensive Care Units
(ICUs) provide continuous monitoring of critically ill patients susceptible to many
complications affecting morbidity and mortality. ICU settings require a high staff-to-patient
ratio and generates a sheer volume of data. For clinicians, the real-time interpretation of data
and decision-making is a challenging task. Machine Learning (ML) techniques in ICUs are
making headway in the early detection of high-risk events due to increased processing power
and freely available datasets such as the Medical Information Mart for Intensive Care
(MIMIC). Techniques in ICU settings using MIMIC data. We assembled the qualified articles
to provide insights into the areas of application, clinical variables used, and treatment
outcomes that can pave the way for further adoption of this promising technology and possible
use in routine clinical decision-making. The lessons learned from our review can provide
guidance to researchers on application of ML techniques to increase their rate of adoption in
healthcare. The ICU patient risk level monitoring system plays a crucial role in improving
patient safety, optimizing resource allocation, and enhancing clinical decision-making in
intensive care settings. By continuously monitoring and analysing patient data, it provides
valuable insights that help healthcare providers intervene promptly and prevent adverse
outcomes.

Keywords: intensive care unit, critical care, MIMIC, machine learning.

 TABLE OF CONTENTS

CHAPTER NO. TITLE PAGE NO.

I ABSTRACT I
II LIST OF TABLES II
III LIST OF FIGURES III
IV LIST OF ABBREVIATIONS IV

S.NO TOPIC PAGE NO.

.
1 INTRODUCTION 1
1.1 AIM 2
1.2 PROJECT SCOPE 2
1.3 DESCRIPTION 3
1.4 LITERATURE SURVEY 4
2. SYSTEM ANALYSIS 5
2.1 EXISTING SYSTEM 6
2.1.1 Disadvantages 6
2.2 PROPOSED SYSTEM 6
2.2.1 Advantages 7
2..3 PRELIMINARY INVESTIGATION 11
2.4 FEASIBILITY STUDY 12
2.4.1 Economical feasibility 12

2.4.2 Technical feasibility 12

3. SYSTEM IMPLEMENTATION 14
3.1 HARDWARE REQUIREMENTS 15
3.2 SOFTWARE REQUIREMENTS 15
4.3 ARCHITECTURE DESIGN 15
I
4. SYSTEM DESIGN 17
4.1 SYSTEM DESIGN 18
4.1.1 Input Design 18
4.1.2 Output Design 19
4.2 FEATURES OF OPENCV 19
4.3 PYCHARM 20
4.4 TECHNOLOGIES USED 21
4.5 DATA FLOW DIAGRAM 22
4.6 UNIFIED MODELLING LANGUAGE 23
4.6.1 Architecture Diagram 23
4.6.2 Sequence Diagram 23
4.6.3 Class Diagram 25
4.6.4 Activity Diagram 26
4.6.5 Collaboration Diagram 27
4.6.6 Entity Relationship Diagram 28
4.6.7 Use Case Diagram 29
4.7 MODULES DESCRIPTION 30
4.7.1 Preprocess Data 20
4.7.2 Clean and Normalize Data 32
4.7.3 Splitting the Data 21
4.7.4 Data Augmentation 21
4.7.5 Model Plotting 37
4.7.6 Running and Saving the Model 22
4.8 ALGORITHMS USED 39
5. CONCLUSION 23
5.1 CONCLUSION 42

II
6. FUTURE ENHANCEMENTS 43
6.1 FUTURE ENHANCEMENTS 44
APPENDIX I – IMPLEMENTATIO 45
APPENDIX II- OUTPUT 49
REFERENCES 52
LIST OF TABLES

FIGURE NO. NAME PAGE NO.

3.1 Hardware Requirements 12
3.2 Software Requirements 12

TABLE OF FIGURES
FIGURE NO. NAME PAGE NO.
4.1 Data Flow Diagram 15
4.2 Architecture Diagram 15
4.3 Sequence Diagram 16
4.4 Class Diagram 17
4.5 Activity Diagram 17
4.6 Collaboration Diagram 18
4.7 Entity Relationship Diagram 19
4.8 Use Case Diagram 19

A2.1 Variation of Accuracy 31

A2.2 Variation of Loss 32
A2.3 Identification of Pill 33

III
 LIST OF ABBREVATIONS

ABBREVATIONS FULL FORM

CNN Convolutional Neural Network
SVM Support Vector Machine
OpenCV Open-source Computer Vision

IV
 CHAPTER 1
INTRODUCTION

1
1.1 AIM
The goal of this project is to create an advanced pill identification system that uses
machine learning and OpenCV to precisely identify pills based on their visual attributes,
such as colour, shape, and markings. The system seeks to improve classification
resilience and accuracy by using pre-trained CNN models such as ResNet, MobileNet,
and EfficientNetB0. In addition, the system aims to simplify the identification process
by incorporating the model into an intuitive online application interface, making it
simple for consumers and healthcare providers to use. In the context of Indian medicines
with a variety of pill properties, the ultimate objective is to enhance patient safety and
pharmaceutical quality control.

1.2 PROJECT SCOPE

Goal: By combining a multimodal CNN architecture with an intuitive web application

interface, the project seeks to create an advanced pill identification system.

Deliverables: The main goal is to create a working system that can recognize pills by
their visual characteristics. Via an online application, the system will provide a smooth
user experience, making pill identification for customers and healthcare providers more
efficient.

Tasks:

Data Gathering and Preparation: Build a varied collection of pill pictures that differ
in terms of colour, shape, and markings. To normalize incoming data, use OpenCV for
preprocessing operations including scaling, normalization, and noise removal.

Feature Extraction and Classification: Use pre-trained CNN models, such as

MobileNet, ResNet, and EfficientNetB0, for feature extraction and classification. This
will allow you to extract high-level features from pill photos. Take out qualities of the
pill, such its texture, colour, and form.

Multimodal Model Architecture: Combine the results from the MobileNet, ResNet,
and EfficientNetB0 models to create a multimodal architecture. Build a multi-layered
neural network to combine retrieved data and facilitate precise categorization.

Training and Evaluation: Using the labelled dataset, train the multimodal model
using transfer learning techniques. To guarantee accurate pill categorization, assess the
model's performance using measures such as accuracy, precision, recall, and F1-score.

2
Web Application Development: Create a front-end and back-end web application with
Flask, HTML, and CSS that is easy to use. Add the trained model to the online
application so that users may input pictures of pills and get identification results in real
time. Add features like interactive identification results display and drag-and-drop
picture upload to improve user experience.

Risks:

Training Data Quality: The diversity and quality of the training dataset have a
significant impact on the system's accuracy. The efficacy of the system may be
impacted by incomplete or skewed data.

Algorithm Complexity: Learning and identifying pill pictures may be difficult

because to the multimodal CNN architecture's complexity. For best results, the
algorithm must be continuously improved.

Timely Implementation: Several variables, including resource availability and

system complexity, may affect how quickly the project is implemented. To guarantee
on-time completion, the project timetable must be closely monitored and adjusted as
needed.

1.3. DESCRIPTION

The development of a strong pill identification system using OpenCV and machine
learning and integrating it with an intuitive web application interface is the focus of
this case study. Pills may be reliably identified by the system using visual
characteristics including form, colour, and markings. The technology seeks to improve
patient safety and expedite pharmaceutical quality control operations by automating
the identification process. The initiative provides a dependable solution for customers,
pharmacists, and healthcare professionals by addressing the issues raised by the varied
qualities of Indian tablets.

Methodology:

Gathering and Preparing Data: Compile a varied collection of pill pictures with a
variety of forms, colours, and markings. Preprocess the photos using OpenCV
methods for noise removal, scaling, and normalization to normalize the data.

3
Extracting and Categorizing Features: Use CNN models that have already been
trained, including ResNet, MobileNet, and EfficientNetB0, to extract high-level
characteristics from the pill pictures. Take note of pertinent details like the form,
colour, and texture of the pills.

Architecture for Multimodal Models: Create a multimodal architecture by merging

the MobileNet, ResNet, and EfficientNetB0 model outputs. Create a multi-layered
neural network to combine the gathered information and carry out precise
categorization.

Instruction and Assessment: Apply transfer learning techniques to the labelled

dataset in order to train the multimodal model. To guarantee accurate pill
identification, assess the model's performance using measures such as accuracy,
precision, recall, and F1-score.

Development of Web Applications: Create a user-friendly online application using the

frontend code written in HTML, CSS, and Flask. Provide real-time identification results
for users who upload pill photos by integrating the learned model into the online
application.
To improve user experience, provide elements like interactive picture recognition
results display and drag-and-drop image upload.

Key Steps in Pill Identification:

1.Data Gathering: Compile a varied collection of tablet photos.

2.Preprocessing: Use OpenCV methods to standardize the data. Employing CNN

models that have already undergone training, extract pertinent characteristics. Create a
multimodal architecture for feature fusion using the model architecture.

3.Training: Apply transfer learning strategies to train the model.

4.Assessment: Make use of assessment measures to gauge the model's effectiveness.

Develop a user-friendly web application interface through web application
development.

5.Integration: Incorporate the web application's learned model into it.

6.Testing: To guarantee precise pill recognition, test the system.

4
Benefits of the Pill Identification System :

1.Improved Patient Safety: By lowering the possibility of prescription mistakes,

accurate pill identification improves patient safety.

2.Streamlined Operations: Healthcare providers and pharmacists can save time and
dollars by automating the identification process.

3.Better Pharmaceutical Quality Control: Accurate identification guarantees that

industry requirements for quality control are followed in the pharmaceutical sector.

4.User-Friendly Interface: The online application promotes usability and

accessibility by giving consumers and healthcare professionals quick access to the
identification system.

5.Effective Decision-Making: Prompt and precise identification facilitates timely

decision-making in healthcare environments, which enhances patient care.

Challenges:

1. Diversity of Datasets: It can be difficult to compile a varied collection of pill

photos that accurately depicts the variety of forms, hues, and patterns available in
Indian tablets. The model's capacity to properly generalize may be hampered by the
scarcity of complete datasets.

2. Data Preprocessing Complexity: Careful thought must be given to preprocessing

pill photos using OpenCV methods in order to standardize the data. Finding the best
preprocessing methods and settings to guarantee accuracy and consistency in feature
extraction may be difficult.

3. Accuracy of Feature Extraction: Using pre-trained CNN models like MobileNet,

ResNet, and EfficientNetB0 to extract pertinent characteristics from pill photos may
not always be easy. For correct pill categorization, it is crucial to make sure that the
chosen models accurately reflect the form, colour, and texture of the pills.

5
4.Model Architecture Design: It might be difficult to create a multimodal
architecture that combines the outputs of several CNN models. Efficient integration of
characteristics derived by several models while preserving model performance and
efficiency necessitates meticulous architectural design and optimization.

5.Adaptation of Transfer Learning: There could be difficulties while using transfer

learning methods to train the multimodal model on the labelled dataset. It takes
rigorous testing and parameter modification to fine-tune pre-trained models to adapt to
the job of pill recognition without overfitting or underfitting.

6.Complexity of Web Application Development: Creating a user-friendly web

application interface with Flask, HTML, and CSS is not easy. online development and
design skills are needed to ensure the trained model integrates smoothly into the online
application while preserving security, scalability, and responsiveness.

7.Requirements for Real-time Identification: Using real-time pill identification in

an online application presents latency and computing resource issues. Providing a
smooth user experience requires optimizing the model inference process to achieve
prompt and accurate identification results while managing variable user loads.

8.Assessment Metric Selection: Selecting the right assessment metrics to gauge the
model's effectiveness might be difficult. Careful evaluation of the objectives and goals
of the application is necessary to balance criteria such as accuracy, precision, recall,
and F1-score to assess the model's efficacy in properly and consistently classifying pill
pictures.

1.4. LITERATURE SURVEY

1.4.1 An Overview of Computer Vision Techniques for Pill Identification:

MERIT: This survey offers a thorough summary of the several computer vision
methods that are applied to pill detection. It gives readers a firm basis for understanding
the complexities of pill identification systems by highlighting the significance of
grasping basic principles prior to using sophisticated strategies.

DEMERIT: Nevertheless, because of its wide reach, the survey could not include all
the information about the newest technology, which could restrict our understanding
of innovative approaches and developments in the industry.

6
1.4.2 An Extensive Review of Deep Learning Techniques for Pill Recognition

MERIT: Convolutional neural networks (CNNs) are emphasized in this review,

which focuses only on deep learning methods for pill detection. Through an
exploration of CNN-based techniques, it offers readers a thorough grasp of
developments in deep learning methods for pill identification.

DEMERIT: However, it's possible that the evaluation would ignore conventional
computer vision techniques, which might limit our understanding of a wide range of
approaches and possibly overlook their benefits in pill recognition tasks.

1.4.3 Pill Identification System Utilizing SVM and Haar Cascades in Real-Time:

MERIT: Using support vector machines and Haar cascades, this work offers a real-
time method for pill identification (SVM). Through the assessment of the system's
effectiveness across various datasets, it provides significant understanding into the
realistic use of pill identification systems in real-world scenarios.

DEMERIT: On the other hand, it's possible that the study did not fully address the
system's scalability for detecting a wide variety of tablets, which might provide
difficulties for implementing the system across a range of pill features.

1.4.4 Recent Progress in Technologies for Pill Identification:

MERIT: The current developments in computer vision, machine learning, and deep
learning techniques for pill detection are the main topics of this review study. In order
to enhance the precision and effectiveness of pill identification, it addresses the fusion
of sophisticated image processing methods with neural networks.

DEMERIT: On the other hand, it's possible that the evaluation missed important
details regarding certain approaches and how they were applied in real-world settings,
which might have limited our understanding of how the developments under
discussion could be applied in practice.

1.4.5 Comparative Analysis of Pill Identification Machine Learning Algorithms:

MERIT: For pill recognition tasks, this study evaluates the effectiveness of many
machine learning methods, such as decision trees, random forests, support vector
machines, and k-nearest neighbours. It assesses the algorithms according to criteria
including scalability, accuracy, and computational economy.

7
DEMERIT: However, the study could exclude deep learning methods, which have
demonstrated notable progress in pill recognition, thereby restricting understanding of
state-of-the-art approaches.

1.4.6 Considering the User Experience in Applications for Pill Identification:

MERIT: The purpose of this study is to examine how user experience (UX) variables
are considered while designing pill identification applications. Specifically, we
highlight the importance of features such device compatibility, ease of use, and
accessibility for those with visual impairments. To create pill identification interfaces
that are easy for users to navigate, it offers insights into design concepts and UX best
practices.

DEMERIT: However, the study could not have sufficient empirical evidence or user
test results to confirm the efficacy of the suggested UX standards, which could restrict
its usefulness for programmers creating pill recognition software.

8
CHAPTER 2
SYSTEM
ANALYSIS

9
2.1 EXISTING SYSTEM

OpenCV is used for pill identification in commercial products like PillCam and RX
Scanner. PillCam uses computer vision techniques to classify and extract features from
pill photos. RX Scanner uses OpenCV for image processing and enables users to take
pill pictures using the camera on their smartphone. Automatic recognition and user-
friendly interfaces are advantages. The private nature of the product, implementation
expenses, and possible difficulties in precisely recognizing tablets with distinctive
properties are some of its drawbacks.

2.1.1 DISADVANTAGES

The efficiency of current pill identification methods is hampered by several issues. Their
accuracy is one significant drawback, especially when trying to detect tablets with
complex patterns or unusual forms. Due to the visual complexity of tablets, these
systems may have trouble identifying them correctly, which might put patients at
danger. Furthermore, imbalances in the training data may give rise to biases that lead
the system to prefer some pill kinds over others. This bias has the potential to jeopardize
patient safety and the validity of the identification results. Furthermore, the accessibility
and scalability of many current technologies in healthcare settings are restricted by their
proprietary nature. Exorbitant implementation expenses compound these difficulties,
making it difficult for healthcare organizations to develop and use these systems
efficiently. Additionally, systems that find it difficult to manage high amounts of
identification requests effectively may experience scaling problems, which can cause
delays and decreased efficacy. To increase the precision, dependability, and usability of
pill identification systems in healthcare, these constraints must be addressed.

2.2 PROPOSED SYSTEM

The suggested pill identification system combines state-of-the-art machine learning

algorithms with computer vision techniques to produce a reliable and effective method
for recognizing pills based on their visual characteristics. The system uses pre-trained
CNN models, including ResNet, MobileNet, and EfficientNetB0, for feature extraction
and OpenCV for image processing to properly categorize pills based on their form,
colour, and markings. The system can identify pills with great accuracy thanks to a
multimodal neural network design that fuses the retrieved data and mixes the outputs of
many CNN models. Additionally, the creation of an intuitive web application with Flask
and HTML/CSS enables smooth system interaction, enabling users and healthcare
providers to quickly input pill photos and get real-time identification results. This
integrated method increases accessibility for users in a variety of healthcare settings
while also improving pill recognition efficiency and accuracy. The suggested approach

10
offers a substantial improvement in pill recognition technology, with the potential to
improve patient safety and ease pharmaceutical quality management.

2.2.1 ADVANTAGES

The difficulties in correctly recognizing tablets based just on their visual characteristics
are addressed in full by the suggested pill identification method. The system attempts to
offer a dependable and effective way for pill detection by combining machine learning
algorithms with sophisticated computer vision capabilities. The employment of pre-
trained CNN models, namely MobileNet, ResNet, and EfficientNetB0, for feature
extraction contributes to the high accuracy of the suggested system, which is one of its
main benefits. These versions provide more accurate identification results by allowing
the system to collect minute characteristics of the colour, shape, and marks of the pill.
The outputs of these models are combined in a multimodal neural network architecture,
which further improves the system's classification skills. Additionally, the creation of a
readily navigable online application interface guarantees accessibility for both
consumers and healthcare providers, enabling them to quickly input pill photos and
receive identification results in real time. The suggested system offers a significant
improvement in pharmaceutical technology since it has the potential to improve patient
safety, expedite pill identification operations, and increase workflow efficiency.

2.3 PRELIMINARY INVESTIGATION

To define the scope and viability of the pill identification system project, important
factors were carefully reviewed during the initial phase of inquiry. First, by highlighting
the drawbacks of human identification techniques, the problem space identification
revealed the need for a precise and effective pill identification solution. After that, a
thorough analysis of the literature was done to clarify current approaches, with a focus
on computer vision and machine learning methods. This review provided the
fundamental knowledge needed to design the suggested system. It was crucial to do a
feasibility study, which involved evaluating the resources, knowledge, and technologies
that were accessible. After that, a thorough requirement analysis was carried out to
outline the conditions that needed to be met for the system to be developed. These
conditions included gathering data, selecting a model, preprocessing methods, and
designing the web application interface. Furthermore, a risk assessment was carried out
to pinpoint any obstacles and create mitigation plans. This initial round of study
provided strong foundations, insightful information, and a roadmap for the project's
effective execution.

11
2.4 FEASIBILITY STUDY

2.4.1 ECONOMIC FEASIBILITY

The preliminary inquiry phase's economic feasibility analysis yielded important

information about the planned pill identification system project's financial viability.
The project's estimated expenses and possible returns on investment were carefully
examined as part of this study.

The first cost estimates were produced by considering several variables, such as the
hardware and software needs, the cost of hiring staff, and any extra charges related to
setting up the infrastructure and obtaining data. To determine the project's potential for
producing income, other revenue streams were investigated, such as system license
fees or web application subscription fees.

In addition, a cost-benefit analysis was carried out to compare the system's estimated
costs and expected benefits. This required putting a number on both concrete and
intangible gains, such increased patient safety, better pharmaceutical quality control,
and time savings for medical staff.

Overall, the study of the economic viability of the suggested pill identification system
showed potential for producing profitable returns on investment. Healthcare
institutions, pharmacies, and customers stand to gain a great deal from the system as it
streamlines pill identification procedures, lowers mistakes, and boosts efficiency.
However, to guarantee that the project stays financially feasible during its
implementation and beyond, constant monitoring and recurring revaluations of costs
and benefits will be necessary.

2.4.2 TECHNICAL FEASIBILITY

The pill identification system project's initial technical feasibility study gave
important information about the project's technological viability and the possibility of
putting the suggested solution into practice. This assessment included a look at several
technical factors, such as hardware, software, and skill levels.

The availability and appropriateness of the required software tools and frameworks
was one of the main factors considered in the technical feasibility investigation.
OpenCV is a potent computer vision toolkit that has become an essential part of
feature extraction and picture preparation. In addition, the convolutional neural
network (CNN) models had to be built and trained, which required choosing suitable
deep learning frameworks like TensorFlow or PyTorch.

12
In order to guarantee compatibility and sufficient computing resources for model
training and inference, the hardware requirements were also carefully examined. This
included determining if cloud computing resources or powerful GPUs were available
to speed up training and enable real-time recognition in the online application.

In addition, a technical expertise evaluation was conducted for the development and
deployment of the system. This includes knowledge of machine learning ideas,
mastery of web development frameworks like Flask for UI design, and fluency with
programming languages like Python for application development.

Overall, the technical feasibility analysis showed that hardware, software, and other
resources needed to support the creation and use of the pill identification system were
easily accessible. The project was declared technically feasible with the right
resources and skills in place, opening the door for future development and execution.

13
 CHAPTER 3
SYSTEM
IMPLEMENTATION

14
3.1 HARDWARE REQUIREMENTS

PROCESSOR
Intel i5 10th Gen

RAM
8 GB

GRAPHICS CARD
NVIDIA GeForce GTX 1650Ti

STORAGE 1TB SSD

3.2 SOFTWARE REQUIREMENTS

OPERATING SYSTEM Windows 11

PROGRAMMING LANGUAGE Python, HTML, CSS

SOFTWARE TOOL PyCharm

PACKAGES USED NumPy. Matplotlib, TensorFlow,

OpenCV, CNN.

3.3 ARCHITECTURE DESIGN

To enable effective and precise pill recognition, the architecture design of the pill
identification system uses a multi-tiered approach that incorporates both software and
hardware components. The overall system architecture is composed of several linked
layers, each of which has a distinct function in the design.

Layer of Data Collection and Preprocessing: This first layer collects a variety of pill
pictures, identifying differences in colour, shape, and markings. The photos are pre-
processed using OpenCV techniques, such scaling, normalization, and noise reduction,
to improve the quality of features retrieved in later stages and standardize the input
data.

15
Classification Layer and Feature Extraction: Feature extraction from the pill photos
is done in this layer using pre-trained convolutional neural network (CNN) models such
as ResNet, MobileNet, and EfficientNetB0.The form, colour, and texture of pills are
among the high-level attributes that these models are effective at capturing. When the
pill pictures are classified into multiple classes that correspond to various types of pills,
the extracted characteristics are used as inputs in the classification layer.

Architecture Layer of the Multimodal Model: In this layer, the outputs from the
separate CNN models (MobileNet, ResNet, EfficientNetB0) are integrated using a
multimodal architecture. Combining the complementing qualities of each CNN model,
a multilayer neural network is created to combine the retrieved information and carry
out classification. The system's higher accuracy and resilience in pill recognition are
made possible by the concatenated characteristics from the separate models.

Layer of Web Application Development: The creation of a web application with

HTML/CSS for the frontend and Flask as the backend framework constitutes the last
layer. Users may input pill photos and get real-time identification results thanks to the
web application's seamless integration of the trained multimodal model. The application's
usability and accessibility are improved by user-friendly features including the
interactive display of identification results and drag-and-drop picture upload. Overall, the
pill identification system's architecture is set up to make feature extraction,
categorization, and data processing more effective while offering a user-friendly
interface that makes system interaction easy. The solution provides an all-inclusive
approach to precise and user-friendly pill detection by merging web application
development with sophisticated computer vision algorithms.

16
 CHAPTER 4
SYSTEM DESIGN

17
4.1 SYSTEM DESIGN

The system design for "Pill Identification Using OpenCV" encompasses various
stages, ensuring accurate and efficient pill identification. The key components of the
system design are detailed below:

4.1.1 INPUT DESIGN

The input design is critical to enabling smooth user-application interaction in the
context of the "Pill Identification Using OpenCV" system. The salient features of
the input design optimized for effective pill detection are as follows:

Input Type:
1.Image Input:
- For the purpose of pill identification, the system mainly accepts picture input. For
analysis, users can submit a single photograph with one or more tablets in it.

2.Data Source:
-Users can choose to upload pre-existing photographs from their devices for analysis, or
they can take pictures using a camera that is linked to the system.

3.Input Resolution and Format:

-The system defines the required resolution and format for input pictures to guarantee
consistency in processing. This keeps consistency across many input sources and
optimizes the analysis process.

4.Data Preprocessing:
-To improve the quality of the input photos, the system applies strong data
preprocessing techniques before to analysis. This might entail adding noise-reduction
filters, cropping photos to a standard size, or modifying lighting to enhance visibility.

5.Error Handling:
-To handle any problems during picture capture or processing, the system includes error
handling techniques. This involves resolving situations like improper picture formats or
broken camera connections to guarantee seamless functioning and user experience.

6.Accessibility Considerations:
To ensure inclusivity for users with a range of requirements, accessibility is a crucial
factor in input design. To support users with varying abilities, the system offers other
input ways, enhancing accessibility and usability for everyone using the program.

18
4.1.2 OUTPUT DESIGN

1.Detected Pills Display:

To help users identify pills accurately, identified pills are shown with comprehensive
information such as their names, dosages, and manufacturers.
2.Real-time Visualization: In real-time applications, users receive immediate feedback
when identifying pills by superimposing identified pills on the live camera stream.

3.Error Handling: When there are identification errors or system malfunctions, users are
greeted with succinct and understandable error messages that help them fix the problem
and guarantee a positive user experience.

By addressing these considerations in the input and output design, the "Pill
Identification Using OpenCV" system can offer a user-friendly experience with robust
input methods and clear output presentations.

4.2 FEATURES OF OPENCV

OpenCV provides several capabilities that are necessary for image processing and
computer vision jobs, as demonstrated in the "Pill Identification System Using OpenCV
and Web Application" . The following are the salient characteristics of OpenCV that
are emphasized in relation to the project:

1.Picture Preparation: Pill pictures may be pre-processed uniformly thanks to OpenCV's

noise reduction, normalization, and scaling features. These methods improve the
consistency and quality of the images for further processing.

2.Extracting Features: OpenCV makes it easier to extract features from pill photos by
using pre-trained CNN models like as ResNet, MobileNet, and EfficientNetB0. High-
level qualities of pills, including form, colour, and texture, are captured by these
models.

3.Finding Things: Pill objects may be found and isolated in photos with the use of
OpenCV's object detection methods, such as contour detection and thresholding. This
skill is essential for classifying medications according to their visual characteristics.

4.Architecture for Multimodal Models: OpenCV integrates CNN model outputs to

19
facilitate the development of a multimodal architecture. This improves the system's
classification capabilities by allowing the merging of information retrieved by models
such as ResNet, EfficientNetB0, and MobileNet.

5.Integration of Web Applications: OpenCV is mostly a computer vision library,

however it works well with web development frameworks such as Flask. This makes it
possible to develop online apps that provide easy-to-use access to the pill identification
system. Through the online interface, users may upload photographs and get real-time
identification results.

6.Real-Time Image Processing: The system can examine pill photos in real-time via the
web application interface thanks to OpenCV's real-time image processing capabilities.
Enhancing user experience and efficiency, this function offers instant feedback on pill
identification.

All these elements work together to strengthen the "Pill Identification System Using
OpenCV and Web Application," making it possible to identify pills visually and
accurately and quickly. For the development of computer vision systems such as the
one presented in the case study, OpenCV is an essential tool due to its extensive feature
set and adaptability.

4.3 PYCHARM

Python developers may work more efficiently with PyCharm, an integrated

programming environment. Among PyCharm's primary features are:

- Smart tools for troubleshooting and altering code.

- Usage of testing framework integration.

- The administration of virtual environments.

-Support for version control.

- Assistance with web development that is specific.

PyCharm facilitates the creation of the "Pill Identification" system and increases
productivity.

20
4.4 TECHNOLOGIES USED

Several technologies were used in the "Pill Identification System Using OpenCV and
Web Application" to create a reliable and effective pill identification system. The
following technologies are among them:

1.OpenCV (Open-Source Computer Vision Library): OpenCV is the system's primary

technology for tasks including object identification, feature extraction, and image
processing. It offers an extensive set of features and methods for picture preprocessing,
feature extraction, and object detection—all crucial tasks for pill identification.

2.Python Programming Language: The main programming language utilized to create

the system is Python. Python is an excellent choice for a variety of tasks, including web
development, data preparation, model training, and integration with OpenCV, due to its
vast library and user-friendliness.

3.The Flask Web Framework: The pill identification system's web application interface
is created using Flask. Flask is a web framework that is lightweight and versatile,
enabling the development of online applications that are easy to use with little
overhead. It makes it easier for the web interface to integrate the trained model,
allowing users to input photographs and get real-time identification results.

4.HTML and CSS: The web application's frontend is designed and styled using CSS
(Cascading Style Sheets) and HTML (Hypertext Markup Language). Web pages are
structured and laid out using HTML, and their look and styling may be customized with
CSS to improve user experience.

5.Machine Learning Libraries(TensorFlow, Keras): The multimodal model utilized in

the pill recognition system is trained and evaluated using machine learning frameworks
like TensorFlow and Keras. In order to make transfer learning and model assessment
chores easier, these libraries provide tools and APIs for creating, honing, and
implementing deep learning models.

6.CNN Models using pre-training (ResNet, MobileNet, and EfficientNetB0): For

feature extraction from pill photos, pre-trained convolutional neural network (CNN)
models such as ResNet, MobileNet, and EfficientNetB0 are used. These models, which
are included in frameworks like TensorFlow and Keras, capture high-level attributes
like form, colour, and texture that are pertinent to pill properties.

Together, these technologies enable the pill identification system to meet the demands
of customers, pharmacists, and healthcare providers with high levels of accuracy,
efficiency, and user-friendliness.

21
4.5 DATA FLOW DIAGRAM

1.User: Describes how a user interacts with the system.

2.Web Application: This is a representation of the website where users may contribute
pictures of pills.

3.Preprocessing: This step is when the submitted photos are processed.

4.Feature Extraction: This stands for the process of removing features from previously
processed pictures.

5.Classification: Shows how the pill is categorized using the characteristics that were
retrieved.

6.Result: Shows the user the result with the pill identified.

FIG 4.1 Data flow Diagram

22
 4.6 UNIFIED MODELLING LANGUAGE

4.6.1 ARCHITECTURE DIAGRAM

FIG.4.2 Architecture diagram

1.Frontend Webpage: Here, users will likely upload photographs for processing and
engage with the system.

2.Grab Image: The process of taking a picture from the user.

3.Trained Model File: An picture that has been uploaded and has been pre-trained by a
machine learning model.

4.Dictionary File: An archive that could include information or sources that the model's
training uses to analyse images.

5.Display Result: On the front-end webpage, the user is presented with the processed
results.

4.6.2 SEQUENCE DIAGRAM

Sequence diagrams show a detailed flow for a specific use case or even just part of
a specific use case. The vertical dimension shows the sequence of messages/calls in
the time order that they occur; the horizontal dimension shows the object instances
to which the messages are sent. The consisting objects are Localizability testing,
structure analysis, network adjustment and localizability aided localization.

23
 The flow of interactions between various parts of the pill identification system is
depicted in the sequence diagram. Each step is explained as follows:

1.User: Uploads a picture of the pill to the website application to start the procedure.

2.WebApp: Takes the user-uploaded picture and sends the pill identification request
to the service layer. To identify the pill, the service layer processes the request and
interacts with the main logic module.

3.CoreLogic: Handles the real identification procedure, which can entail running a
database query to retrieve pill reference data.

4.Database: Acquires the query from the central logic module and provides
pertinent details about the medication in response.

5.CoreLogic: Gathers data from the database about the pill and sends it back to the
service layer.

6. ServiceLayer: Returns the information about the detected pill to the web
application after receiving it from the core logic module.

7.WebApp: Shows the user the identification results based on the data that was
received.

All things considered, this flow shows how the user interacts with the web application
to identify a pill, and how the many underlying parts cooperate to process the request
and deliver precise outcomes.

24
 Fig.4.3 Sequence Diagram

4.6.3 CLASS DIAGRAM

The elements of the web application and OpenCV-based pill identification system are
depicted in this class diagram. This is the justification:

1. Image Input: This module is the one in charge of processing picture input. It provides
functions for uploading images upload image and taking pictures with a camera capture
image.

2. Data Preprocessor: Prior to categorization, this module preprocesses the input

pictures. To be ready for categorization, it has functions like preprocess, resize, and
normalize.

3. Classifier: Manages how to categorize pictures of pills. It contains functions for

predicting the future predict, categorizing photos classify, and assessing the model
evaluate.

4.WebApp: Describes the front end and back end of a web application. The user may
input photographs and obtain identification results through interaction with the system.
The web application is started by the run function, and upload image manages
uploading images. The pill detector oversees finding tablets in the photos that have
been submitted. The detect pills function in this module uses visual cues to identify
pills.

5.PillDatabase: Oversees the database that contains data on pills. It has functions for
retrieving pill information get pill and saving pill information save pill.

25
6.Assignations:(1-to-1) are used to illustrate the connections between the classes,
showing that each class is connected to precisely one instance of the related class. For
instance, a single Data Preprocessor instance is linked to every Image Input instance,
and so on.

FIG 4.4 Class Diagram

4.6.4 ACTIVITY DIAGRAM

First, the user uploads a picture in the activity diagram. If the picture upload is
successful, the service layer receives the request. To identify the pill, the service layer
calls upon the core logic. The website obtains information about the pill and shows the
identification outcomes if the pill is correctly recognized. At each step, relevant error
messages are shown if there are any issues. After showing the outcomes or error
warnings, the procedure ends.

26
 FIG 4.5 Activity Diagram
4.6.5 COLLABORATION DIAGRAM
The collaboration diagram illustrates the interactions between the many parts of the
system that take place throughout the pill identification process.
An photograph is uploaded to the online application by the user actor.
To the Service Layer, the Web Application submits a request.

For pill identification, the Core Logic is called by the Service Layer.
After processing the picture, the Core Logic sends the identification results to the
Service Layer.
The results are returned to the Web Application by the Service Layer.
Ultimately, the User is presented with the identification results by the Web Application.

FIG 4.6 COLLABORATION DAIGRAM

27
4.6.6 ENTITY RELATIONSHIP DIAGRAM
1.User: Stands in for users of the website who submit pictures to be identified by pills.

2.Image: Contains the filename, URL, upload timestamp, and other metadata associated
with the submitted photos.

3.Pill: Identifies the pills and includes information about them, including name, shape,
colour, and markings.

4.IdentificationResult: This object displays the identification text, confidence score, and
timestamp for the pill.

Relationship:
1. Upload: Users supply the system with picture uploads.

2.Analyze: To detect pills, uploaded photos are examined.

3Identification: Results of identification are linked to identified medications.

The Pill Identification System may describe entities and their connections using this
ERD, which offers a basic framework for doing so. The ERD can be further enhanced
and developed in accordance with the needs and features of the system.

28
 FIG 4.7 ER DIAGRAM

4.6.7 USE CASE DIAGRAM

1.User (U): Identifies common users who post pill photos for identification.

2.Health Professional (HP): This class represents health professionals who use the
system to find medications that are appropriate for a patient's needs.

3.System (S): Stands for the actual Pill Identification System.

29
Use Cases:
1. submit Image (UC1): To identify pills, users can submit photographs of them.

2.Identify Pill (UC2): Users or medical professionals may ask to have a pill identified.

3.View Identification Result (UC3): The system's identification result is available for
users and medical professionals to view.

The interactions between the actors and the system are shown in this use case diagram,
which also highlights the key features of the pill identification system. Users may
request pill identification, provide photos, and see the results of their identification.
These features can also be used by medical personnel to care for patients.

FIG 4.8 USE CASE DIAGRAM

4.7 MODULES DESCRIPTION

4.7.1 DATA COLLECTION AND PREPROCESSING:

Description:

Gathering a wide dataset of pill pictures and getting them ready for feature extraction
and classification are the tasks assigned to this module. It guarantees that the photos are
correctly processed to improve the identification system's accuracy.

Module Overview:

1.Data Collection: The goal of this submodule is to compile an extensive dataset of pill
pictures. To guarantee variation in terms of pill kinds, sizes, colours, and markings, it
entails gathering photos from several sources.

2.Image Preprocessing: To standardize and maximize their quality for analysis, the

30
gathered pictures go through preprocessing. OpenCV libraries are used to apply
methods like noise reduction, scaling, and normalizing in order to make feature
extraction algorithms more consistent and efficient.

3.Dataset Annotation: To manually annotate each image in the dataset with pertinent
details such pill type, shape, and colour, it may occasionally be necessary. This
annotated dataset is used as training and assessment ground truth.

4.Quality Assurance: To make sure the gathered dataset satisfies the necessary
requirements; this submodule performs quality checks. It might entail checking the
quality of the images, eliminating duplicates, and fixing any discrepancies in the
dataset.

5.Data Augmentation: Rotation, flipping, and brightness modification are examples of

data augmentation techniques that may be used to create more training samples in order
to increase the variety of the dataset and boost model generalization.

Functionality:

1.Assembles a varied pill picture collection.

2. Uses OpenCV methods to preprocess pictures.

3. Adds pertinent annotations to the dataset.

4. Makes use of quality assurance techniques to guarantee dataset quality.

5. Adds more data to the model to increase its resilience.

Benefits:

1.Guarantees that a varied and representative dataset is used to train the identification
system.

2.Preprocesses photos to eliminate noise and standardize their format, which improves
the system's accuracy and resilience.

31
3.Enables efficient feature extraction and categorization by supplying input data of the
highest caliber.
uses data augmentation approaches to enhance model performance and generalization.

4.7.2 FEATURE EXTRACTION AND CLASSIFICATION:

Description:

This module oversees taking pre-processed pill photos and extracting high-level
characteristics from them. Then, it uses pre-trained Convolutional Neural Network
(CNN) models to classify the images. It attempts to determine distinctive qualities of
tablets by analysing visual features including texture, colour, and form.

Module Overview:

1.Feature Extraction: The goal of this submodule is to extract pertinent characteristics

from pill pictures that have already been analysed. It makes use of CNN models that
have already been trained, such MobileNet, ResNet, and EfficientNetB0, to identify
unique patterns and characteristics that set one pill apart from another.

2.Feature Representation: Following their extraction, the features are formatted

appropriately for additional processing and categorization. To do this, the extracted
characteristics may need to be encoded into numerical vectors or other representations
that work with the classification methods.

3.Classifier Selection: Using the retrieved characteristics as a guide, the module

chooses a suitable classifier to categorize the pill pictures. Support Vector Machines
(SVM), Random Forests, and Neural Networks are popular classifiers that use the
extracted features as training data to generate predictions.

4.Model Evaluation: The module uses measures like accuracy, precision, recall, and F1-
score to assess the classifier's performance once it has been trained. This aids in
evaluating the categorization model's efficacy and pinpointing areas in need of
development.

5.Model Optimization: The module may include fine-tuning the pre-trained CNN
models or refining the classifier's hyperparameters to improve classification efficiency
and accuracy. Achieving optimal performance on the provided dataset is the goal of this
iterative approach.

32
Functionality:

1.Uses pre-trained CNN models to extract high-level information from pre-processed

pill pictures.

2.Gives the extracted characteristics a format that is appropriate for categorization.

3.Uses the retrieved characteristics to determine which classifier is most suited for
classifying pill pictures.

4.Uses a variety of indicators to assess the categorization model's performance.

5.Enhances the categorization model's efficiency and accuracy.

Benefits:

1.Permits the system to recognize distinct visual characteristics of tablets for precise
categorization.

2.Makes use of cutting-edge CNN models to efficiently extract features.

3.Makes precise predictions using strong classifiers based on the characteristics that are
extracted.

4.Makes it easier for the classification model to be improved continuously through

optimization and assessment.

4.7.3 MULTIMODAL ARCHITECTURE MODULE:

The Multimodal Model Architecture Module creates a neural network for feature fusion
and classification and combines several feature extraction models into a holistic
architecture. This module attempts to improve the pill identification system's resilience
and accuracy by merging several information modalities.

33
Module Overview:

1.Integration of Feature Extraction Models: The integration of distinct feature

extraction models, including ResNet, EfficientNetB0, and MobileNet, is the main
objective of this submodule. From the pill photos, each model extracts visual attributes,
such as shape, colour, and texture.

2.Feature Fusion: To produce a complete representation of the pill pictures, the features
acquired from several models are fused together. Concatenation, summation, and
weighted averaging are examples of feature fusion techniques that are used to combine
the extracted features effectively.
3.Neural Network Architecture Design: The module creates a neural network
architecture that performs classification using the fused characteristics as input.
Convolutional, pooling, and fully connected layers are among the layers that generally
make up this neural network in order to learn hierarchical representations of the input
characteristics and provide predictions.

4.Training and Fine-Tuning: This module starts with the design of the neural network
architecture and ends with the model being trained using transfer learning on the
labelled dataset. Transfer learning makes use of the information gleaned from
previously trained models to modify the neural network for tasks involving the
detection of pills. The performance of the model can also be further optimized by using
fine-tuning approaches.

5.Validation and Assessment: A variety of measures, including accuracy, precision,

recall, and F1-score, are used to validate and assess the trained multimodal model. This
stage evaluates the model's suitability for practical use by determining how well it can
identify pill pictures using their visual characteristics.

Functionality:

1.Combines different feature extraction methods to extract various visual characteristics

from pill pictures.

2.Fuses generated a thorough representation by employing suitable fusion algorithms to

extract features.

3.Constructs a neural network structure for classifying and fusing features.

34
4.Trains the multimodal model with transfer learning and fine-tuning methods on
labelled datasets.

5.makes sure the model is reliable and effective by validating and assessing its
performance using pertinent metrics.

Benefits:

1.Uses a variety of information modalities to improve the pill recognition system's

resilience and accuracy.

2. Allows the pill's many visual properties to be captured by the algorithm for a more
precise categorization.

3.Makes efficient use of sophisticated neural network topologies for feature fusion and
categorization.

4.Allows for the model's performance to be continuously improved through validation,

assessment, and training.

4.7.4 INSTRUCTION AND ASSESSMENT MODULE:

Description:

The Training and Evaluation Module oversees applying transfer learning techniques to
train the multimodal model on the labelled dataset and assessing its effectiveness
through a variety of criteria. This module is essential for fine-tuning the parameters of
the model and evaluating how well it can identify pill photos according to their visual
characteristics.

Module Overview:

1. Preparation: The preparation of the labelled dataset is the first step in the module's
operation. It makes sure that the collection of pill pictures is varied and that the labels
adhere to ground truth. To make training and evaluating the model easier, the dataset is
separated into training, validation, and test sets.

2.Transfer Learning: For tasks involving the detection of pills, transfer learning

35
approaches are used to capitalize on the information obtained by pre-trained CNN
models (such as MobileNet, ResNet, and EfficientNetB0. To adapt the pre-trained
models to the pill recognition domain while maintaining the learnt characteristics, their
weights are adjusted on the particular dataset.

3.Model Training: The input pictures are sent through the neural network architecture
created in the Multimodal Model Architecture Module in order to train the multimodal
model using the pre-processed dataset. The model gains the ability to identify pertinent
aspects from the pill photos during training and forecast outcomes using these
attributes.

4. Hyperparameter tuning: To maximize the performance of the model,

hyperparameters including learning rate, batch size, and optimization techniques are
adjusted. Methods such as grid search or random search can be utilized to identify the
ideal set of hyperparameters that optimizes the accuracy and generalizability of the
model.

5. Validation and Evaluation: Prior to a final assessment on the test set, the trained
model's performance is assessed on the validation set. The efficacy of the model in
accurately classifying pill pictures is assessed by the computation of many metrics,
including accuracy, precision, recall, and F1-score. Furthermore, methods such as
cross-validation can be employed to guarantee the stability and dependability of the
assessment outcomes.

6.Model Optimization: To enhance the performance of the model, further optimization

approaches may be used in light of the evaluation outcomes.

Functionality:

1.Gets the labelled dataset ready for model assessment and training.

2. Uses transfer learning methods to optimize CNN models that have already been
trained for tasks involving pill identification.

3.Uses the provided dataset and optimal hyperparameters to train the multimodal
model.

4.Evaluates the model's performance in terms of accuracy and generalizability using a

variety of measures.

5.Enhances the model's performance and dependability by optimizing it in light of

36
evaluation findings.

Benefits:

1.Makes use of transfer learning to effectively train models by utilizing the information
from previously trained models.

2.Makes ensuring that the model's performance is accurately and consistently evaluated
using strict validation procedures.

3.Helps to continuously optimize and tune the hyperparameters of the model, enhancing
its resilience and accuracy.

4.Gives information about the model's advantages and disadvantages, directing future
improvements to get better results in pill detection tasks.

4.7.5 WEB APPLICATION DEVELOPMENT MODULE:

The Web Application Development Module oversees designing an interface that is easy
for users to use so they can communicate with the pill identification system. It creates a
web application that allows users to upload pictures of pills and get instant
identification results. By bridging the gap between the end users and the underlying
machine learning system, this module offers a smooth way for users to access and make
use of the pill recognition features.

Module Overview:

1.Backend construction: Using frameworks such as Flask, the module starts with the
construction of the web application's backend components. In order to identify pills, the
backend must interface with the machine learning model, analyse submitted photos, and
handle user requests.

2.Frontend Development: To produce a user experience that is both intuitive and

aesthetically pleasing, the web application's frontend components are designed using
HTML, CSS, and JavaScript. Users may examine identification results, add pill photos,
and communicate with the system through the interface.

3.User Authentication and Authorization: The web application may provide capabilities
for user authentication and authorization in order to protect user privacy and security.

37
This entails putting in place login procedures and access control to limit certain features
according to user roles or permissions.

4.Processing and Uploading of photos: The online application lets users drag and drop
or upload files to upload photos of pills. The submitted photographs are processed by
the backend upon receipt to get them ready for recognition by the machine learning
model.

5.Results of Real-Time Identification: The web application communicates with the

machine learning model to identify pills in real-time after processing the supplied
photos. The user is then presented with the recognized pills and pertinent details
including the name, dose, and usage guidelines.

6.User Interaction and Feedback: To improve the user experience, the online
application includes interactive elements including feedback systems for reporting
identification mistakes or giving more details about pills that have been detected. This
enables users to have an active role in raising the accuracy and dependability of the
system.

7.Cross-browser compatibility and responsive design: To guarantee optimum

performance on a range of screens and devices, the web application is designed with a
responsive design. To reach a larger user base, compatibility with various web browsers
is also guaranteed.

Functionality:

1.Offers a simple user interface for submitting pictures of pills and getting
identification results in real time.

2. Carries out backend logic to handle user authentication, process user requests, and
communicate with the machine learning model.

3.Creates front-end elements with interactive features and a user-friendly design to

facilitate smooth user interaction.

4.Uses data encryption, user authentication, and authorization to provide security and
privacy.

5.Allows for the uploading, processing, and aesthetically pleasing presentation of

identification results for images.

38
Benefits:

1.Makes the pill identification method easily accessible to anyone without

technological knowledge.

2.Improves user experience through responsive interface, real-time feedback, and

intuitive design.

3.Permits the easy incorporation of machine learning features for more accessibility
into web-based applications.

4.Enables extensibility and scalability for further improvements and new features.

4.8 ALGORITHMS USED

Overview:

The "Algorithms Used" module highlights common algorithms employed in pill

identification using OpenCV, offering insights into their functionalities.

Algorithms:

1.Preprocessing and Data Collection:

Procedure: OpenCV Methods (such as scaling, adjusting, and eliminating noise)

Use: Standardizing the input data for subsequent analysis by preprocessing pill
pictures.

2.Semantic Analysis and Feature Encoding:

Procedure: CNN models with prior training (such as MobileNet, ResNet, and
EfficientNetB0) are the algorithm.

Use: Taking high-level information from the photos about the colour, texture, and
shape of the pills.

3.Multimodal Architecture Model:

Procedure: Neural Network Fusion Methods is the algorithm.

39
 Use: Merging the results of many pre-trained models (such as ResNet, MobileNet,
and EfficientNetB0) to establish a multimodal pill classification architecture.

4. Instruction and Assessment:

Transfer learning is the algorithm used to fine-tune pre-trained models for the
objective of pill detection using a labelled dataset.

5.Development of Web Applications:

Procedure: HTML/CSS (frontend), Flask (backend framework)

Use: Creating an online application interface that allows users to submit pictures of
pills and get instant identification results.

40
 CHAPTER 5
CONCLUSION

41
5.1 CONCLUSION

In conclusion, the project, "Pill Identification using OpenCV," offers a promising

solution for efficient and accurate pill recognition. Leveraging the MobileNetV2
architecture in conjunction with OpenCV's computer vision capabilities, the
system aims to provide a practical and reliable tool for identifying pills based on
visual attributes.

The strategic integration of MobileNetV2, designed for mobile and edge devices,
aligns with the project's goal of achieving real-time and resource-efficient pill
identification. OpenCV complements this by providing essential functionalities
for image preprocessing, feature extraction, and classification.

It's crucial to acknowledge potential challenges, such as variations in pill

appearance, diverse image qualities, and possible occlusions, which need careful
consideration for the system to perform effectively in real-world scenarios. The
success of the project hinges on the quality and diversity of the training dataset,
and ongoing optimization may be necessary for practical deployment.

In summary, the collaborative use of MobileNetV2 and OpenCV presents a potent

synergy for pill identification, with the potential to enhance medication
management, healthcare efficiency, and patient safety. As the project advances,
continuous refinement and adaptation to diverse pill types and environmental
conditions will be key to ensuring the system's accuracy and reliability in practical
applications. The project holds promise for advancing medication recognition
technology, contributing positively to healthcare and patient well-being.

42
 CHAPTER 6
FUTURE
ENHANCEMENTS

43
 6.1 FUTURE ENHANCEMENTS

Pill identification using OpenCV is a dynamic field with potential avenues for future
improvement and innovation. Consider the following areas for enhancement:
1. Multi-modal Pill Identification:
• Explore integrating multiple modalities, such as shape analysis, colour
recognition, and texture identification, to enhance pill identification
accuracy.
• Incorporate additional sensory data like RFID tags or barcode scanning to
provide complementary information for identification.
2. Real-time Identification Optimization:
• Investigate techniques to optimize real-time pill identification, aiming for
improved speed and accuracy.
• Implement model quantization and pruning to reduce the model's size,
enabling faster inference without compromising accuracy.
3. Personalized Pill Identification:
• Develop mechanisms to personalize the identification model based on
individual user preferences or medical history.
• Allow users to input additional information, such as pill variations or
specific markings, to tailor the identification process.
4. Generalized Pill Identification:
• Expand the model's capability to recognize a wider variety of pills by
incorporating a more extensive pill database.
• Investigate the use of transfer learning techniques to adapt the model to new
pill types without extensive retraining.
5. Improved Data Augmentation Techniques:
• Enhance data augmentation methods to create a diverse and representative
dataset for training.
• Experiment with advanced augmentation techniques, such as generative
adversarial networks (GANs), to synthesize realistic pill variations.
6. User-Friendly Interface:
• Design an intuitive and user-friendly interface for the pill identification
system, allowing users to easily capture and submit pill images.
• Implement feedback mechanisms to guide users in capturing high-quality
images for improved identification.
7. Accessibility Considerations:
• Ensure the pill identification system is accessible to users with varying
needs and abilities.
• Implement features such as voice-guided instructions or compatibility with
assistive technologies to enhance accessibility.

44
 APPENDIX I
IMPLEMENTATION:

A1.1 Main.py (Main file)

import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras import layers, models, optimizers
from sklearn.metrics import classification_report,
confusion_matrix import matplotlib.pyplot as plt import seaborn as
sns
gpus =
tf.config.experimental.list_physical_devices('GPU') if
gpus: try: for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
except RuntimeError as e:
print(e)

# Set your dataset path

dataset_path = "C:\\Users\\mukes\\OneDrive\\Pictures\\Pills dataset"

# Image pre-processing def create_data_generators(data_path,

target_size=(224, 224), batch_size=32):
train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
validation_split=0.2
)
train_generator = train_datagen.flow_from_directory(
data_path,
target_size=target_size,
batch_size=32,
class_mode='categorical',
subset='training'
) validation_generator =
train_datagen.flow_from_directory( data_path,
target_size=target_size,
batch_size=32,
class_mode='categorical',
subset='validation'
)
return train_generator,
validation_generator
train_generator, validation_generator =
create_data_generators(dataset_path) num_classes =
len(train_generator.class_indices)
# Build model architecture with MobileNetV2
base_model = tf.keras.applications.MobileNetV2(input_shape=(224, 224,
3), include_top=False, weights='imagenet') base_model.trainable = False

45
model = models.Sequential([
base_model,
layers.GlobalAveragePooling2D(),
layers.Dense(128, activation='relu'),
layers.Dropout(0.5),
layers.Dense(num_classes, activation='softmax')
])

# Compile the model

model.compile(
optimizer=optimizers.Adam(learning_rate=0.0001),
loss='categorical_crossentropy',
metrics=['accuracy']
)

# Train the model with early

stopping history = model.fit(
train_generator, epochs=50,
validation_data=validation_generator,
callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5)]
)

# Plot accuracy and loss plt.figure(figsize=(12,

4))

# Plot training & validation accuracy values

plt.subplot(1, 2, 1)
plt.plot(history.history['accuracy'], label='Training Accuracy')
plt.plot(history.history['val_accuracy'], label='Validation
Accuracy') plt.title('Model accuracy') plt.xlabel('Epoch')
plt.ylabel('Accuracy') plt.legend()

# Plot training & validation loss values

plt.subplot(1, 2, 2)
plt.plot(history.history['loss'], label='Training Loss')
plt.plot(history.history['val_loss'], label='Validation
Loss') plt.title('Model loss') plt.xlabel('Epoch')
plt.ylabel('Loss') plt.legend()

plt.tight_layout()
plt.show()

# Evaluate the model

evaluation = model.evaluate(validation_generator)
print(f'\nEvaluation Loss: {evaluation[0]}')
print(f'Evaluation Accuracy: {evaluation[1]}')

# Generate predictions
y_pred =
model.predict(validation_generator)
y_pred_classes = tf.argmax(y_pred, axis=1)
true_labels = validation_generator.classes
# Classification Report
print("\nClassification Report:\n", classification_report(true_labels,
y_pred_classes))

46
# Confusion Matrix
conf_matrix = confusion_matrix(true_labels, y_pred_classes)
plt.figure(figsize=(8, 6))
sns.heatmap(conf_matrix, annot=True, fmt='g', cmap='Blues',
xticklabels=train_generator.class_indices,
yticklabels=train_generator.class_indices)
plt.title('Confusion Matrix') plt.xlabel('Predicted
labels') plt.ylabel('True labels') plt.show()

# Save the model

model.save('pill_identifier_model.h5')

tf.keras.backend.clear_session()

A1.2 pill_dictionary.py (Pill data file):

class PillDictionary:
def __init__(self):
self.pill_info = {
'Aceclofenac, Paracetamol & Serratiopeptidase': {
'benefits': 'Benefit information for Aceclofenac, Paracetamol &
Serratiopeptidase.',
'usage': 'Usage information for Aceclofenac, Paracetamol &
Serratiopeptidase.',
'tablets': ['Aceclospa', 'Paraserr', 'Zerodol-SP']
},
'Itraconazole': {
'benefits': 'Benefit information for Itraconazole.',
'usage': 'Usage information for Itraconazole.',
'tablets': ['Itrasys', 'Canditral', 'Itrazol']
},
'Montelukast Sodium & Levocetirizine Hydrochloride': {
'benefits': 'Benefit information for Montelukast Sodium &
Levocetirizine Hydrochloride.',
'usage': 'Usage information for Montelukast Sodium &
Levocetirizine Hydrochloride.',
'tablets': ['Montair-LC', 'Levocet-M', 'Montecip-LC']
},
'Paracetamol': {
'benefits': 'Benefit information for Paracetamol.',
'usage': 'Usage information for Paracetamol.',
'tablets': ['Crocin', 'Calpol', 'Dolo-650']
}
}
def get_pill_info(self, pill_name): return
self.pill_info.get(pill_name, {'benefits': 'Not available', 'usage': 'Not
available', 'tablets': []})

47
 A1.3 identify.py (Pill identifying file):

from tensorflow.keras.preprocessing import image

from tensorflow.keras.models import load_model
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
from pill_dictionary import PillDictionary
from tkinter import Tk, filedialog

# Load the trained model

model = load_model('pill_identifier_model.h5')

# Load the pill dictionary

pill_dict = PillDictionary()

# Function to preprocess the input image

def preprocess_image(img_path):
img = image.load_img(img_path, target_size=(224,
224)) img_array = image.img_to_array(img)
img_array = np.expand_dims(img_array, axis=0)
img_array /= 255.0 # Normalize the image return
img_array

# Function to make a prediction and display pill information

def predict_pill(image_path):
img_array = preprocess_image(image_path)
predictions = model.predict(img_array)
predicted_class = np.argmax(predictions)
predicted_pill = list(pill_dict.pill_info.keys())[predicted_class]
# Display the image img
= Image.open(image_path)
plt.imshow(img)
plt.axis('off')
plt.title(f'Predicted Pill: {predicted_pill}')

# Display pill information

info = pill_dict.get_pill_info(predicted_pill) print(f'\nBenefits:
{info["benefits"]}\nDosage: {info["usage"]}\nTablets:
{info["tablets"]}')
plt.show()

# Create a Tkinter root window (hidden)

root = Tk()
root.attributes('-topmost', True)
root.withdraw()

# Open a file dialog for image selection

file_path = filedialog.askopenfilename(title="Select Image", filetypes=[("Image
files", "*.jpg;*.jpeg;*.png")])

# Check if a file is
selected if file_path: #
Example usage
predict_pill(file_path)

48
APPENDIX II

A2.1 OUTPUT IN GRAPH:

The graph explained the accuracy and loss of the data through 100 Epochs.

RESULT – ACCURACY:
The graph refers to the accuracy values recorded during the training phase, and validation
phase. These values are plotted against the number of epochs, which are represented on
the x-axis. The accuracy values are represented on the y-axis.

Fig.A2.1 Result – Variation of Accuracy

49
LOSS:
The graph refers to the loss values recorded during the training phase, and validation
phase. These values are plotted against the number of epochs, which are represented on
the x-axis. The loss values are represented on the y-axis.

Fig.A2.2 Result – Variation of Loss

50
A2.2 OUTPUT

Fig.A2.3 Identification of Pill

51
 REFERENCES

1. P. Chotivatunyu and N. Hnoohom, "Medicine Identification System on Mobile

Devices for the elderly," 2020 15th International Joint Symposium on Artificial

Intelligence and Natural Language Processing (iSAI-NLP), 2020, pp. 1-6, doi:

10.1109/iSAI-NLP51646.2020.9376837.

2. Sangamithra, B. "Personalized Ranking Mechanism Using Yandex Dataset on

Machine Learning Approaches." Proceedings of the International Conference on

Cognitive and Intelligent Computing. Springer, Singapore, 2022.

3. Ting, HW., Chung, SL., Chen, CF. et al. A drug identification model developed using

deep learning technologies: experience of a medical center in Taiwan. BMC Health

Serv Res 20, 312 (2020).

4. Rizwan Patan, Suresh Kallam, and Mohamed Yasin Noor Mohamed. "Segmentation

of nuclei in histopathology images using fully convolutional deep neural

architecture." In 2020 International Conference on computing and information

technology (ICCIT-1441), pp. 1-7. IEEE, 2020.

52
53