ISL Gesture Recognization using CNN 2023-24

CHAPTER 1
INTRODUCTION

Sign Language is also known as a vision-based language which uses a variety of visual
signs like body movements, hand movements, locality of hands, facial expressions,
orientation etc. A gesture as defined in a dictionary means “Motion of limbs or, body
made to express thought or to emphasize speech”. A gesture consists of meaningful
actions which includes physical motion of body parts like hand, head, Arms, fingers
and facial expressions etc. These gestures convey meaningful information. However,
the majority of the people Are not aware of the semantics of these gestures and they
face difficulty in understanding their language. Due to this the impaired community are
not able to communicate to the outside world and this causes a gap between the normal
community and the impaired community. So, they take an assistance of another person
to convey their messages.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 1

ISL Gesture Recognization using CNN 2023-2024

1.1 Existing system

Related Work
1. Sign Language Recognition Techniques: A Comprehensive Review’This review
article provides an overview of various techniques used in sign language
recognition, with a focus on both glove-based and vision-based approaches. The
authors highlight the significance of vision-based methods, emphasizing the
advantages of not requiring additional hardware like Gloves. The review covers key
advancements in the field and discusses the challenges associated with recognizing
diverse sign languages, laying the foundation for the importance of addressing the
unique characteristics of Indian Sign Language (ISL).[1]
2. Vision-Based Gesture Recognition Systems for Sign Language: A Survey-This
survey paper delves into the vision based approach for gesture recognition in sign
languages. It outlines the evolution of vision-based systems, discussing the
transition from early sensor-based approaches to contemporary image processing
algorithms. The survey emphasizes the importance of dataset availability and the
impact of varied sign language grammatical structures on recogniti on accuracy.
This Is particularly relevant to our work in ISL, as it sheds light on The challenges
specific to the language. [2]
3. Deep Learning Models for Sign Language Recognition: A comparative analysis
focusing on the application of deep learning models, particularly convolutional
neural network of ISL compared to American Sign Language (ASL).- Challenge:
The lack of a standardized dataset and variations in ISL pose challenges to
researchers. The scarcity of literature on ISL recognition techniques implies a need
for more dedicated efforts in this specific domain.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 2

ISL Gesture Recognization using CNN 2023-24

1.2 Problem Statement

“Indian Sign Language Gesture Recognition Using Convolutional Neural Network
(CNN)”

The goal of this project is to develop a CNN-based model for accurate recognition of
static Indian Sign Language (ISL) gestures. The key challenges include handling
variations in hand positioning, occlusions, and the complexity of the ISL gesture set.
This project aims to address the problem of Indian Sign Language gestures recognition
using CNN, ensuring effective communication for individuals with hearing and speech
impairments

1.3 Objectives
1. Design and train a CNN model to identify static ISL gestures from input images.
2. Achieve high accuracy in recognizing a diverse set of ISL gestures.
3.Develop a model that can effectively manage changes in hand positioning & lighting
conditions.

1.4 Scope
The scope of Indian Sign Language (ISL) gesture recognition using Convolutional
Neural Networks (CNN) is quite promising and multifaceted. Here are some key areas
where it can have significant impact:
1. Communication for the Deaf and Hard of Hearing real-time translation: CNNs can
be used to develop systems that translate ISL gestures into text or spoken language in
real-time, facilitating better communication. Accessibility tools: Apps or devices that
recognize ISL can help deaf individuals interact more effectively with those who don’t
know sign language.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 3

ISL Gesture Recognization using CNN 2023-2024

2. Education and Learning aids: Interactive tools for learning ISL can be created,
helping both hearing-impaired individuals and those interested in learning the
language. Teaching resources: Enhanced educational materials for schools and
institutions that cater to deaf students.
3. Technology Integration smart environments: Integrating ISL recognition in smart
homes and offices to allow control of devices and systems through gestures. Human-
Computer Interaction (HCI): Developing more intuitive and inclusive user interfaces
for computers and mobile devices.
4. Healthcare telemedicine: Enabling better communication between healthcare
providers and deaf patients through sign language recognition.
5. Assistive technologies: Enhancing existing assistive devices for individuals with
speech and hearing impairments.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 4

ISL Gesture Recognization using CNN 2023-24

CHAPTER 2

LITERATURE SURVEY

 Sign Language Recognition Techniques: A Comprehensive Review" provides

an overview of sign language recognition techniques, highlighting the
importance of vision-based methods. It emphasizes challenges in recognizing
diverse sign languages.
 Vision-Based Gesture Recognition Systems for Sign Language: A Survey -
Outlines the evolution of vision-based systems for sign language recognition,
discussing the impact of dataset availability and variation in grammatical
structures on recognition accuracy.
 Deep Learning Models for Sign Language Recognition: A Comparative
Analysis – analysis the performance of deep learning models, particularly
Convolutional Neural Networks (CNNs), in sign language recognition,
highlighting their advantages in handling complex spatial and temporal
features.
 Challenges in Sign Language Recognition: A Case Study on Indian Sign
Language - Explores the specific challenges faced in ISL recognition, such as
the lack of standardized datasets and linguistic variations.
 Indian Sign Language Datasets: A Review and Analysis - Critically analysis
existing ISL datasets, discussing their limitations and providing
recommendations for future dataset development

Dept. of ISE SKSVMACET, Lakshmeshwar Page 5

ISL Gesture Recognization using CNN 2023-2024

CHAPTER 3

DOMAIN ANALYSIS

Introduction
Domain analysis involves understanding the problem space in which a system
operates, identifying the key elements, and defining the requirements for the system.
In the context of our project on Indian Sign Language (ISL) Gesture Recognition
using Convolutional Neural Network (CNN), domain analysis helps us understand
the specifics of sign language recognition, the challenges involved, and the
technological requirements.

Purpose
The purpose of this domain analysis is to gather, analyze, and define the
requirements and functionalities needed for developing a robust and efficient ISL
gesture recognition system. This analysis also aids in identifying the stakeholders,
understanding the system’s context, and establishing the foundation for the system’s
architecture and design.

Scope
The scope of this analysis includes:
 Understanding Indian Sign Language: Studying the gestures, signs, and their
meanings in ISL.
 Image Processing Techniques: Identifying and analyzing the methods for
capturing, processing, and segmenting hand images.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 6

ISL Gesture Recognization using CNN 2023-24

 Machine Learning and CNN: Exploring the use of Convolutional Neural

Networks for image classification and gesture recognition.
 System Requirements: Defining the functional and non-functional
requirements for the system.
 Stakeholders: Identifying the primary users and stakeholders of the system.
 Indian Sign Language (ISL): Indian Sign Language is a visual language used
by the deaf and hard-of-hearing community in India. It uses hand gestures,
facial expressions, and body movements to convey meaning.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 7

ISL Gesture Recognization using CNN 2023-2024

CHAPTER 4

REQUIREMENT SPECIFICATION

4.1 Hardware Requirements

 Processor: Intel Core i5 or equivalent AMD processor (minimum).Intel Core i7

or equivalent AMD processor (recommended for faster performance).
 Memory (RAM): 8 GB RAM (minimum).16 GB RAM or more (recommended
for handling large datasets and faster training).
 Storage: At least 256 GB of available storage (minimum).SSD with 512 GB or
more (recommended for faster data access and model training).
 Graphics Processing Unit (GPU): NVIDIA GPU with CUDA support (e.g., GTX
1060 or better) for faster training and inference.
 CUDA- enabled GPU with at least 4 GB VRAM (minimum).NVIDIA RTX
series GPU with 8 GB or more VRAM (recommended).
 Display: Full HD display (1920x1080 resolution) for better visualization and
development experience.

4.2 Software Requirements

 Operating System: Windows 10 or later, macOS, or a Linux distribution (Ubuntu

18.04 or later). Programming Language : Python 3.6 or later.
 Integrated Development Environment (IDE):Visual Studio Code, PyCharm, or
Jupyter Notebook.
 Tensor flow : For building and training the CNN model.
Dept. of ISE SKSVMACET, Lakshmeshwar Page 8
ISL Gesture Recognization using CNN 2023-24

 Keras: High-level neural networks API.

 OpenCV: For image capture and processing tasks.
 NumPy: For numerical computations.
 Pandas: For data manipulation and analysis.
 Matplotlib/Seaborn: For data visualization and plotting.
 Scikit-learn: For model evaluation and performance metrics.

4.3 Functional Requirements

 Image Capture: Capture and process hand gesture images in real-time using a
camera.
 Hand ROI Segmentation: Detect and segment the hand region from the captured
image.
 Finger Segmentation: Segment individual fingers from the hand ROI.
 Image Normalization: Normalize segmented images to a consistent size and
format for the CNN model.
 Gesture Recognition: Classify gestures using the CNN model with high
accuracy.
 Model Training and Evaluation: Train the CNN model on labeled datasets and
evaluate its performance.
 User Interface: Provide an intuitive interface for capturing gestures and
displaying results.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 9

ISL Gesture Recognization using CNN 2023-2024

4.4 Non-Functional Requirements

 Performance: Ensure real-time processing with minimal latency.

 Scalability: Handle a large number of users and support new gesture additions.
 Accuracy: Maintain high recognition accuracy across different conditions.
 Robustness: Handle variations in background, lighting, and hand positions
effectively.
 Usability: Offer an easy-to-use interface requiring minimal user training.
 Security: Ensure the privacy and security of captured images and user data.
 Maintainability: Design for easy maintenance and updates with well-documented
code.
 Portability: Ensure compatibility across different devices and platforms.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 10

ISL Gesture Recognization using CNN 2023-24

CHAPTER 5

PROPOSED METHODOLOGY

The proposed methodology involves a multi-step process, Starting with image pre-
processing techniques such as RGB conversion, grayscale processing, and thresholding
for binary Image generation. A skin mask is applied to focus on relevant regions, and
data augmentation techniques, like image rotation, enhance dataset diversity. These
processed images are then fed into a deep learning model, specifically a ResNet50
model Using Convolutional Neural Network (CNN), designed to learn And classify
ISL gestures. The model architecture comprises convolutional layers, max-pooling
layers, and fully connected layers. The use of data augmentation during training
enhances the model’s robustness to variations in input data.

The model starts by importing essential libraries and setting up paths. The dataset is
loaded, shuffled, and split into training and testing sets. Various image processing
functions, such as simple vision and canny vision, are defined to enhance the dataset.
Visualizations illustrate the impact of these techniques. The data is then augmented
using Keras Image Data Generator for better model generalization. The model, a

Dept. of ISE SKSVMACET, Lakshmeshwar Page 11

ISL Gesture Recognization using CNN 2023-2024

Convolutional Neural Network (CNN), is constructed using Keras, featuring

convolutional layers, batch normalization, dropout for regularization, and fully
connected layers. The model is compiled with Adam optimizer and categorical cross-
entropy loss. Training incorporates early stopping and model check pointing– to
prevent over fitting. Evaluation on a separate test set is followed by visualizing
predictions alongside original images. This research design aims to contribute to ISL
gesture recognition, employing a robust combination of image processing and deep
learning techniques.The implementation phase utilizes a dataset of ISL gestures
collected from diverse sources. Image data augmentation is employed to increase the
diversity of the training dataset. The dataset is split into training and testing sets, and a
CNN using ResNet50 model is trained using the training set. The model’s performance
is evaluated on the testing set using metrics such as accuracy, precision and recall.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 12

ISL Gesture Recognization using CNN 2023-24

Dataset Collection and Description

“We collected the dataset from Kaggle

https://www.kaggle.com/datasets/vaishnaviasonawane/indiansign-language-dataset [7],
which consists of various Indian sign Language gestures. The dataset is organized into
35 folders, each corresponding to alphabets (A to Z) and digits (0 to 9). Alphabets
folder contains 500 images and digits folder contains 600 images, resulting in a diverse
and substantial dataset for training and testing. The labels assigned to the folders range
from 0 to 34, providing a structured mapping of gestures for model training. In total,
the dataset comprises 18400 images, with 3500 images allocated for testing and 14000
for training.

Analysis Method

For image preprocessing, a standardized size was applied to resize the images, ensuring
uniformity in their dimensions. Subsequently, a series of transformations were
implemented, including RGB conversion to capture color information, grayscale
processing for simplified representation, and thresholding to generate binary images.
Additionally, a skin mask was employed to focus on relevant regions of interest. To
enhance the diversity of the dataset, data augmentation techniques were incorporated,
such as image rotation. These steps collectively contribute to a comprehensive
preprocessing pipeline, preparing the images for subsequent analysis and feature
extraction.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 13

ISL Gesture Recognization using CNN 2023-2024

Dept. of ISE SKSVMACET, Lakshmeshwar Page 14

ISL Gesture Recognization using CNN 2023-24

CHAPTER 6

EXPERIMENTAL RESULTS

Dept. of ISE SKSVMACET, Lakshmeshwar Page 15

ISL Gesture Recognization using CNN 2023-2024

Shows overall accuracy evolution of model. In which it has been seen that validation
loss is decreasing and validation accuracy is increasing noticeabl achieved training
accuracy of approximately 93.49%. achieved a validation accuracy of approximately
100% on the ISL gesture recognition task.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 16

ISL Gesture Recognization using CNN 2023-24

CHAPTER 7

CONCLUSION AND FUTURE SCOPE

The Sign Language Recognition (SLR) system is a method for recognising a collection
of formed signs. The significance of gesture recognition can be seen in the
development of effective human-machine interactions. We attempted to build a
ResNet50 model using a Convolutional Neural Network in this project. This results in a
validation accuracy of about 100 the future scope of Indian Sign Language (ISL)
gesture recognition involves integrating advanced communication technologies for
real-time translation into text or speech, fostering effective communication between the
deaf and hearing communities. Additionally, potential lies in immersive experiences
through VR and AR applications, enhancing inclusivity for individuals.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 17

ISL Gesture Recognization using CNN 2023-2024

CHAPTER 8

REFERENCES

[1] M. Varsha and C. S. Nair, “Indian Sign Language Gesture Recognition Using Deep
Convolutional Neural Network,” 2021 8th International Conference on Smart
Computing and Communications (ICSCC), Kochi, Kerala, India, 2021, pp. 193-197,
doi: 10.1109/ICSCC51209.2021.9528246.

[2] Rokade, Yogeshwar Jadav, Prashant. (2017). “Indian Sign Language recognition
System”. International Journal of Engineering and Technology. 9. 189-196.
DOI:10.21817/ijet/2017/v9i3/170903S03.

[3] Rupali Kadwade, Akanksha Tangade, Neha Pakhare, Samiksha Kolhe, Hajara
Waikar, S. J. Wagh, 2023, “Indian Sign Language Recognition System”,
INTERNATIONAL JOURNAL OF ENGINEERING RESEARCH TECHNOLOGY
(IJERT) Volume 12.

[4] J. Joy, K. Balakrishnan and M. Sreeraj, “SignQuiz: A Quiz Based tool for Learning
Fingerspelled Signs in Indian Sign Language Using ASLR,” in IEEE Access, vol. 7, pp.
28363-28371, 2019, doi:10.1109/ACCESS.2019.2901863.

[5] Shagun Katoch, Varsha Singh, Uma Shanker Tiwary, “Indian Sign Language
recognition system using SURF with SVM and CNN”, Array, Volume 14,
2022,100141, ISSN 2590-0056,https://doi.org/10.1016/j.array.2022.100141.

Dept. of ISE SKSVMACET, Lakshmeshwar Page 18

ISL Gesture Recognization using CNN 2023-24

[6] K. Shenoy, T. Dastane, V. Rao and D. Vyavaharkar, “Real-time Indian Sign

Language (ISL) Recognition,” 2018 9th International Conference on Computing,
Communication and Networking Technologies (ICCCNT), Bengaluru, India, 2018, pp.
1-9, doi: 10.1109/ICCCNT.2018.8493808.

[7]https://www.kaggle.com/datasets/vaishnaviasonawane/indian-signlanguage-dataset .

Dept. of ISE SKSVMACET, Lakshmeshwar Page 19