HAND GESTURE VOLUME CONTROL BASED ON COMPUTER

VISION AND MACHINE LEARNING

A PROJECT REPORT

Submitted by

LAKSHMAN K (212105445)

SARATHY M (212105462)

YADHU KRISHNA S (212105422)

In partial fulfillment of the requirement for the award of the Degree of

BACHELOR OF COMPUTER APPLICATION

UNIVERSITY OF MADRAS

Under the guidance of

Dr. G. MONIKA, M.Sc., Ph.D., HOD

JEPPIAAR COLLEGE OF ARTS AND SCIENCE

Padur, Chennai - 603103

2023 – 2024
 DECLARATION

We, Lakshman K (212105445), Sarathy M (212105462) and Yadhu Krishna S (212105422)

of III BCA, hereby declare that this project report on “HAND GESTURE VOLUME
CONTROL BASED ON COMPUTER VISION AND MACHINE LEARNING” submitted
to University of Madras in partial fulfilment of the requirement for the award of the
Degree of Bachelor of Computer science, under the guidance of Dr. G. MONIKA
M.S.c., Ph.D., HOD has not been submitted earlier to any other university or institute
for the award of any degree

Place: (Lakshman K)

Date: (Sarathy M)

(Yadhu Krishna S)
 BONAFIDE CERTIFICATE

This is to certify that the project titled “Hand Gesture Volume Control Based on Computer
Vision And Machine Learning” is the bonafide work done by Lakshman K (212105445),
Sarathy M (212105462) and Yadhu Krishna S (212105422) final year students of Jeppiaar
College of Arts and Science, Padur, Chennai in partial fulfilment of the requirement for the
award of the Degree of Bachelor of Computer Science during the academic year 2023-2024

PROJECT GUIDE

HEAD OF THE DEPARTMENT

Date:
 VIVA VOCE EXAMINATION
This viva voce examination of the project titled “Hand Gesture Volume Control Based

on Computer Vision And Machine Learning” is submitted by, Lakshman K (212105445),

Sarathy M (212105462) and Yadhu Krishna S (212105422) of III BCA. University of Madras

Examination held on ___________________

INTERNAL EXAMINER EXTERNAL EXAMINER

 ACKNOWLEDGEMENT

We would like to express our sincere thanks to our beloved Chairperson Dr.Regeena
J. Murli and our Director Mr. Murli S for motivating us in all our endeavours.

We wish to thank Dr.Lavanya L, Principal, Jeppiaar College of Arts and Science, for
the help and moral support extended to us.

Our profound thanks to Dr.Monika G, Head Department of Computer Science and

Application, Jeppiaar College of Arts and Science for her encouragement and valuable support
throughout this project.

Lakshman K

Sarathy M

Yadhu Krishna S
Chapter No Particulars Page No

1 Introduction 1
1.1 Hand Gesture Tracking
1.2 Parameters and challenges inHand Tracking
1.3 Analysis of techniques for Hand Gesture recognition 2
Scope
Objectives
Existing System
Proposed System
2 5
Software Used – OpenCV
Application and Funtionality

3 Algorithm Used
Hand Tracking Algorithm (Mediapipe) 8
Volume Control Al gorithm ( pycaw)
Web Designing Framework (Streamlit)

4 Sample Coding 11

5 Result Screenshot 16

6 Conclusion 22

References 23
 HAND GESTURE VOLUME CONTROL BASED ON COMPUTER

VISION AND MACHINE LEARNING

Abstract:

Human-machine interaction is important in human-machine partnerships. These encounters

are only feasible with efficient communication, whether verbal or nonverbal. Emotion
recognition, one of the important nonverbal mechanisms by which this communication
occurs, aids in determining the person's mood and state. Human-machine collaboration
becomes more natural when communication takes place through nonverbal channels such as
emotions. Machines can help us more if they can perceive and track the human hand for
gesture control. Hand Tracking and gesture based methods are prominent among the several
approaches to gesture recognition. This system works by detecting our hand from the
webcam. To identify our hand gestures, the hand detection system used to identify the hands
in a real-time video. The objective of this project is to control the volume of the devices by
using hand detection system to operate our devices with different hand gestures that will
increase and decrease the volume of our devices.
 CHAPTER - 1

INTRODUCTION

Hand gesture recognition for human computer interaction is an area of active research in
computer vision and machine learning (Maung, 2009). One of the primary goals of gesture
recognition research is to create systems, which can identify specific gestures and use them to
convey information or to control a device. Though, gestures need to be modelled in the spatial and
temporal domains, where a hand posture is the static structure of the hand and a gesture is the
dynamic movement of the hand. Being hand-pose one of the most important communication tools
in human’s daily life, and with the continuous advances of image and video processing techniques,
research on human-machine interaction through gesture recognition led to the use of such
technology in a very broad range of possible applications (Mitra and Acharya, 2007, Bourennane
and Fossati, 2010), of which some are here highlighted: • Virtual reality: enable realistic
manipulation of virtual objects using ones hands (Yoon et al., 2006, Buchmann et al., 2004), for
3D display interactions or 2D displays that simulate 3D interactions. • Robotics and Tele-presence:
gestures used to interact with robots and to control robots (Trigueiros et al., 2011) are similar to
fully-immersed virtual reality interactions, however the worlds are often real, presenting the
operator with video feed from cameras located on the robot. Here, for example, gestures can
control a robots hand and arm movements to reach for and manipulate actual objects, as well as
its movement through the world.

• Desktop and Tablet PC Applications: In desktop computing applications, gestures can

provide an alternative interaction to mouse and keyboard (Vatavu et al., 2012, Wobbrock et al.,
2007, Li, 2010, Kratz and Rohs, 2011). Many gestures for desktop computing tasks involve
manipulating graphics, or annotating and editing documents using pen-based gestures. • Games:
track a player’s hand or body position to control movement and orientation of interactive game
objects such as cars, or use gestures to control the movement of avatars in a virtual world. Play
Station 2 for example has introduced the Eye Toy (Kim, 2008), a camera that tracks hand
movements for interactive games, and Microsoft introduced the Kinect (Chowdhury, 2012) that is
able to track users full body to control games. • Sign Language: this is an important case of
communicative gestures. Since sign languages are highly structural, they are very suitable as test-
beds for vision-based algorithms (Zafrulla et al., 2011, Ong and Ranganath, 2005, Holt et al., 2010,
Tara et al., 2012). There are areas where this trend is an asset, as for example in the application of
these technologies on interfaces that can help people with physical disabilities, or areas where it
is a complement to the normal way of communicating. Sign language, for example, is the most
natural way of exchanging information among deaf people, although it has been observed that
they have difficulties in interacting with normal people. Sign language consists of a vocabulary of
signs in exactly the same way as spoken language consists of a vocabulary of words. Sign
languages are not standard and universal and the grammars differ from country to country.

The Portuguese Sign Language (PSL), for example, involves hand movements, body
movements and facial expressions (Wikipedia, 2012). The purpose of Sign Language Recognition
(SLR) systems is to provide an efficient and accurate way to convert sign language into text or
voice has aids for the hearing impaired for example, or enabling very young children to interact
with computers (recognizing sign language), among others. Since SLR implies conveying
meaningful information through the use of hand gestures (Vijay et al., 2012), careful feature
selection and extraction are very important aspects to consider In terms of hand gesture
recognition, there are basically two types of approaches: vision-based approaches and data glove
methods. This paper focuses on creating a vision-based approach, to implement a system capable
of performing posture and gesture recognition for real-time applications. Vision-based hand
gesture recognition systems were the main focus of the work since they provide a simpler and
more intuitive way of communication between a human and a computer. Using visual input in this
context makes it possible to communicate remotely with computerized equipment, without the
need for physical contact or any extra devices.

1.1Hand Gesture Recognition

Gesture recognition provides real-time data to a computer to make it fulfill the user’s
commands. Motion sensors in a device can track and interpret gestures, using them as the
primary source of data input. A majority of gesture recognition solutions feature a combination
of 3D depth-sensing cameras and infrared cameras together with machine learning systems.
Machine learning algorithms are trained based on labeled depth images of hands, allowing them
to recognize hand and finger positions.

Gesture recognition consists of three basic levels:

• Detection. With the help of a camera, a device detects hand or body movements, and a
machine learning algorithm segments the image to find hand edges and positions.
 • Tracking. A device monitors movements frame by frame to capture every movement and
provide accurate input for data analysis.

• Recognition. The system tries to find patterns based on the gathered data. When the system
finds a match and interprets a gesture, it performs the action associated with this gesture.
Feature extraction and classification in the scheme below implements the recognition
functionality.

HGR System

any solutions use vision-based systems for hand tracking, but such an approach has a lot of
limitations. Users have to move their hands within a restricted area, and these systems struggle
when hands overlap or aren’t fully visible. With sensor-based motion tracking, however, gesture
recognition systems are capable of recognizing both static and dynamic gestures in real time.

1.2 Parameters and challenges in Hand Tracking

Challenge 1–Object Interaction

Our hands are one of our main ways to effect change in the environment around us. Thus, one of
the main reasons to visualise hands in VR is to facilitate and encourage interactions with the virtual
environment. From opening doors to wielding weapons, computer-generated hands are an integral
part of many game experiences across many platforms. As outlined above, these manual
interactions are typically generated by reverse-engineering interactions with a held controller.
Challenge 2–Tracking Location

Tracking the location of our hand is a another major challenge that we face.To overcome this
challenge there is the package called “Mediapipe”.Through this package we can easily track the
location of our hand.We can talk about this package further.

1.3 Analysis of techniques for hand gesture recognition

A hand tracking system however, usually has a more variable number of interactions that it can
support, since hand tracking systems usually track either the volume of the hand, or the individual
joint and finger positions. This allows for a theoretically unlimited number of interactions with
digital objects, just as we would use our hands to interact with objects in the real world, but can
sometimes run into problems with occlusion – what does the system do when your fist is closed?
Where are your fingers when one hand is behind the other? Is that your left hand palm up, or your
right hand palm down? Because gesture systems are trained on a few specific gestures, they don’t
have the same problems, but they also don’t have the same flexibility.

Figure1.Hand Skeleton
 As an analogy that may help to understand the difference between hand tracking and gesture
recognition, think of the difference between a phone with a keypad, and a modern touchscreen.
The keypad has defined buttons, a limited number of them, which perform a specific set of
interactions very well, but a touchscreen can perform an unlimited number of actions, although
some of the time you might more easily hit the wrong key or letter because of the more nuanced
nature of the interactions. Neither system is inherently better or worse than the other, instead, it’s
important to choose which is optimal for your specific use case and user needs.

SCOPE:

Our project on hand gesture recognition in social media platforms has a broad scope that
touches on many crucial facets of this vital area. By offering a more approachable and userfriendly
experience for examining the gesture responses by the hand tracking , our suggested system seeks
to address some of the shortcomings of existing models.

Our system's ability to offer a comprehensive analysis of the live gesture controls to control
social media video with gesture position is one of its important capabilities.Users is able to control
their media devices without touching or having any external devices. Our project's lack of need
for a lab setting or any specific hardware or software is another important component. We intend
to increase the usability and accessibility of our system for a wider variety of users, regardless of
their location or level of technical proficiency, by providing it as an.exe file.

To ensure that our system delivers accurate and trustworthy results, our project will also use a
number of complex algorithms, including the mediapipe and the pycaw method. These algorithms
will assist us in precisely identifying and analyzing various hand gestures in live to control the
volume of media videos, enabling us to provide our users with insights and suggestions.

OBJECTIVES:

The objectives of our project are to develop a new model for hand detection in social media
that offers an overall analysis of gestures in the real time. The proposed methodology aims to solve
the limitations of existing models by eliminating the requirement for a laboratory setting and
making the technology accessible to a wider range of users. Our project has several specific goals,
• Develop a hand detection model: This objective is the core of the project, as it involves
the development of a model that can accurately detect the gestures in hand. This will
require a deep understanding of computer vision and machine learning, as well as expertise
in hand gesture recognition.
• Enable real-time hand detection: While many existing hand gesture detection models are
able to detect gesture accurately, they often require significant processing time to analyze
the position of the hand. This objective involves developing a model that can detect the
gestures in real-time, which will be critical for the improvement of the project.

• Provide overall analysis of gestures over time: One of the key drawbacks of many
existing hand gesture detection models is that they only provide certain hand gesture
controls only. This objective involves developing a model that can provide an overall
analysis of gestures over time via visualizing it, which will be useful in understanding how
gestures change throughout a video.
• Make the model accessible to users: A major barrier to the use of many hand tracking
detection models is that they require a laboratory environment or specialized tools to use.
This objective involves developing a model that is accessible to a wide range of users,
regardless of their location or resources. This will be achieved by making the model
available as an executable file that can be downloaded and installed on any
internetconnected device.

• Contribute to the development of social media: Hand gesture detection is a critical tool
in social media, as it can provide the capability of controlling the media volume to various
types of content. This objective involves contributing to the development of social media
by developing a model that is more accurate, accessible, and comprehensive than existing
model

EXISTING SYSTEM

Hand tracking technology allows for the detection and monitoring of hand movements in
real-time, enabling more natural and intuitive interactions with digital devices and
interfaces. These systems typically use a combination of sensors, such as cameras or radars,
along with advanced algorithms to interpret and track the position, orientation, and gestures
of the hands. One example is the Leap Motion controller, which uses infrared cameras to
 create a 3D model of the hands and fingers. This technology has applications in virtual and
augmented reality, where users can interact with virtual objects using their hands, as well
as in other fields like healthcare, robotics, and gaming. Hand tracking offers a more
immersive and hands-free way of interacting with technology, potentially revolutionizing
how we engage with digital interfaces in various domains.

PROPOSED SYSTEM

Streamlit is an open-source Python library that makes it easy to create web applications for
machine learning and data science projects. It provides a simple and intuitive way to build
interactive dashboards and data-driven applications, allowing you to focus on your data and
analysis rather than on the complexities of web development. Streamlit can be used to create a
wide range of applications, from simple data visualizations to complex machine learning
models.
 CHAPTER 2

SOFTWARE USED:

Computer Vision

Computer vision is a process by which we can understand the images and videos how they are
stored and how we can manipulate and retrieve data from them. Computer Vision is the base or
mostly used for Artificial Intelligence. Computer-Vision is playing a major role in self-driving
cars, robotics as well as in photo correction apps.

Open CV

OpenCV is the huge open-source library for the computer vision, machine learning, and image
processing and now it plays a major role in real-time operation which is very important in today’s
systems. By using it, one can process images and videos to identify objects, faces, or even
handwriting of a human. When it integrated with various libraries, such as NumPy, python is
capable of processing the OpenCV array structure for analysis. To Identify image pattern and its
various features we use vector space and perform mathematical operations on these features.

Figure 2

The first OpenCV version was 1.0. OpenCV is released under a BSD license and hence it’s
free for both academic and commercial use. It has C++, C, Python and Java interfaces and supports
Windows, Linux, Mac OS, iOS and Android. When OpenCV was designed the main focus was
realtime applications for computational efficiency.
Look at the following images
 Figure 3 Figure 4

from the above original image, lots of pieces of information that are present in the original
image can be obtained. Like in the above image there are two faces available and the person(I) in
the images wearing a bracelet, watch, etc so by the help of OpenCV we can get all these types of
information from the original image.

Applications of Open CV:

There are lots of applications which are solved using OpenCV, some of them are listed below

• face recognition

• Automated inspection and surveillance

• number of people – count (foot traffic in a mall, etc)

• Vehicle counting on highways along with their speeds

• Interactive art installations

• Anomaly (defect) detection in the manufacturing process (the odd defective products)

• Street view image stitching

• Video/image search and retrieval

• Robot and driver-less car navigation and control

• object recognition
OpenCV Functionality

• Image/video I/O, processing, display (core, imgproc, highgui)

• Object/feature detection (objdetect, features2d, nonfree)

• Geometry-based monocular or stereo computer vision (calib3d, stitching, videostab)

• Computational photography (photo, video, superres)

• Machine learning & clustering (ml, flann)

• CUDA acceleration (gpu)

 CHAPTER 3

ALGORITHM USED:

Mediapipe algorithm:

MediaPipe is a Framework for building machine learning pipelines for processing time-series
data like video, audio, etc. This cross-platform Framework works on Desktop/Server, Android,
iOS, and embedded devices like Raspberry Pi and Jetson Nano.

A Brief History of MediaPipe:

Since 2012, Google has used it internally in several products and services. It was initially
developed for real-time analysis of video and audio on YouTube. Gradually it got integrated into
many more products; the following are some.

1. Perception system in NestCam

2. Object detection by Google Lens

3. Augmented Reality Ads

4. Google Photos

5. Google Home

MediaPipe powers revolutionary products and services we use daily. Unlike power-hungry
machine learning Frameworks, MediaPipe requires minimal resources. It is so tiny and efficient
that even embedded IoT devices can run it. In 2019, MediaPipe opened up a new world of
opportunity for researchers and developers following its public release.

Figure 5
 MediaPipe Toolkit

MediaPipe Toolkit comprises the Framework and the Solutions. The following diagram
shows the components of the MediaPipe Toolkit.

Figure 6

Graphs

The MediaPipe perception pipeline is called a Graph. Let us take the example of the first
solution, Hands. We feed a stream of images as input which comes out with hand landmarks
rendered on the images.

Figure 7
 Pycaw Algorithm:

Pycaw was created by the creator Andre Miras in the Git Hub. This algorithm is used in the
mediapipe. As we all know we can only detect the object through mediapipe, but with this
algorithm we can control the volume of the devices with gestures.

Pcaw codes: from ctypes import cast, POINTER from

comtypes import CLSCTX_ALL from pycaw.pycaw import

AudioUtilities, IAudioEndpointVolume devices =

AudioUtilities.GetSpeakers() interface = devices.Activate(

IAudioEndpointVolume._iid_, CLSCTX_ALL, None)

volume = cast(interface,

POINTER(IAudioEndpointVolume)) volume.GetMute()

volume.GetMasterVolumeLevel()

volume.GetVolumeRange()

volume.SetMasterVolumeLevel(-20.0, None)

With this code of pycaw we can able to control the volume levels with the hand gesture
recognition.
 STREAMLIT (Web Designing Framework)

Streamlit is an open-source Python library that simplifies the creation of web applications for
data science and machine learning projects. With a straightforward API, Streamlit enables users
to develop interactive apps using Python syntax, including layouts, widgets for user input, and
dynamic content updates. It offers a variety of built-in widgets for user interaction, supports
popular data visualization libraries, and provides easy sharing and deployment options.
Additionally, Streamlit allows for the creation of custom components to extend app functionality,
making it a powerful yet accessible tool for building interactive data-driven applications.
CHAPTER 4

SAMPLE CODING:

import cv2 import time import numpy as np import

HandTrackingModule as htm import math from comtypes

import CLSCTX_ALL from pycaw.pycaw import

AudioUtilities, IAudioEndpointVolume

######################################

wCam, hCam = 1280, 720

######################################

cap =

cv2.VideoCapture(0)

cap.set(3, wCam)

cap.set(4, hCam) pTime =

detector = htm.handDetector(detectionCon=0.7)

devices = AudioUtilities.GetSpeakers()
interface = devices.Activate(
IAudioEndpointVolume._iid_, CLSCTX_ALL,
None) volume
= interface.QueryInterface(IAudioEndpointVolume)

# volume.GetMute() #

volume.GetMasterVolumeLevel()
volRange =

(volume.GetVolumeRange()) minVol =

volRange[0] maxVol = volRange[1] vol

= 0 volBar = 400 volPer = 0 while True:

success, img = cap.read()

img = detector.findHands(img) lmList =

detector.findPosition(img, draw=False) if

len(lmList) !=0:

#print(lmList[4],lmList[8])

x1, y1 = lmList[4][1], lmList[4][2]

x2, y2 = lmList[8][1], lmList[8][2]

cx, cy = (x1+x2)//2, (y1+y2)//2

cv2.circle(img, (x1, y1), 15, (255, 0, 255), cv2.FILLED)

cv2.circle(img, (x2, y2), 15, (255, 0, 255), cv2.FILLED)

cv2.line(img, (x1,y1),(x2,y2),(255, 0, 255),3)

cv2.circle(img, (cx, cy), 15, (255, 0, 255), cv2.FILLED)

length = math.hypot(x2 - x1, y2 - y1)

#print(length)

# Hand range 50 - 200

# volume range -65 - 0

 vol = np.interp(length,[50,200],[minVol, maxVol])

volBar = np.interp(length, [50, 200], [400, 150]) volPer

= np.interp(length, [50, 200], [0, 100]) print(int(length),

vol) volume.SetMasterVolumeLevel(vol, None)

if length<50:

cv2.circle(img, (cx, cy), 15, (0, 255, 0), cv2.FILLED)

cv2.rectangle(img, (50, 150), (85, 400), (255,0, 0), 3) cv2.rectangle(img, (50,

int(volBar)), (85, 400), (255, 0, 0), cv2.FILLED)

cv2.putText(img,f'{int(volPer)} %',(40,450), cv2.FONT_HERSHEY_COMPLEX,

1,(255, 0, 0),3)

cTime = time.time() fps = 1/(cTime-pTime) pTime = cTime

cv2.putText(img,f'FPS: {int(fps)}',(40,50), cv2.FONT_HERSHEY_COMPLEX,

1,(255,0,0),3)

cv2.imshow("Img",img)

cv2.waitKey(1)
CHAPTER 5

RESULT SCREENSHOT:

Figure 8

Figure 9
Figure 10

Figure 11
 CHAPTER 6

CONCLUSION

Hand gestures are a powerful way for human communication, with lots of potential
applications in the area of human computer interaction. Vision advantages compared with
traditional devices. However, hand gesture recognition is a difficult problem and the current work
is only a small contribution towards achieving the results needed in the The main objective of this
work was to study and implement solutions that could be generic enough, with the help of machine
learning algorithms, allowing its application in a wide range of human interfaces, for online
gesture and posture recognition.
REFERENCES:

[1] A. Mehrabian, "Communication without words", Psychol. Today, (1986).

[2] U. Sderstrm and H. Li. Emotion recognition and estimation from tracked lip
features.

[3] Widanagamaachchi WN. Facial emotion recognition with a neural network

approach.
University of Colombo; 2009.

[4] P. Ekman and W. Friesen, "Pictures of Facial Affect. Consulting Psychologist",

(1976).

[5] Shbib, Reda & Zhou, Shikun. (2015). Facial Expression Analysis using Active
Shape Model. International Journal of Signal Processing, Image Processing and
Pattern Recognition.8.9-22.
10.14257/ijsip.2015.8.1.02.

[6] Goren D, Wilson HR. Quantifying facial expression recognition across viewing
conditions.
Vision Res. 2006 Apr;46(8-9):1253-62. Epub 2005 Dec 20. PubMed PMID:16364393.

[7] https://opencv.org/

[8] Kari Pulli, Anatoly Baksheev, Kirill Kornyakov, and Victor Eruhimov. 2012.
Realtime Computer Vision with OpenCV. Queue 10, 4, Pages 40 (April 2012), 17
pages. DOI: http://dx.doi.org/10.1145/2181796.2206309

[9] https://docs.scipy.org/doc/numpy-dev/user/quickstart.html

[10] https://docs.python.org/2/library/glob.html

[11] http://www.tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm
[12] http://docs.python.org/3.4/library/random.html

[13] - Kanade, T., Cohn, J. F., & Tian, Y. (2000). Comprehensive database for facial
expression analysis. Proceedings of the Fourth IEEE International Conference on
Automatic
Face and Gesture Recognition (FG'00), Grenoble, France, 46-53. - Lucey, P., Cohn, J. F.,
Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The Extended Cohn-Kanade
Dataset (CK+): A complete expression dataset for action unit and emotion-specified
expression.
Proceedings of the Third International Workshop on CVPR for Human

[14] GitHub - AndreMiras/pycaw: Python Core Audio Windows Library