A Mini Project Report on

CONTROL MOUSE USING HAND GESTURE

Project submitted in partial fulfillment of the requirements for the award of the degree of

BACHELLOR OF TECHNOLOGY
IN

COMPUTER SCIENCE AND ENGINEERING

BY

T.TEJA (21C91A05G8)

P.REVANTH (21C91A05I6)
G.ANUHYA (22C95A0516)

Under the Esteemed guidance of

Mrs.CH.VIJAYA JYOTHI
Assistant Professor
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

HOLY MARY INSTITUTE OF TECHNOLOGY & SCIENCE

(COLLEGE OF ENGINEERING)

(Approved by AICTE New Delhi, Permanently Affiliated to JNTU Hyderabad, Accredited by NAAC with ‘A’
Grade) Bogaram (V), Keesara (M), Medchal District -501 301.

2023 - 2024
 HOLY MARY INSTITUTE OF TECHNOLOGY & SCIENCE
(COLLEGE OF ENGINEERING)
(Approved by AICTE New Delhi, Permanently Affiliated to JNTU Hyderabad, Accredited by NAAC with ‘A’ Grade) Bogaram
(V), Keesara (M), Medchal Dist-501301.

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

CERTIFICATE
This is to certify that the major project entitled “CONTROL MOUSE USING HAND GESTURE” is being submitted
by T.TEJA (21C91A05G8), P.REVANTH (21C91A0G8),G.ANUHYA(22C95A0516) in Partial
fulfillment of the academic requirements for the award of the degree of Bachellor of Technology in
“COMPUTER SCIENCE AND ENGINEERING” from HOLY MARY INSTITUTE OF TECHNOLOGY &
SCIENCE, JNTU Hyderabad during the year 2023- 2024.

INTERNAL GUIDE HEAD OF THE DEPARTMENT

Mrs.CH.VIJAYA JYOTHI Dr .B.NARSIMHA

Assistant Professor Professor & HoD
Dept. of Computer Science & Engineering Dept. of Computer Science & Engineering

EXTERNAL EXAMINER

1
 ACKNOWLEDGEMENT

The satisfaction and euphoria that accompany the successful completion of any task would be
incomplete without the mention of the people who made it possible, who’s constant guidance and
encouragement crowns all effort with success.

We take this opportunity to express my profound gratitude and deep regards to our Guide
Mrs.CH.VIJAYA JYOTHI, Assistant Professor, Dept. of Computer Science & Engineering,
Holy Mary Institute of Technology & Science for his / her exemplary guidance, monitoring and
constant encouragement throughout the project work.

Our special thanks to Dr. B. Narsimha, Head of the Department, Dept. of Computer Science &
Engineering, Holy Mary Institute of Technology & Science who has given immense support
throughout the course of the project.

We also thank Dr. P. Bhaskara Reddy, the honorable Director of my college Holy Mary Institute
of Technology & Science for providing me the opportunity to carry out this work.

At the outset, we express my deep sense of gratitude to the beloved Chairman A. Siddartha Reddy
of Holy Mary Institute of Technology & Science, for giving me the opportunity to complete my
course of work

We are obliged to staff members of Holy Mary Institute of Technology & Science for the valuable
information provided by them in their respective fields. We are grateful for their cooperation during
the period of my assignment.

Last but not the least we thank our Parents, and Friends for their constant encouragement without
which this assignment would not be possible.

T.TEJA (21C91A05G8)
P.REVANTH (21C91A05I6)

G.ANUHYA (22C95A0516)

2
 DECLARATION

This is to certify that the work reported in the present project titled “CONTROL
MOUSE USING HAND GESTURE ” is a record of work done by us in the
Department of Computer Science & Engineering, Holy Mary Institute of Technology
and Science.

To the best of our knowledge no part of the thesis is copied from

books/journals/internet and wherever the portion is taken, the same has been duly
referred to in the text . The reports are based on the project work done entirely by us
not copied from any other source.

T.TEJA(21C91A05G8)

P.REVANTH(21C91A05I6)

G.ANUHYA(22C95A0516)

1
Contents Page Number
CHAPTER 1:INTRODUCTION 6
1.1 INTRODUCTION 6
1.2 MOTIVATION 6
1.3 AIM AND SCOPE OBJECTIVE 7
1.4. OUR CONTRIBUTION 7

CHAPTER 2: LITERATURE REVIEW 7

2.1 PROPOSED SYSTEM 7
2.2 ADVANTAGES 8
2.3 USE OF PROPOSED WORK 8
2.4 EXISTING SYSTEM 8
2.5 DISADVANTAGES 9

CHAPTER 3: ALGORITHMS AND FLOWCHART 9

3.1 ALGORITHM 10
3.2 FLOWCHART 11

CHAPTER 4: SYSTEM SPECIFICATIONS 12

4.1 Software Specifications 12
4.1 Hardware Specifications 12

2
CHAPTER 5: METHODOLOGY 13
5.1 HAND CONTROL EXTRACTION 13
5.2 HAND TRACKING 14
5.3 GESTURE RECOGNITION 15
5.2 CURSOR CONTROL 16

CHAPTER 6: MODEL 17
6.1 DIAGRAM 17

CHAPTER 7: RESULT 18
7.1 PERFORMANCE ANALYSIS 18

CHAPTER 8: TESTING 19
8.1 TEST CASE 19

9. SOURCE CODE 24

10. OUTPUT IMAGES 33

11.FUTURE SCOPE 38

12. CONCLUSION 39

13. REFERENCES 41

3
ABBREVIATIONS USED

GUI Graphical User Interface

HCI Human Computer Interaction

MHI Motion History Images

IDE Integrated Development Environment

OpenCV Open Source Computer Vision

NUI Natural User Interface

4
 ABSTRACT :
As technology grows day by day, people are becoming accustomed to new
technologies where less time and Harder work is spent and more time to relax our minds
and bodies. Gesture-based interaction systems are Becoming increasingly prominent
both in the workplace and in the home. The projects anticipate to construct a Handheld
device that may be used as an included tool to connect with a PC that may be linked to
a number of PC Operations We have suggested a Capture move technique with the
gesture of a hand and a feature using a Webcam.
In this way, it is basically a system that uses image processing, technique to control the
position of the Cursor with the bare hands without using any electronic device.several
new interfaces provides Possibilities for the Realisation of human-computer interaction
including the controlling systems based on human Movement the human-computer
interaction is able to detect the cognitive actions.

There have been many developments towards the Human Computer Interaction(HCI).
Many modules have been developed to help the physical world interact with the digital
World.

5
1.INTRODUCTION

1.1 INTRODUCTION
Computer technology has tremendously grown up over the past years and has become a necessary part of
everyday life. The first Computer part that human’s access for Human Computer Interaction (HCI) is the mouse.
The mouse is less appropriate for HCI in some reality things, like with Human Machine Interaction. There have
been many explorations on different ways to control the computer mouse for HCI. The first natural and intuitive
technique for HCI, that is a viable replacement for the computer mouse can be the introduction of hand gestures
into computer systems. This project is thus mainly focused towards developing a computer management system
using hand gestures. Most laptops and systems which are home these days are equipped with webcams that
have greatly help to deploy in security applications utilizing face recognition so as to get the extended use of a
digital camera, it may be utilized for vision based mostly CC, which might constructively helps us to eliminate
the necessity for a mouse or mouse pad. The utilization of a digital camera may be greatly extended to different
HCI (Human Computer Interaction) applications like a signal language database or motion controller. Over the
past decades there have been vital advancements in HCI technologies for gaming functions. We are looking
forward to applying techniques to use a camera and computer technology, such as image segmentation and
gesture recognition to manage tasks that are performed by Mouse and show how it can perform everything that
the current mouse devices can.

1.2 MOTIVATION
The main purpose of implementing this system is to help the ambitious physically challenged people
who want to work on a computer. There are many ambitious physically challenged people in society who
want to work on a computer. This system would help them. There are many people who don't feel
comfortable with a touchpad. This system would provide flexibility to people and would be convenient

6
for users who are not comfortable with a touchpad. We all want to mirror our experiences in the real
world into a computer. This system would allow that.

1.3 AIM AND SCOPE OBJECTIVE

This project attempts to use gestures to control the computer cursor rather than pointing and clicking a mouse
or directly touching the display, lowering hardware costs and increasing the quality of life for physically
challenged persons. One of the project's goals is to assist people who are suffering from Carpal Tunnel
Syndrome. Carpal Tunnel Syndrome affects more than 10 million people in India each year. This would be
beneficial to them.

1.4.OUR CONTRIBUTION

We developed an algorithm to control mouse movement and other mouse activities by using hand gestures to
eliminate the usage of mouse as far as possible. To summarize, this paper makes the following contributions:
● This project provides a major prevention to the Carpal Tunnel Syndrome. Recently all the work has been
running on PCs. Due to the heavy usage of mouse, many people are affected with Carpal Tunnel Syndrome.
As per the statistics, there are more than 10 million cases per year in India. Carpal tunnel syndrome is a
common condition that causes pain, numbness, and tingling in the hand and arm. The condition occurs when
one of the major nerves to the hand, the median nerve is squeezed or compressed as it travels through the
wrist.

● The algorithm works in such a way that people with physical challenges could use the PCs easily through
hand gestures.

● We have implemented this project in our homes so that the elderly people find it much easier to interact
with the PCs.

2.LITERATURE REVIEW
2.1 PROPOSED SYSTEM
Using the current system even-though there are a number of quick access methods available for the hand and
mouse gesture for the laptops, using our project we could make use of the laptop or web-cam and by recognizing
the hand gesture we could control mouse and perform basic operations like mouse pointer controlling, select
and deselect using left click, and a quick access feature for file transfer between the systems connected via
network LAN cable. The project done is a “Zero Cost” hand recognition system for laptops, which uses simple
algorithms to determine the hand, hand movements and by assigning an action for each movement[2]. But we
have mainly concentrated on the mouse pointing and clicking actions along with an action for the file transfer

7
between connected systems by hand action and the movements. The system we are implementing which is been
written in python code be much more responsive and is easily implemented since python is a simple language
and is platform independent with a flexibility and is portable which is desirable in creating a program which is
focused in such an aim for creating a Virtual Mouse and Hand Recognition system. The system be much more
extendable by defining actions for the hand movement for doing a specific action. It could be further modified
to any further extent by implementing such actions for the set of hand gestures, the scope is restricted by your
imagination.
2.2 ADVANTAGES
• Easy to install
• It can be used as easy mouse control for users
• It is not an electronic based system so one can easily make use of laptops to install.

2.3 USE OF PROPOSED WORK

This Virtual Mouse Hand Recognition application uses a simple color cap on the finger without the additional
requirement of the hardware for the controlling of the cursor using simple gestures and hand control. This is
done using vision based hand gesture recognition with inputs from a webcam.

2.4 EXISTING SYSTEM

The existing system consists of the generic mouse and trackpad system of monitor controlling and the non-
availability of a hand gesture system. The remote accessing of monitor screen using the hand gesture is
unavailable. Even-though it is largely trying to implement the scope is simply restricted in the field of virtual
mouse. The existing virtual mouse control system consists of the simple mouse operations using the hand
recognition system, where we could perform the basic mouse operation like mouse pointer control, left click,
right click, drag etc. The further use of the hand recognition is not been made use of. Even-though there are a
number of systems which are used for hand recognition, the system they made used is the static hand
recognition which is simply recognition of the shape made by hand and by defining an action for each shape
made, which is limited to a number of defined actions and a large amount of confusion.

2.5 DISADVANTAGES

• It Requires a Lot of Memory

• The Alarm it has Limited Accuracy

8
3.ALGORITHM
Step 1: Start

Step 2: Start the webcam video capture and initialize the system.

Step 3: Frame capture with a webcam.

Step 4: Using Media Pipe and OpenCV, detect hands and hand tips and draw hand landmarks and a box around

the hand.

Step 5: Draw a rectangle around the computer window area where we'll be using the mouse.

Step 6: Determine which finger is raised.

Step 6.1: The gesture is neutral if all five fingers are up, and the next step is taken.

Step 6.2: The cursor moves to step 2 if both the middle and index fingers are raised.

Step 6.3: A double click is performed when both index and middle fingers are joined side by side, and step 2

is performed.

Step 6.4: If both index and middle fingers are down, perform a left click and proceed to step 2.

Step 6.5: If the middle finger is down and the index finger is up, the right click is performed and the process

proceeds to step 2.

Step 6.6: Volume up and down are accomplished by joining the thumb and index fingers and moving them up

and down.

Step 7: To exit, press the EXIT key.

The preceding algorithm shows how to simulate virtual mouse control with hand gestures step by step. This
makes it easier to swap a physical mouse for a virtual mouse. This will aid in the conversion of input hand
gestures into mouse clicking functions.

4.FLOWCHART
The preceding procedure, i.e. the algorithm, is depicted diagrammatically in flow chart Fig-1.1.

9
Fig-1.1 flowchart

10
5.METHODOLOGY
5.1 HAND CONTOUR EXTRACTION
After obtaining the skin segmented binary image, edge detection is used to obtain the hand contour inside the
image. Many edge detection algorithms are available, including Laplacian edge detection, canny edge
detection, and boundary discovery. The OpenCV function cvFindContours() searches the picture for contours
using an order finding edge detection algorithm. The key benefit of the border finding edge detection technique
is that each contour identified in the image is saved in an array. This means that we can determine the hand
shape by analyzing each contour in the image individually. The canny and Laplacian edge detectors can
recognize the contours in a picture, but they don't give us access to every single one. As a result, in the planned
design, the boundary finding edge detection technique was adopted. We have a propensity to be interested in
extracting the hand contour as part of the contour extraction approach so that form analysis may be performed
on it to determine the hand gesture. The little contours are likely to be noise and should be ignored. The
assumption was made that the hand contour is the largest contour in the image, thereby ignoring all noise
contours. If the face shape is larger than the hand contour, this assumption may be false. To address this flaw,
the face section of the frame should be removed. The assumption was that the hand was the only moving item
in the photograph, and that the face was relatively motionless in comparison to the hand. This means that
background subtraction can be used to remove rid of the image's stationary pixels, as well as the face region.
This is frequently done using the OpenCV function "BackgroundSubtractorMOG2."

5.2 HAND TRACKING

Hand gestures are used to control the pointer's movement. The center of the palm should be discovered initially
in order to locate the hand. The approach for determining the hand center form has the benefit of being simple
and straightforward to perform. The shortest distance between each point inside the inscribed circle and the
contour was measured, with the point with the greatest distance being recorded as the center. The radius of the
hand was calculated as the distance between the center of the hand and the hand contour.The hand center was
determined for each successive frame, and the tip of the finger was known and used for hand tracking utilizing
the hand center.

5.3 GESTURE RECOGNITION

The proposed design's gesture recognition technology is a hybrid of two technologies proposed by Yeo and
Balazs. The hand contour's convexity faults must first be computed. The hand contour's convexity defects were
determined using the OpenCV built-in function "cvConvexityDefects." The convexity defect's parameters (start
point, end point, and depth point) are stored in a series of arrays.

5.4 CURSOR CONTROL

Once the hand gestures have been identified, mapping completely separate hand gestures to specific mouse
operations will be a breeze.

11
6.MODEL

Fig-1.2 MATHEMATICAL MODEL

12
Fig-1.3 flowchart

13
Fig-1.4 basic block diagram of the system

14
We presented the block diagram and flow chart of Virtual Mouse Control Using Hand Gesture Recognition in
this section. We also provided a brief description of the system's operation and components.

Fig-1.5

CLASSIFICATION MODEL
The AI Virtual Mouse System uses a camera. The proposed AI virtual mouse system is based on
frames captured by a laptop or PC's webcam. The video capture object is created using the Python
computer vision library OpenCV, and the web camera begins capturing video. The web camera
captures images, which are then sent to the AI virtual system.
•The video is currently being recorded and processed. The AI virtual mouse system uses a webcam to
capture each frame until the program is terminated.

15
Fig-model graph of media pipe

16
• Detecting which finger is up and carrying out mouse functions. At this point, we use the tip Id of the
respective finger discovered using the MediaPipe and the respective co-ordinates of the up fingers, as
shown in Fig-3.3, to determine which finger is up, and then we perform the appropriate mouse
function.
• Mouse functions based on hand gestures and hand tip detection using computer vision.

7.SYSTEM SPECIFICATIONS

7.1 SOFTWARE SPECIFICATIONS

• pycharm
• OS: Windows 10.
7.2 HARDWARE SPECIFICATIONS

• Webcam
• Minimum 2gb Ram
• Intel core i5 processor

17
8.RESULT
8.1 PERFORMANCE ANALYSIS
A simplification used in this project, which was not found in any recognition methods researched,
is the use of a wrist band to remove several degrees of freedom. This enabled three new recognition
methods to be devised. The recognition frame rate achieved is comparable to most of the systems
in existence (after allowance for processor speed) but the number of different gestures recognized
and the recognition accuracy are amongst the best found. Following shows several of the existing
gesture recognition systems along with recognition statistics and method .

Mouse function performed Success Failure Accuracy(%)

Mouse movement 100 0 100%

Left button click 98 2 98%

Right button click 99 1 99%

18
 Scroll function 93 7 93%

Brightness control 95 5 95%

Volume control 96 4 96%

NO ACTION PERFORMED 100 0 100%

Result 681 19 97.28%

From Table , it can be seen that the proposed AI virtual mouse system had achieved an accuracy
of about 97%. From this 97% accuracy of the proposed AI virtual mouse system, we come to
know that the system has performed well. As seen in Table, the accuracy is low for “Scroll
function” as this is the hardest gesture for the computer to understand. The accuracy for scroll
function is low because the gesture used for performing the particular mouse function is harder.
Also, the accuracy is very good and high for all the other gestures. Compared to previous
approaches for virtual mouse, our model worked very well with 97% accuracy.
This is a method in which we hide the secret information inside the other text file, the main
motive in this process is to share our secret information with the help of another normal
message which doesn‟t get suspicious.

19
9 TECHNIQUES USED
For the purpose of detection of hand gestures and hand tracking, the MediaPipe framework is used,
and OpenCV library is used for computer vision. The algorithm makes use of the machine learning
concepts to track and
recognize the hand gestures and hand tip.
MediaPipe is a framework which is used for applying in a machine learning pipeline, and it is an
opensource framework of Google. The MediaPipe framework is useful for cross platform
development since the framework is built using the time series data. The MediaPipe framework is
multimodal, where this framework can be applied to various audios and videos. The MediaPipe
framework is used by the developer for building and analyzing the systems through graphs, and it also
been used for developing the systems for the application purpose. The steps involved in the system
that uses MediaPipe are carried out in the pipeline configuration. The pipeline created can run in
various platforms allowing scalability in mobile and desktops. The MediaPipe framework is based on
three fundamental parts; they are performance evaluation, framework for retrieving sensor data, and
a collection of components which are called calculators , and they are reusable. A pipeline is a graph
which consists of components called calculators, where each calculator is connected by streams in
which the packets of data flow through. Developers are able to replace or define custom calculators
anywhere in the graph creating their own application. The calculators and streams combined create a
data-flow diagram; the graph is created with MediaPipe where each node is a calculator and the nodes
are connected by streams.

Single-shot detector model is used for detecting and recognizing a hand or palm in real time. The
single-shot detector model is used by the MediaPipe. First, in the hand detection module, it is first
trained for a palm detection model because it is easier to train palms. Furthermore, the non maximum
suppression works significantly better on small objects such as palms or fists . A model of hand
landmark consists of locating joint or knuckle co-ordinates in the hand region,
OpenCV is a computer vision library which contains image-processing algorithms for object
detection. OpenCV is a library of python programming language, and real-time computer vision
applications can be developed by using the computer vision library. The OpenCV library is used in
image and video processing and also analysis such as face detection and object detection .

20
OpenCV is an open-source software library for computer vision and machine learning. The
OpenCV full form is Open Source Computer Vision Library. It was created to provide a shared
infrastructure for applications for computer vision and to speed up the use of machine
perception in consumer products. OpenCV, as a BSD-licensed software, makes it simple for
companies to use and change the code. There are some predefined packages and libraries that
make our life simple and OpenCV is one of them.
PyAutoGUI lets your Python scripts control the mouse and keyboard to automate interactions
with other applications. The API is designed to be simple. PyAutoGUI works on Windows,
macOS, and Linux, and runs on Python 2 and 3
PyAutoGUI has several features:

Moving the mouse and clicking in the windows of other applications.

Sending keystrokes to applications (for example, to fill out forms).

Take screenshots, and given an image (for example, of a button or checkbox), and find it on the screen.

Locate an application’s window, and move, resize, maximize, minimize, or close it (Windows-
only,currently).

Display alert and message boxes.

21
10 TESTING

10.1 TEST CASE

Test Scenario Boundary Expected Actual Status

case Value Result Result
id
1 Used in >90% In normal Hand Passed
normal environment gestures got
environment. hand easily
gestures can recognized
be and work
recognized properly.
easily.
2 Used in >60% In brighter In bright Passed
bright environment, conditions the
environment. software software
should work works very
fine as it well.
easily
detects the
hand
movements
but in a more
brighter
conditions it
may not
detect the
hand
gestures as
expected.

3 Used in dark <30% In dark In dark Failed

environment environment, environment
It should software
work didn’t work
properly. properly in
detecting
hand
gestures.

22
4 Used at a >80% At this It works fine Passed
near distance, this and all
distance software features
(15cm) from should works
the web cam. perform properly.
perfectly.
5 Used at a far >95% At this At this Passed
distance distance, this distance, it is
(35cm) from software working
the web cam. should work properly.
fine.
6 Used at a >60% At this At this Passed
farther distance, distance,
distance their will be The functions
(60cm) from some of this
the web cam. problem in software
detecting works
hand properly.
gestures but
it should
work fine.

11 SOURCE CODE

# Imports

import cv2
import mediapipe as mp
import pyautogui
import math
from enum import IntEnum
from ctypes import cast, POINTER
from comtypes import CLSCTX_ALL
from pycaw.pycaw import AudioUtilities, IAudioEndpointVolume
from google.protobuf.json_format import MessageToDict
import screen_brightness_control as sbcontrol

pyautogui.FAILSAFE = False
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands

# Gesture Encodings
class Gest(IntEnum):

23
 # Binary Encoded
FIST = 0
PINKY = 1
RING = 2
MID = 4
LAST3 = 7
INDEX = 8
FIRST2 = 12
LAST4 = 15
THUMB = 16
PALM = 31
# Extra Mappings
V_GEST = 33
TWO_FINGER_CLOSED = 34
PINCH_MAJOR = 35
PINCH_MINOR = 36

# Multi-handedness Labels
class HLabel(IntEnum):
MINOR = 0
MAJOR = 1

# Convert Mediapipe Landmarks to recognizable Gestures

class HandRecog:

def __init__(self, hand_label):

self.finger = 0
self.ori_gesture = Gest.PALM
self.prev_gesture = Gest.PALM
self.frame_count = 0
self.hand_result = None
self.hand_label = hand_label

def update_hand_result(self, hand_result):

self.hand_result = hand_result
def get_signed_dist(self, point):
sign = -1
if self.hand_result.landmark[point[0]].y < self.hand_result.landmark[point[1]].y:
sign = 1
dist = (self.hand_result.landmark[point[0]].x - self.hand_result.landmark[point[1]].x)**2
dist += (self.hand_result.landmark[point[0]].y - self.hand_result.landmark[point[1]].y)**2
dist = math.sqrt(dist)
return dist*sign

def get_dist(self, point):

dist = (self.hand_result.landmark[point[0]].x - self.hand_result.landmark[point[1]].x)**2
dist += (self.hand_result.landmark[point[0]].y - self.hand_result.landmark[point[1]].y)**2
dist = math.sqrt(dist)
return dist

def get_dz(self,point):

24
 return abs(self.hand_result.landmark[point[0]].z - self.hand_result.landmark[point[1]].z)

# Function to find Gesture Encoding using current finger_state.

# Finger_state: 1 if finger is open, else 0
def set_finger_state(self):
if self.hand_result == None:
return

points = [[8,5,0],[12,9,0],[16,13,0],[20,17,0]]
self.finger = 0
self.finger = self.finger | 0 #thumb
for idx,point in enumerate(points):

dist = self.get_signed_dist(point[:2])
dist2 = self.get_signed_dist(point[1:])
try:
ratio = round(dist/dist2,1)
except:
ratio = round(dist1/0.01,1)

self.finger = self.finger << 1

if ratio > 0.5 :
self.finger = self.finger | 1

# Handling Fluctations due to noise

def get_gesture(self):
if self.hand_result == None:
return Gest.PALM

current_gesture = Gest.PALM
if self.finger in [Gest.LAST3,Gest.LAST4] and self.get_dist([8,4]) < 0.05:
if self.hand_label == HLabel.MINOR :
current_gesture = Gest.PINCH_MINOR
else:
current_gesture = Gest.PINCH_MAJOR

elif Gest.FIRST2 == self.finger :

point = [[8,12],[5,9]]
dist1 = self.get_dist(point[0])
dist2 = self.get_dist(point[1])
ratio = dist1/dist2
if ratio > 1.7:
current_gesture = Gest.V_GEST
else:
if self.get_dz([8,12]) < 0.1:
current_gesture = Gest.TWO_FINGER_CLOSED
else:
current_gesture = Gest.MID
else:
current_gesture = self.finger

25
 if current_gesture == self.prev_gesture:
self.frame_count += 1
else:
self.frame_count = 0

self.prev_gesture = current_gesture

if self.frame_count > 4 :
self.ori_gesture = current_gesture
return self.ori_gesture

# Executes commands according to detected gestures

class Controller:
tx_old = 0
ty_old = 0
trial = True
flag = False
grabflag = False
pinchmajorflag = False
pinchminorflag = False
pinchstartxcoord = None
pinchstartycoord = None
pinchdirectionflag = None
prevpinchlv = 0
pinchlv = 0
framecount = 0
prev_hand = None
pinch_threshold = 0.3

def getpinchylv(hand_result):
dist = round((Controller.pinchstartycoord - hand_result.landmark[8].y)*10,1)
return dist

def getpinchxlv(hand_result):
dist = round((hand_result.landmark[8].x - Controller.pinchstartxcoord)*10,1)
return dist
def changesystembrightness():
currentBrightnessLv = sbcontrol.get_brightness()/100.0
currentBrightnessLv += Controller.pinchlv/50.0
if currentBrightnessLv > 1.0:
currentBrightnessLv = 1.0
elif currentBrightnessLv < 0.0:
currentBrightnessLv = 0.0
sbcontrol.fade_brightness(int(100*currentBrightnessLv) , start = sbcontrol.get_brightness())

def changesystemvolume():
devices = AudioUtilities.GetSpeakers()
interface = devices.Activate(IAudioEndpointVolume._iid_, CLSCTX_ALL, None)
volume = cast(interface, POINTER(IAudioEndpointVolume))
currentVolumeLv = volume.GetMasterVolumeLevelScalar()

26
 currentVolumeLv += Controller.pinchlv/50.0
if currentVolumeLv > 1.0:
currentVolumeLv = 1.0
elif currentVolumeLv < 0.0:
currentVolumeLv = 0.0
volume.SetMasterVolumeLevelScalar(currentVolumeLv, None)

def scrollVertical():
pyautogui.scroll(120 if Controller.pinchlv>0.0 else -120)

def scrollHorizontal():
pyautogui.keyDown('shift')
pyautogui.keyDown('ctrl')
pyautogui.scroll(-120 if Controller.pinchlv>0.0 else 120)
pyautogui.keyUp('ctrl')
pyautogui.keyUp('shift')

# Locate Hand to get Cursor Position

# Stabilize cursor by Dampening
def get_position(hand_result):
point = 9
position = [hand_result.landmark[point].x ,hand_result.landmark[point].y]
sx,sy = pyautogui.size()
x_old,y_old = pyautogui.position()
x = int(position[0]*sx)
y = int(position[1]*sy)
if Controller.prev_hand is None:
Controller.prev_hand = x,y
delta_x = x - Controller.prev_hand[0]
delta_y = y - Controller.prev_hand[1]

distsq = delta_x**2 + delta_y**2

ratio = 1
Controller.prev_hand = [x,y]

if distsq <= 25:

ratio = 0
elif distsq <= 900:
ratio = 0.07 * (distsq ** (1/2))
else:
ratio = 2.1
x , y = x_old + delta_x*ratio , y_old + delta_y*ratio
return (x,y)

def pinch_control_init(hand_result):
Controller.pinchstartxcoord = hand_result.landmark[8].x
Controller.pinchstartycoord = hand_result.landmark[8].y
Controller.pinchlv = 0
Controller.prevpinchlv = 0
Controller.framecount = 0

27
 # Hold final position for 5 frames to change status
def pinch_control(hand_result, controlHorizontal, controlVertical):
if Controller.framecount == 5:
Controller.framecount = 0
Controller.pinchlv = Controller.prevpinchlv

if Controller.pinchdirectionflag == True:
controlHorizontal() #x

elif Controller.pinchdirectionflag == False:

controlVertical() #y

lvx = Controller.getpinchxlv(hand_result)
lvy = Controller.getpinchylv(hand_result)

if abs(lvy) > abs(lvx) and abs(lvy) > Controller.pinch_threshold:

Controller.pinchdirectionflag = False
if abs(Controller.prevpinchlv - lvy) < Controller.pinch_threshold:
Controller.framecount += 1
else:
Controller.prevpinchlv = lvy
Controller.framecount = 0

elif abs(lvx) > Controller.pinch_threshold:

Controller.pinchdirectionflag = True
if abs(Controller.prevpinchlv - lvx) < Controller.pinch_threshold:
Controller.framecount += 1
else:
Controller.prevpinchlv = lvx
Controller.framecount = 0

def handle_controls(gesture, hand_result):

x,y = None,None
if gesture != Gest.PALM :
x,y = Controller.get_position(hand_result)

# flag reset
if gesture != Gest.FIST and Controller.grabflag:
Controller.grabflag = False
pyautogui.mouseUp(button = "left")

if gesture != Gest.PINCH_MAJOR and Controller.pinchmajorflag:

Controller.pinchmajorflag = False

if gesture != Gest.PINCH_MINOR and Controller.pinchminorflag:

Controller.pinchminorflag = False

# implementation
if gesture == Gest.V_GEST:
Controller.flag = True

28
 pyautogui.moveTo(x, y, duration = 0.1)

elif gesture == Gest.FIST:

if not Controller.grabflag :
Controller.grabflag = True
pyautogui.mouseDown(button = "left")
pyautogui.moveTo(x, y, duration = 0.1)

elif gesture == Gest.MID and Controller.flag:

pyautogui.click()
Controller.flag = False

elif gesture == Gest.INDEX and Controller.flag:

pyautogui.click(button='right')
Controller.flag = False

elif gesture == Gest.TWO_FINGER_CLOSED and Controller.flag:

pyautogui.doubleClick()
Controller.flag = False

elif gesture == Gest.PINCH_MINOR:

if Controller.pinchminorflag == False:
Controller.pinch_control_init(hand_result)
Controller.pinchminorflag = True
Controller.pinch_control(hand_result,Controller.scrollHorizontal, Controller.scrollVertical)

elif gesture == Gest.PINCH_MAJOR:

if Controller.pinchmajorflag == False:
Controller.pinch_control_init(hand_result)
Controller.pinchmajorflag = True
Controller.pinch_control(hand_result,Controller.changesystembrightness,
Controller.changesystemvolume)

'''
---------------------------------------- Main Class ----------------------------------------
Entry point of Gesture Controller
'''

class GestureController:
gc_mode = 0
cap = None
CAM_HEIGHT = None
CAM_WIDTH = None
hr_major = None # Right Hand by default
hr_minor = None # Left hand by default
dom_hand = True

def __init__(self):
GestureController.gc_mode = 1
GestureController.cap = cv2.VideoCapture(0)

29
 GestureController.CAM_HEIGHT = GestureController.cap.get(cv2.CAP_PROP_FRAME_HEIGHT)
GestureController.CAM_WIDTH = GestureController.cap.get(cv2.CAP_PROP_FRAME_WIDTH)

def classify_hands(results):
left , right = None,None
try:
handedness_dict = MessageToDict(results.multi_handedness[0])
if handedness_dict['classification'][0]['label'] == 'Right':
right = results.multi_hand_landmarks[0]
else :
left = results.multi_hand_landmarks[0]
except:
pass

try:
handedness_dict = MessageToDict(results.multi_handedness[1])
if handedness_dict['classification'][0]['label'] == 'Right':
right = results.multi_hand_landmarks[1]
else :
left = results.multi_hand_landmarks[1]
except:
pass

if GestureController.dom_hand == True:
GestureController.hr_major = right
GestureController.hr_minor = left
else :
GestureController.hr_major = left
GestureController.hr_minor = right

def start(self):

handmajor = HandRecog(HLabel.MAJOR)
handminor = HandRecog(HLabel.MINOR)

with mp_hands.Hands(max_num_hands = 2,min_detection_confidence=0.5,

min_tracking_confidence=0.5) as hands:
while GestureController.cap.isOpened() and GestureController.gc_mode:
success, image = GestureController.cap.read()

if not success:
print("Ignoring empty camera frame.")
continue

image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)

image.flags.writeable = False
results = hands.process(image)

image.flags.writeable = True
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)

30
 if results.multi_hand_landmarks:
GestureController.classify_hands(results)
handmajor.update_hand_result(GestureController.hr_major)
handminor.update_hand_result(GestureController.hr_minor)

handmajor.set_finger_state()
handminor.set_finger_state()
gest_name = handminor.get_gesture()

if gest_name == Gest.PINCH_MINOR:
Controller.handle_controls(gest_name, handminor.hand_result)
else:
gest_name = handmajor.get_gesture()
Controller.handle_controls(gest_name, handmajor.hand_result)

for hand_landmarks in results.multi_hand_landmarks:

mp_drawing.draw_landmarks(image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
else:
Controller.prev_hand = None
cv2.imshow('Gesture Controller', image)
if cv2.waitKey(5) & 0xFF == 13:
break
GestureController.cap.release()
cv2.destroyAllWindows()

# uncomment to run directly

gc1 = GestureController()
gc1.start()

SAMPLE CODE
import cv2
import mediapipe as mp
import pyautogui
cap = cv2.VideoCapture(0)
hand_detector = mp.solutions.hands.Hands()
drawing_utils = mp.solutions.drawing_utils
screen_width, screen_height = pyautogui.size()
index_y = 0
while True:
_, frame = cap.read()
frame = cv2.flip(frame, 1)
frame_height, frame_width, _ = frame.shape
rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
output = hand_detector.process(rgb_frame)
hands = output.multi_hand_landmarks

31
 if hands:
for hand in hands:
drawing_utils.draw_landmarks(frame, hand)
landmarks = hand.landmark
for id, landmark in enumerate(landmarks):
x = int(landmark.x*frame_width)
y = int(landmark.y*frame_height)
if id == 8:
cv2.circle(img=frame, center=(x,y), radius=10, color=(0, 255, 255))
index_x = screen_width/frame_width*x
index_y = screen_height/frame_height*y

if id == 4:
cv2.circle(img=frame, center=(x,y), radius=10, color=(0, 255, 255))
thumb_x = screen_width/frame_width*x
thumb_y = screen_height/frame_height*y
print('outside', abs(index_y - thumb_y))
if abs(index_y - thumb_y) < 20:
pyautogui.click()
pyautogui.sleep(1)
elif abs(index_y - thumb_y) < 100:
pyautogui.moveTo(index_x, index_y)
cv2.imshow('Virtual Mouse', frame)
cv2.waitKey(1)

12.OUTPUT IMAGES

32
33
34
35
36
37
13.FUTURE SCOPE

The primary goal of the AI virtual mouse system is to control mouse cursor functions with hand gestures rather
than a physical mouse. The proposed system can be realized by using a webcam or an in-built camera that
detects hand gestures and hand tips and processes these frames to perform the specific mouse functions.
Based on the model's results, conclude that the proposed AI virtual mouse system performed well and
has higher accuracy than existing models, and that the model overcomes most of the limitations of
existing systems. The AI virtual mouse can be used for real-world applications because the proposed
model is more accurate and can be used virtually using hand gestures rather than the traditional physical
mouse.

> It reduces the workspace and burden of extra hardware devices

> It removes the burden ofdevices, it brings the user and the workspace morecloser.

14.CONCLUSIONS
Gesture recognition gives the best interaction between human and machine. Gesture recognition is also
important for developing alternative human computer interaction modalities. It enables human to interface with
machine in a more natural way. Gesture recognition can be used for many applications like sign language
recognition for deaf and dumb people, robot control etc.
This technology has wide applications in the fields of augmented reality, computer graphics, computer gaming,
prosthetics, and biomedical instrumentation. Digital Canvas is an extension of our system which is gaining
popularity among artists, by which the artist could create 2D or 3D images using the Virtual Mouse technology
using the hand as brush and a Virtual Reality kit or a monitor as display set. This technology can be used to
help patients who don’t have control of their limbs. In case of computer graphics and gaming this technology
has been applied in modern gaming consoles to create interactive games where a person’s motions are tracked
and interpreted as commands. The major extension to this work can be done to make system able to work at
much complex background and compatible with different light conditions. It can be made as an effective user
interface and which can include all mouse functionalities. And also, it would be ideal to research into advanced
mathematical materials for image processing and investigate on different hardware solutions that would result
in more accurate hand detections. Not only did this project show the different gesture operations that could be
done by the users but it also demonstrated the potential in simplifying user interactions with personal computers
and hardware systems.

This project will do away with the requirement for a mouse or any other physical device to control cursor
movement. For the implementation of our proposed task, we will use OpenCV for object detection. The mouse
can be moved with a great level of precision and accuracy. Better Human-Computer Interaction(HCI) is
achieved. It is also useful in Augmented Reality, current gaming, and computer graphics. In the context of
computer graphics and gaming, this technology has been implemented in modern gaming consoles to produce

38
interactive games that detect and interpret a person's movements as orders. The major goal is to eliminate the
usage of any physical device, such as a mouse, and instead rely on a webcam, which is readily available with
the laptop. Although this initiative offers numerous benefits, it also has significant disadvantages. If the
background picture clashes with the given image, it may offer an incorrect result and may not perform properly.
As a result, it's best to use this technique when the background light and the colour of the object don't combine.
On PCs with poor resolution and computational capability, the system may run slower. If the camera has a high
resolution, the system may be slow, but this problem can be rectified by lowering the image resolution.

15.REFERENCE
1) OpenCV Website – www.opencv.org

2) MSDN Microsoft developers network – www.msdn.microsoft.com

3) Code project – www.codeproject.com/Articles/498193/Mouse-Control-via-Webcam

4) Aniket Tatipamula’s Blog - http://anikettatipamula.blogspot.in/2012/02/hand-gesture-using-opencv.html

5) Microsoft Research Paper- http://research.microsoft.com/en-us/um/people/awf/bmvc02/project.pdf

[1]Computer vision based mouse, A. Erdem, E. Yardimci, Y. Atalay, V. Cetin, A. E. Acoustics, Speech, and
Signal Processing, 2002. Proceedings. (ICASS). IEEE International Conference.
[2]International Journal of Computer Trends and Technology (IJCTT) – volume 9 number 1– Mar 2014
[3]Virtual mouse vision based interface, Robertson P., Laddaga R., Van Kleek M. 2004.
[4]Vision based Men-Machine Interaction http://www.ceng.metu.edu.tr/~vbi/
[5]Chu-Feng Lien, Portable Vision-Based HCI - A Real-time Hand Mouse System on Handheld Devices.

Abhik Banerjee, Abhirup Ghosh, Koustuvmoni Bharadwaj,” Mouse Control using a Web Camera based on Color
Detection”,IJCTT,vol.9, Mar 2014.

[2] Angel, Neethu.P.S,”Real Time Static & Dynamic Hand Gesture Recognition”, International Journal of Scientific &
Engineering Research Volume 4, Issue3, March-2013.

[3] Q. Y. Zhang, F. Chen and X. W. Liu, “Hand Gesture Detection and Segmentation Based on Difference Back-
ground Image with Complex Background,” Proceedings of the 2008 International Conference on Embedded Soft-
ware and Systems, Sichuan, 29-31 July 2008, pp. 338-343.

39