ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
VIRTUAL MOUSE USING HAND GESTURES
1Rohini Shirkar, 2Pranjali Salunke, 3Paritosh Singh, 4Sachin Shirke,
5Prof. Jagat Gaydhane
Department of Information Technology,
Datta Meghe College of Engineering
Airoli, India
Abstract —Hand Gesture Recognition plays a key role in human-computer interactions. As we can see that there are so
many new Technological advancements happening such as biometric authentication which we can see frequently in our
smart phones, similarly hand gesture recognition is a modern way of human-computer interaction i.e., we can control our
system by showing our hands in front of webcam and hand gesture recognition can be useful for all kinds of people. Based
upon this idea this paper is presented. With new changes seen in computer technology day by day, it has become quite
essential for us to find specific new ways of interaction with computer systems as its need is increasing in society every day.
Today, every device is making the use of touch screen technology on its systems, which isn't affordable to be used in all
applications. A specific interactive module like a virtual mouse that makes use of Object Tracking and Gestures that will
help us to interact can be an alternative way for the traditional touch screen and the physical mouse. The objective is to
create an Object Tracking application that interacts with the system.
Index Terms -Object Tracking, Hand Gestures, Image capture, Computer Vision, Open CV, Deep Learning, Image
Processing.
I. INTRODUCTION
A Computer Mouse is an input device that helps to point
and to interact with whatever that is being pointed. There are so many types of mouse in the current trend, there’s the mechanical
mouse that consists of a single rubber ball which can rotate in any direction and the movement of the pointer is determined by the
motion of that rubber ball. Later the mechanical mouse is replaced by the Optical Mouse. Gestures can be in any form like hand
image or pixel image or any human given pose that require less computational difficulty or power for making the devices required
for the recognition to make work. Different techniques are being proposed by the companies for gaining necessary
information/data for recognition handmade gestures recognition models. Some models work with special devices such as data
glove devices and color caps to develop a complex information about gesture provided by the user/human. In a Similar way the
virtual mouse that we will discuss in this paper is made up of OpenCV, pyautogui. OpenCV is a python library for computer
vision which is used for capturing images from webcam. Pyautogui is a python library that is used for specifying keyboard and
mouse operations. Previously when we want to operate and work with system’s it’s been a really different thing. So once the
technology is being developed the things like separate applications and soft wares are created and developed for easy use of
system. When the technology had introduced technologies like artificial Intelligence and machine learning they have become
booming technologies that help us easily to operate and give commands to systems without any human interactions. So here the
model is also a part of these technologies which can help us work with our systems with ease. The techniques are developed in
such a way even machine / System have the intelligence that can identify and vary different thing which are easy for human now
became easy for systems to understand the same things with a new way of learning. Virtual Mouse using Hand gesture
recognition allows users to control mouse with the help of hand gestures and system’s webcam is used for tracking hand gestures.
Computer vision techniques are used for gesture recognition. OpenCV consists of a package called video capture which is used to
capture data from a live video.
II. Problem Description & Overview
The proposed virtual mouse system can be used to overcome problems in the real world such as situations
where there is no space to use a physical mouse and also for the persons who have problems in their hands and are not able to
control a physical mouse. To design a virtual mouse that detects hand gestures and performs mouse operations only using the
fingers. we used different combinations of fingers to perform various operations of the mouse according to which particular
combination of fingers is recognized. In the proposed system, users don't have to colour their fingers with a specific colour and
are not required to use any device or sensors.
Objective. The main objective of the proposed AI virtual
mouse system is to develop an alternative to the regular and traditional mouse system to perform and control the mouse functions,
and this can be achieved with the help of a web camera that captures the hand gestures and hand tip and then processes these
frames to perform the particular mouse function such as left click, right click, and scrolling function.
III. Algorithm Used for Hand Tracking
For the purpose of detection of hand gestures and hand
tracking, the MediaPipe framework is used, and OpenCV
library is used for computer vision. The algorithm makes use of the machine learning concepts to track and recognize the hand
gestures and hand tip.
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2686
ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
3.1. Media Pipe.
Media Pipe is a framework which is used for applying in a machine learning pipeline, and it is an opensource framework of
Google. The Media Pipe frame work is useful for cross platform development since the framework is built using the time series
data. The pipeline created can run in various platforms allowing scalability in mobile and desktops. The MediaPipe framework is
based on three fundamental parts; they are performance evaluation, framework for retrieving sensor data, and a collection of
components which are called calculators , and they are reusable. A pipeline is a graph which consists of components called
calculators, where each calculator is connected by streams in which the packets of data flow through. Developers are able to
replace or define custom calculators anywhere in the graph creating their own application.
3.2. OpenCV.
OpenCV is a computer vision library which contains image-processing algorithms for object detection . OpenCV is a library of
python programming language and real-time computer vision applications can be developed by using the computer vision library.
The OpenCV library is used in image and video processing and also analysis such as face detection and object detection.
IV. Methodology
The various functions and conditions used in the system are explained in the flow Diagram of the real-time AI virtual mouse
System
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2687
ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
V. Experimental Results and Evaluation
In the proposed AI virtual mouse system, the concept of advancing the human-computer interaction using computervision is given.
Commands to start the working:
Capturing video using the webcam the area of the computer screen where we can move the cursor.
Mouse cursor moving around the computer window.
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2688
ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
Gesture for the computer to perform left button click.
Gesture for the computer to perform right button click.
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2689
ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
Gesture for the computer to perform right button click and scroll down function.
5.1 Comparison between Existing and Proposed
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2690
ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
VI. FUTURE SCOPE
⚫ We can give more advanced virtual mouse including more facilities.
⚫ Implement some extra features from time to time for enhancing the software quality
The above-mentioned points are the enhancements that can be done to increase the applicability and usage of this project. We
have left all the options open so that if there is any other future requirement in the system by the user or students for the
enhancement of the application then it is possible to implement them. In the last, we would like to thank all the persons involved
in the development of the application directly or indirectly. We hope that the project will serve the purpose for which it is
developed thereby underlining the
success of the Process.
VII. CONCLUSION
The proposed system can be achieved by using a webcam or a built-in camera which detects the hand gestures and hand tip and
processes these frames to perform the particular mouse functions. From the results of the model, we can come to a con
clusion that the proposed AI virtual mouse system has
performed very well and has a greater accuracy compared to the existing models and also the model overcomes most of the
limitations of the existing systems. Since the proposed model has greater accuracy, the AI virtual mouse can be used for real-
world applications, and also, it can be used to reduce
the spread of COVID-19, since the proposed mouse system can be used virtually using hand gestures without using the traditional
physical mouse.
VIII. REFERENCES:
1. Vantukal Reddy, Thumma Dhyanchand , Galla Vamsi Krishna,Satish Maheshwaram “Virtual Mouse Control Using Colored
Fingertips and Hand Gesture Recognition”,11-12 Sep 2020 https://ieeexplore.ieee.org/document/9242677
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2691
ISSN: 2455-2631 April 2023 IJSDR | Volume 8 Issue 4
2. Kollipara Sai Varun. I Puneeth, Dr.T.Prem Jacobi”Virtual Mouse Implementation Using Opencv “,2019
https://ieeexplore.ieee.org/document/8862764
3. Sherin Mohammed, V H preetha “Hand Gesture - Virtual Mouse For Human Computer Interaction ”,2018
https://ieeexplore.ieee.org/abstract/document/8748401/authors
4. Ashish Mhetar.B K Srioop,Kavya AGS,Ramanath Nayak,Ravikumar Javali,Suma K V “Virtual Mouse” 12th March 2015
5. Monali Shetty,Christina Daniel,Manthan Bhatkar,Ofrin Lopes “Virtual Mouse Using Object Tracking” 10 th july 2020
https://ieeexplore.ieee.org/document/9137854
6. Kabid Shibly,Samrat Dey,Aminul islam, Shahriar Showrav”Design and Development of Hand Gesture based Virtual
Mouse”,19 Dec 2019 https://ieeexplore.ieee.org/document/8934612
7. J. Katona, “A review of human–computer interaction and virtual reality research fields in cognitive InfoCommunications,”
Applied Sciences, vol. 11, no. 6, p. 2646, 2021.
8. D.-H. Liou, D. Lee, and C.-C. Hsieh, “A real time hand gesture recognition system using motion history image,” in
Proceedings of the 2010 2nd International Conference on Signal Processing Systems, July 2010.
9. K. P. Vinay, “Cursor control using hand gestures,” International Journal of Critical Accounting, vol. 0975–8887, 2016.
10. L. Thomas, “Virtual mouse using hand gesture,” International Research Journal of Engineering and Technology IRJET, vol.
5, no. 4, 2018.
11. P. Nandhini, J. Jaya, and J. George, “Computer vision system for food quality evaluation—a review,” in Proceedings of the
2013 International Conference on Current Trends in Engineering and Technology (ICCTET), pp. 85–87, Coimbatore, India,
July 2013.
12. J. Jaya and K. anushkodi, “Implementation of certain system for medical image diagnosis,” European Journal of Scientifific
Research, vol. 53, no. 4, pp. 561–567, 2011.
13. P. Nandhini and J. Jaya, “Image segmentation for food quality evaluation using computer vision system,” International
Journal of Engineering Research and Applications, vol. 4, no. 2, pp. 1–3, 2014.
14. J. T. Camillo Lugaresi, “MediaPipe: A Framework for Building Perception Pipelines,” 2019,
https://arxiv.org/abs/1906.08172.
15. V. Bazarevsky and G. R. Fan Zhang. On-Device, MediaPipe for Real-Time Hand Tracking.
IJSDR2304417 International Journal of Scientific Development and Research (IJSDR) www.ijsdr.org 2692
