Game Controlling Using Hand Gestures

M Vinod Adharsh Katraj Nikhil Marturi

Assistant Professor, Electronics and Electronics and Communication Electronics and Communication
Communication Engineering Engineering Engineering
MLR Institute Of Technology MLR Institute Of Technology MLR Institute Of Technology
Hyderabad, India Hyderabad, India Hyderabad, India
vinod.m@mlrinstitutions.ac.in adharshkatraj114@gmail.com nikhilmarturi9866@gmail.com

Sreeja Paluvatla Savaram. Ch V S Ruthvik Dr S.V.S Prasad HOD

Electronics and Communication Electronics and Communication Electronics and Communication
Engineering Engineering Engineering
MLR Institute Of Technology MLR Institute Of Technology MLR Institute Of Technology
Hyderabad, India Hyderabad, India Hyderabad, India
sreejareddy1717@gmail.com rnruthvik@gmail.com hodece@mlrinstitutions.ac.in

Abstract— The main goal of this system is to demonstrate game sales, such as the Nintendo Wii, which sold over 50
how to play a computer game using human gestures. The million systems in its first year. Hand gestures create a Natural
system's secondary goal is to develop a system that allows a User Interface and are particularly intuitive and effective for
player to play a game without using a physical controller. This one-to-one engagement with computers (NUI).
system seeks to create a gesture recognition application. The
system's attached camera or webcam can be used to identify Novel gadgets and approaches for cursor control via hand
human hand gestures. The operations on the Game will be gestures have been extensively researched. Hand gesture
performed with the game's default gaming controls based on the recognition is employed in sign language identification in
system's study of detecting human Hand Gestures. A collection addition to HCI, making hand gesture recognition even more
of instructions is included in this system for identifying human important.
hand movements. The palms of the hands should be used to
make the movements. User interface, gesture recognition, and A. Problem Statement
analysis will comprise the three aspects of the system. The user Hand gestures may be used to interact with systems in a
interface module gives the user all of the necessary graphical variety of ways, including video games, operating UAVs, and
user interface to register the user's arm positions for gestures.
controlling medical equipment. Handicapped persons can also
The motions recognition module may be used to recognise
use these hand gestures to engage with technology.
gestures. Finally, depending on the estimated analysis, the
analysis module will evaluate the human hand gesture and Traditional interface technologies such as the keyboard,
perform game controls. Fully reliable hand gesture recognition mouse, and touchscreen may constrain how we utilise the
systems are currently being explored and developed. This system. In order to connect with these technologies, they all
system effort can serve as a first step in laying a foundation that require physical contact. Without physically engaging with
can be expanded in the future. the interface devices, gestures can interpret the same
functionality. The difficulty is in comprehending these
Keywords— OpenCV, Mediapipe, VSC, Hand gestures, movements, as the same motion may appear differently to
graphs, Framework. various persons when executing the same action. The adoption
of Deep Learning methods may be able to solve this problem.
I. INTRODUCTION Convolutional neural networks (CNNs) are proven to be the
Over the last decade, computer technology has advanced best tool for processing these types of recognition systems.
dramatically and has become an indispensable element of The main drawback of deep learning methods is that they may
daily life. For Human Computer Interaction, the keyboard is perform badly in real-world recognition. Processing motions
the most crucial computer accessory (HCI). In some real-life necessitates a lot of computational power.
scenarios, such as Human Robot Interaction, the keyboard is II. LITERATURE SURVEY
unsuitable for HCI. Hand gestures are the most natural and
intuitive HCI approach that can be used as a feasible There have been several and diverse approaches to vehicle
alternative for the computer keyboard. As a result, the goal of tracking, monitoring, and warning systems proposed thus far.
this system is to investigate and design a Computer Control [1] Gupta, Bhumika (2017) proposed object detection is a
(CC) system that uses hand gestures. A camera may be used well-known computer technique that focuses on finding
for vision-based CC to fully use its capabilities, practically items or instances of a specific class (such as individuals,
eliminating the need for a computer keyboard. Other HCI flowers, and animals) in digital photos and videos. Face
applications, such as a sign language database or a motion identification, character recognition, and vehicle
controller, can substantially benefit from the usage of a calculation are just a few examples of well-studied object
camera. detection applications. Object detection has a wide range
HCI technology for gaming have advanced significantly in of applications, including retrieval and surveillance. This
recent decades, with examples being the Microsoft Kinect and research describes many basic principles used in object
Nintendo Wii. detection while utilising the OpenCV library of Python 2.7
to increase the efficiency and accuracy of object
These gaming technologies make playing video games identification.
more natural and participatory. Motion controllers are the
future of gaming, and they have significantly increased video

1
[2] Kartik Umesh Sharma (2017) suggested an object This output is passed to feature extraction for further
identification system that detects real-world items in a processing. This is where the hand contour and intricate
digital image or video, where the object might be from any moment take place. The feature vector is the result of this
class of objects, such as persons, automobiles, or other stage.
vehicles. To detect an item in an image or video, the
system requires a few components, including a model The final phase is utilising artificial neural networks to
classify the data. Next, hand counter-based artificial neural
database, a feature detector, a hypothesis, and a hypothesis
verification. This article provides an overview of the many networks and complicated moments-based artificial neural
networks are evaluated and compared. The preceding
strategies for detecting, localizing, categorizing, extracting
features, appearance information, and other tasks in photos paragraphs outline the complete procedure involved in this
project.
and videos.
[3] Occlusions, appearance alterations, egomotion, and This project may be used to play any game that requires
the program's coded keys. Aside from that, you may use the
lighting changes are all common effects of object motion
and scene size fluctuations, so Mukesh Tiwari (2017) coded keys to manage your system.
proposed object recognition and tracking as one of the The strategy utilised to construct our hand gesture
essential topics of research. 'Hand Gestures for Game recognition system is shown in Figure 1..
Control 6' Feature selection is very important in object
tracking. It's used in a variety of real-time applications,
including as vehicle perception and video surveillance. To
circumvent the problem of detection, monitoring of object
movement and appearance is used. The tracking algorithm
is at the heart of the algorithm, which smooths out the
video sequence. Forth the other hand, only a few
approaches make advantage of previously collected data
such as item shape, color, texture, and so on.
[4] Abdul Muhsin M (2019) stated that everyone ought to live
independently, including those who are disabled. In recent
decades, technology has focused on disabled people in
order to offer them as much control over their lives as
possible. In this paper, an assistive system for the blind is
proposed, which uses YOLO for fast identifying objects
inside photos and video streams utilizing deep neural
networks for reliable detection, and OpenCV under Python
on the Raspberry Pi3. The results revealed that employing
a user-friendly device based on a person and object
recognition model, the recommended model was
successful in allowing blind persons to move around in
new indoor-outdoor situations.
III. METHODOLOGY
The basic idea and working behind our project Game
Controlling using hand gestures is explained below.
The program is first created in Visual Studio code. When
the code is compiled, you can see the live video streaming
going on with the help of OpenCV. When you show your
Figure-1 An overview of the methodology utilised to create
hand, you can see a graph projector on your palm on this
our hand gesture recognition system.
screen. This is done with the help of mediapipe. Once you see
the projections on your palm, you can start playing the game C. Working
A. Technologies and platforms To begin, import all relevant libraries; in our instance, only
three are required. Then, for detecting and painting landmarks
 OpenCV on the picture, establish the Holistic model and drawing utils.
 Mediapipe The picture will be analyzed for facial and hand landmarks.
 Python
The holistic model analyses the image and creates
 Visual Studio code landmarks for the Face, Left Hand, and Right Hand, as well as
detecting the hand movement's Pose. Now, using OpenCV,
B. Overview capture frames from the camera on a regular basis. By
The picture is captured first, and then it is checked to see modifying the first line of the main function to "cap =
if it requires scaling, rotation, or translation. If everything is cv2.VideoCapture(0)," we can capture a live stream from the
as expected, it goes to the next step that is image processing. camera.
Smoothing and filtering actions are done at this stage. The
result is a picture of a hand area. Transform the BGR picture to RGB and use an initialised
holistic model to make predictions. The landmarks may be
accessed using results.face landmarks, results.right hand

2
landmarks, and results.left hand landmarks, respectively, from If motion is detected, it evaluates the time interval and
the predictions generated by the holistic model. Using the grabs and senses the current frames if the time interval of the
draw landmarks function from drawing utils, draw the uploaded frame is larger than three seconds.
discovered landmarks on the picture. Use the generated Otherwise, it will continue to check for movement. The
picture as a posture movement based on the image window's programme ends if we stop running it at any point throughout
axis drawing. its execution.
D. Sequence Diagram
Abbreviations
A sequence diagram is a graphical representation of a
situation that depicts item interaction in a time-based order:  OpenCV – Open-source computer vision library
what occurs first, then what happens next.  VSC – Visual Studio code
Sequence diagrams define the function of objects and aid
 HCI – Human Computer Interaction
in the determination of class responsibilities and interfaces.
Because they are basic and easy to understand, this style  HRI – Human Robot Interaction
of diagram is best employed at the early analysis phase of  ANN – Artificial Neural Networks
design. Use cases are frequently related with sequence  CC – Computer Control
diagrams.
 NUI – Natural User Interface

IV. RESULTS
To begin, open Visual Studio Code and write the
necessary code. When you're through with the code, execute
'python main.py' from the current directory's terminal. We
will be able to see a window which shows the camera feed.
Once we are able to see the window, it’s the indication
that we will be able to play the game. Open the game which
you want to play and begin playing with the help of hand
gestures.

A. Hill Climb Game

Here are few of the results when we attempted to play Hill
climb game.
Figure-2 Sequence Diagram

E. Dataflow Diagram

Figure-4 Hand gesture used while playing Hill climb game for
gas
In the Hill climb game, we basically use two keys, i.e.,
gas and brake in order to move or stop the vehicle.
For gas, the player needs to open his palm and for brake,
Figure-3 Dataflow Diagram the player needs to close his palm.
In figure-4, we can see the hand gesture used for gas while
The application is initialized after we start it, and live playing the Hill climb game. When the palm is open, it
video streaming begins. indicates that the player needs to use gas.
If motion detection is detected, the software verifies or
moves on to the next condition; otherwise, video streaming Similarly, if the player needs to use brake, he has to close
continues. his palm. It can be seen the figure below i.e., figure-5.

3
 Figure-5 Hand gesture used while playing Hill climb game for Figure-7 Hand gesture used to slide while playing Subway
brake surfers game
To slide, we need to show any four fingers of the hand. In
B. Subway Surfers figure-7, we can see the hand gestures used to slide while
Here are few of the results when we attempted to play playing the game.
Subway surfers game. Subway surfers has four 3. Use the left and right arrows to change lanes
functionalities.
To dodge trains, barriers, and other obstructions, change
They are jump, slide, left and right. lanes.
1. Jump
This allows you to leap over barriers and gather coins
while flying.
Unless you have the Super Sneakers powerup, you won't
be able to leap high enough to get on top of a train.
To jump, we need to show any two fingers of our hand.
Figure-6 shows the hand gesture used to jump.

Figure-8 Hand gesture used to switch to left lane while playing

Subway surfers game
To switch between lanes, i.e., to switch to left and right
lanes, we need to show opened and closed palm respectively.
In the figure above, we can see the hand gesture used to
switch to the left lane while playing the game. Figure-9 shows
the hand gesture used to switch to right lane while playing the
subway surfers game
Figure-6 Hand gesture used to jump while playing Subway
surfers game
2. Slide
In the subway surfers game, you'll have to roll to go
beneath low obstacles.

Figure-9. Hand gesture used to switch to right lane while

playing Subway surfers game

4
 V. FUTURE SCHOPE IX. REFERENCES
This application may be regarded a starting point for HCI [1] J. Levine, C. B. Congdon, M. Ebner, G. Kendall, S. M.
application development, hence it can benefit from Lucas, R. Miikkulainen, T. Schaul, and T. Thompson,
significant improvements. Using some additional HCI “General video game playing,” in press.
principles and Open CV Algorithms, this programme may be
[2] D. Perez, S. Samothrakis, and S. Lucas, “Knowledge-
developed to control the mouse cursor. With the use of Neural
based fast evolutionary MCTS for general video game
Networks-based logics, far higher precision may be achieved.
playing,” in IEEE Conference on Computational
Performance tracking may be enhanced to achieve better
Intelligence and Games (CIG’14), 2014, pp. 1–8.
outcomes. When the template matching hand gesture
identification approach is combined with a machine learning [3] D. Perez, S. Samothrakis, J. Togelius, T. Schaul, S. M.
classifier, the accuracy of hand gesture recognition may be Lucas, A. Couetoux, J. Lee, C.-U. Lim, and T.
enhanced. This will take a long time to develop, but the Thompson, “The 2014 general video game playing
precision of gesture detection will improve. competition,” IEEE Transactions on Computational
Intelligence and AI in Games, 2015.
VI. ADVANTAGES
[4] G. Malik, D. Nau, and P. Traverso, Automated Planning
 People who are physically handicapped (without fingers)
Theory and Practice. Elsevier, 2004.
can also play the game with the help images of the hand
gestures. [5] J. Hoffmann and B. Nebel, “The FF planning system:
 People who are paralysed or in hospitals can play games fast plan generation through heuristic search,” Journal
for fun. of Artificial Intelligence Research, vol. 14, pp. 253–
 Instead of using a keyboard or mouse, children prefer to 302, 2001.
play games with their hands and gestures.. [6] J. Hoffman, “FF: The Fast-Forward Planning System,”
 The biggest benefit of employing hand gestures is that AI Magazine, no. 1, pp. 57–62, 2001.
they allow you to communicate with the computer
without having to touch it. [7] S. Yoon, A. Fern, and R. Givan, “Learning Heuristic
Functions from Relaxed Plans.,” in Proceedings of the
VII. CONCLUSION International Conference on Automated Planning and
The Open CV package is used to construct the machine Scheduling (ICAPS’06), 2006.
vision-based keyboard cursor control utilising hand gesture [8] J.-H. Wu, R. Kalyanam, and R. Givan, “Stochastic
system in Python. When playing a game, the system may enforced hill- climbing,” Journal of Artificial
monitor the user's hand and control the movement of a Intelligence Research, pp. 815–850, 2011.
keyboard key. Different hand gestures will be used to execute
the keyboard key operations. The device has the potential to [9] J. Hoffmann, “The Metric-FF planning system:
be a viable computer keyboard replacement, but owing to the translating ‘ignoring delete lists’ to numeric state
limits faced, it will not be able to totally replace the computer variables,” Journal of Artificial Intelligence Research,
keyboard. When the template matching hand gesture vol. 20, pp. 291–341, 2003.
identification approach is combined with a machine learning [10] M. Genesereth, N. Love, and B. Pell, “General game
classifier, the accuracy of hand gesture recognition may be playing: Overview of the AAAI competition,” AI
enhanced. When MMX (Multimedia Extension) and SSE magazine, vol. 26, no. 2, 2005.
instructions are available, Open CV is primarily geared for
real-time vision applications. [11] A. Babadi, M. Feyzbakhsh-Rankooh, and G. Ghassem-
Sani, “LePoP: a learning-based heuristic partial order
VIII. ACKNOWLEDGEMENT planner,” unpublished.
We express our profound thanks to the management of [12] J. Clune, “Heuristic evaluation functions for general
MLR Institute of Technology, Dundigal, Hyderabad, for
game playing,” in Proceedings of the AAAI Conference
supporting us to complete this project.
on Artificial Intelligence (AAAI’07), 2007, pp. 1134–
We take immense pleasure in expressing our sincere 1139.
thanks to Dr. K. Srinivasa Rao, Principal, MLR Institute of
Technology, for his kind support and encouragement. [13] S. Schiffel and T. Michael, “Fluxplayer: A Successful
General Game Player,” in Proceedings of the AAAI
We are very much grateful to Dr S.V.S Prasad, Professor Conference on Artificial Intelligence (AAAI’07), 2007,
& Head of the Department, MLR Institute of Technology, for pp. 1191–1196.
encouraging us with his valuable suggestions.
[14] Y. Bjornsson and H. Finnsson, “CadiaPlayer: a
We are very much grateful to M. Vinod, Assistant simulation-based general game player,” IEEE
Professor for his unflinching cooperation throughout the Transactions on Computational Intelligence and AI in
project.
Games, vol. 1, no. 1, 2009.
We would like to express our sincere thanks to the
[15] J. Méhat and T. Cazenave, “A parallel general game
teaching and nonteaching faculty members of ECE Dept.,
player,” KI- Künstliche Intelligenz, vol. 25, no. 1, pp.
MLR Institute of Technology, who extended their help to us
in making our project work successful. 43–47, 2011.
[16] M. Swiechowski, J. Mandziuk, and Y. S. Ong,

5
 “Specialization of a UCT-based General Game Playing
Program to Single-Player Games,” IEEE Transactions
on Computational Intelligence and AI in Games, 2015.
[17] Prateek Joshi, “Open CV with Python By Example”
[18] Triesch, J., von der Malsburg, C.: Robust classification
of hand postures against complex backgrounds. Paper
Presented at the Automatic Face and Gesture
Recognition, 1996. Proceedings of the Second
International Conference on (1996).
[19] Chang, C.C., Chen, J.J., Tai, W.K., Han, C.C.: New
approach for static gesture recognition. J. Inf. Sci. Eng.
22(5), 1047–1057 (2006)
[20] Huang, D.Y., Hu, W.C., Chang, S.H.: Vision-based
hand gesture recognition using PCA+gabor filters and
SVM. Paper Presented at the Intelligent Information
Hiding and Multimedia Signal Process- ing, 2009. IIH-
MSP’09. Fifth International Conference on (2009)
[21] Ankit, C.: Recent Trends in Hand Gesture Recognition,
vol. 3. Science Gate Publishing P.C. (2015)
[22] Dong, G., Yan, Y., Xie, M.: Vision-based hand gesture
recognition for human–vehicle interaction. Paper
Presented at the Proceedings of the International
conference on Control, Automation and Com- puter
Vision (1998)
[23] Yoon, H.S., Soh, J., Bae, Y.J., Seung Yang, H.: Hand
gesture recognition using combined features of location,
angle and veloc- ity. Pattern Recognit. 34(7), 1491–
1501 (2001). doi:10.1016/ S0031-3203(00)00096-0
[24] A Ghotkar, P Vidap and K Deo, "Dynamic Hand
Gesture Recognition using Hidden Markov Model by
Microsoft Kinect Sensor[J]", International Journal of
Computer Applications, vol. 150, no. 5, pp. 5-9,
September 2016.
[25] D Yu, W Jin, W Wei et al., "Surface EMG-Based Inter-
Session Gesture Recognition Enhanced by Deep
Domain Adaptation[J]", Sensors, vol. 17, no. 3, pp. 458,
2017.
[26] H P Gupta, H S Chudgar, S Mukherjee et al., "A
Continuous Hand Gestures Recognition Technique for
Human-Machine Interaction Using Accelerometer and
Gyroscope Sensors[J]", IEEE Sensors Journal, vol. 16,
no. 16, pp. 6425-6432, 2016.
[27] Y Park, J Lee and J Bae, "Development of a Wearable
Sensing Glove for Measuring the Motion of Fingers
Using Linear Potentiometers and Flexible Wires[J]",
Industrial Informatics IEEE Transactions on, vol. 11,
no. 1, pp. 198-206, 2015.
[28] H Koskimäki, P Siirtola and J Röning, "MyoGym:
introducing an open gym data set for activity
recognition collected using myo armband[C]//", ACM
International Joint Conference on Pervasive and
Ubiquitous Computing and the 2017 ACM International
Symposium on Wearable Computers, pp. 537-546,
2017.