Controlling Computer Using Hand Gesture Recognition

CHAPTER 1
INTRODUCTION

1.1 Introduction:

The use of hand gestures provides an attractive alternative to cumbersome

interface devices for human-computer interaction (HCI). In particular, visual
interpretation of hand gestures can help in achieving the ease and naturalness desired for
HCI. The goal of our project is to develop a real-time system capable of understanding
commands given by Hand gestures. The user must be able to communicate to computer,
all basic commands required by a human-computer interface.
Our system will provide following functions:
1: Mouse movement on screen according to the hand movement.
2: Right click, left click, double clicks of mouse.
3: Navigation key press which are up, down, left, right.
4: Number recognition from 0 to 5.

Another advantage is that the user not only can communicate from a distance, but
need have no physical contact with the computer. However, unlike audio commands, a
visual system would be preferable in noisy environments or in situations where sound
would cause a disturbance. The visual system chosen was the recognition of hand
gestures. The amount of computation required to process hand gestures is much greater
than that of the mechanical devices, however standard desktop computers are now quick
enough to make this project hand gesture recognition using computer vision — a viable
proposition. This interface is simple enough to be run using an ordinary webcam and
requires little training.

SCOE, Information Technology, 2010-11.

1
 Controlling Computer Using Hand Gesture Recognition

1.2 Literature Survey:

1) A simplification used in similar type of projects, which was not found in any
recognition methods researched, is the use of a wrist band to remove several degrees of
freedom. This enabled three new recognition methods to be devised. The recognition
frame rate achieved is comparable to most of the systems in existence (after allowance
for processor speed) but the number of different gestures recognized and the recognition
accuracy are amongst the best found. Figure shows several of the existing gesture
recognition systems along with recognition statistics and method.

Table 1.0: Literature survey

Paper Primary Number Backgroun Additiona Number Accurac Fram

method of of d of gesture l markers of y e rate
recognition gestures images required training
recognize e.g. wrist images
d band
[Bauer Hidden 97 General Multi- 7-hours 91.7% -
& Hienz, Markov colored signing
2000] Models Gloves
[Starner, Hidden 40 General No 400 97.6% 10
Weaver Markov training
& Models sentences
Pentland
, 1998]
[Davis & Finate state 7 Static Markers 10 98% 10
Shah, machine/mode on glove sequence
1994] l matching s of 200
frames

SCOE, Information Technology, 2010-11.

2
 Controlling Computer Using Hand Gesture Recognition

2) Enabling Simple Window Management Using Pinch Gesture Recognition by

Benjamin Baldus, Debra Lauterbach, Juan Lizarraga: In this project, they developed a
machine-user interface which implements pinch gesture recognition using simple
computer vision techniques. The interface allows the user to use their hands to control the
mouse as well as easily move and resize windows that are open on their screen. This
interface is simple enough to be run using an ordinary webcam and requires little
training.

One task which is especially cumbersome to perform with the mouse is moving
and resizing open windows. This is something that many computer users do quite often -
opening several applications, folders, browser windows, etc. on their screen, and then
switching between them frequently. But, on most computers, the only way to move a
window is by clicking and dragging it by its header/title bar, and the only way to resize is
by clicking and dragging on the bottom-right corner. This can be annoying to do for
managing multiple windows, which is the problem that this application aims to solve.

Fig shows screenshots of their projects.

SCOE, Information Technology, 2010-11.

3
 Controlling Computer Using Hand Gesture Recognition

Fig.1.2 Window management system

These are further more paper published related to this project.

1) Hand Gesture Recognition Using Computer Vision By -Ray Lockton, Balliol
College, Oxford University
 Identify the letters and number based on the American one-handed sign
language.
 Perform mouse operations
 Text editor
SCOE, Information Technology, 2010-11.

4
 Controlling Computer Using Hand Gesture Recognition

2) Interprets your Hand Gestures By Ashutosh Saxena, Aditya Awasthi, Indian

Institute of Technology Kanpur.
 perform mouse operation

1.3 Project Plan :

Plan of our project is given below. 1st part of plan contains whole image
processing. There are lots of steps to be performed in image processing given below. 2 nd
part contains only gesture implementation part which uses system programming for
implementation as shown below.

Part 1:

 Image processing
 Video capturing from camera.
 Detecting wristband from image.
 Calculating wrist band to hand ratio.
 Calculating region of interest.
 Skin pixel detection.
 Hand segmentation.
 Hand detection and identification of gestures.

Part 2:

 Gesture implementation
 Perform mouse operations depending on hand gestures.
 Perform few keyboard operations like navigation keys,
numbering from (0 to 5).
Block diagram of project is given below:

SCOE, Information Technology, 2010-11.

5
 Controlling Computer Using Hand Gesture Recognition

Fig.1.3 Block Diagram

Block diagram of this project shown above has six blocks like capture image,
detect wristband, hand segmentation, hand detection, gesture detection and gesture
implementation etc.

Capture image is nothing but capturing video input through camera. This input is
given for image processing in further steps. If camera quality is better, captured video is
also better and results to be achieved are also better and efficient. This captured video is
given to next step called as detect wrist band. While using this system user should have
wrist band tied with his/her hand. Without wrist band system is not going to work. Detect
wrist band step scans whole image and selects region of interest where wrist band is
present. Only region of interest of wrist band is detected here no other processing is done.
Next step is hand segmentation. In this step region of interest above the wrist
band where probable hand is present is determined and only that much area is used for
SCOE, Information Technology, 2010-11.

6
 Controlling Computer Using Hand Gesture Recognition

further processing. Skin pixels present in that area is detected, which are nothing but skin
pixels of our hand. In this way hand is segmented. For every people wrist band to hand
ratio is approximately constant so it is beneficial to us while segmenting hand.

Segmented hand is given for hand detection technique. Here two techniques
namely convex hull and convex defect are used for actual detection of hand. Convex hull
technique joins extreme points of polygon which are present in certain area. In our
project convex hull technique joins points which are present on tip of every finger with
each other and to the base of palm. Furthermore convex defect technique calculates actual
defects present between fingers. In this way actual hand is detected.

Detected hand is given for further processing. Gesture detection technique takes
this hand and identifies gestures. Detected gesture could be related to left click, right
click or number counting from 0 to 5.

After identifying gestures, are implemented in gesture implementation technique.

Here some amount of system level programming is required for implementing gestures.
For implementing mouse operations, mouse events are needed to be captured. In this way
our system is implemented for specified operations.

1.3.1 Project Development Life Cycle:

SCOE, Information Technology, 2010-11.

7
 Controlling Computer Using Hand Gesture Recognition

Waterfall model we are going to use for this project implementation.

Fig 1.3.1 Waterfall model

Waterfall model is the most well known software lifecycle development

model. It is very simple to understand and use. Each next phase in this model
must beginning only after the previous phase is over.

 Following are Phases of This Waterfall model:

a) Requirement Analysis and Project Planning:

In this first stage of waterfall model is to understand the requirements.

When the requirements have been noted, it is important to make sure that the
requirements are detailed and accurate and there is no place for any ambiguity.
Before analysis, requirement is gathered from various sources. The sources of
requirements could be anything like internet or any other resources.
SCOE, Information Technology, 2010-11.

8
 Controlling Computer Using Hand Gesture Recognition

After gathering and analysis of requirements project planning is carried

out. Project planning includes actual plan of project including stepwise working
of project and estimation of total cost along with estimation of maintenance cost.
Project planning is done in such a way that project should be cost effective and
efficient.

b) System Design:

The requirements that are gathered in the previous phase are broken down
into logical units, so that the software process becomes easy for implementation.
This is the stage, when the software requirements along with the hardware
requirements for every unit are identified. Then the designs are made accordingly.

Design of system is prepared after whole study of actual plan. Design is

nothing but implementation of plan in rough model which is further refined in
next phases of water fall model. Design of system gives actual idea of system,
how system will look, how it will perform actions etc.

c) System Implementation:

In this phase the actual development of the software takes place. This
phase is also known as coding and verification phase. Based on the algorithms
written in the previous phase, software program is written.

System is implemented with help of coding. Coding can be done in any

language which depends upon developer’s choice and efficiency of coding
language to yield best software. Verification phase verifies code of system for
errors. It checks whether lines of code are sufficient or not. In this way system is
implemented with the help of coding and verification. System is implemented in

SCOE, Information Technology, 2010-11.

9
 Controlling Computer Using Hand Gesture Recognition

various modules as modules are small and easy to implement compared to whole
system at a time.

d) System Integration and Testing:

In this phase all the modules are integrated, after which the software is
tested for correct output. All the bugs introduced due to integration are removed.
There are number of techniques are used for integration and testing depending on
efficiency of technique. Testing techniques used here are unit testing for
individual module, integration testing for integrated modules and system testing
for testing whole system.

If there is any bug in the software, it is removed while testing. Tested

software is given for next phase i.e. deployment phase of water fall model.

e) System Deployment and Maintenance:

This makes for the final phase of the waterfall model, where the software
is deployed at the user side, after it has undergone thorough testing. After the
deployment of the software, routine maintenance work is carried out. In
maintenance phase software is repaired if any problem occurs and returned back
to user. There must be maintenance phase after delivery of software as it
beneficial for maintaining good relationship with customers.

Following are some advantages of Waterfall model

 Simple and easy to use. 
 Easy to manage due to the rigidity of the model – each phase has specific
deliverables and a review process. 
 Phases are processed and completed one at a time.

SCOE, Information Technology, 2010-11.

10
 Controlling Computer Using Hand Gesture Recognition

 Works well for smaller projects where requirements are very well
understood. 

CHAPTER 2
REQUIREMENT GATHERING

SCOE, Information Technology, 2010-11.

11
 Controlling Computer Using Hand Gesture Recognition

2.1 Problem Statement :

“Controlling Computer Using Hand Gesture Recognition ”

Basic purpose is to develop system such that we are able to operate computer
with the help of hand gestures without using mouse. This project helps us in introducing
new computer interface, provided interface will be more natural and user friendly. It also
reduces physical contact with devices and minimizes hardware required for interfacing.

In this project wrist band of single color is used. We first detect wristband using
camera, and calculates region of interest. Wrist band helps in removing background.
Camera detects skin color and segment hand from background image. Finally hand
gesture is identified and respective operations are performed on computer.

SCOE, Information Technology, 2010-11.

12
 Controlling Computer Using Hand Gesture Recognition

Fig.2.1 Gesture recognition system

2.2 Vision:

2.2.1 Background and Basics:

Controlling Computer using Hand Gesture Recognition is a kind of image

processing and research oriented project. It also contains computer vision and
multimedia. In this project we are going to use webcam for capturing videos as
hardware. Any camera that supports streaming video input can be used. No other
external hardware is required. We implement this project in C with the help of
editor DevC++ and OpenCV libraries. OpenCV libraries are special kind of
computer vision libraries which help in capturing videos using webcam easily and
storing images. DevC++ is configured in order to use OpenCV libraries.
WindowsXP OS is used for operations.

SCOE, Information Technology, 2010-11.

13
 Controlling Computer Using Hand Gesture Recognition

A visual system would be preferable in noisy environments or in

situations where sound would cause a disturbance. The visual system chosen was
the recognition of hand gestures. The amount of computation required to process
hand gestures is much greater than that of the mechanical devices, however
standard desktop computers are now quick enough to make this project hand
gesture recognition using computer vision — a viable proposition.

This project helps us in introducing new computer interface, provided

interface will be more natural and user friendly. It also reduces physical contact
with devices and minimizes hardware required for interfacing.

2.2.2 Scope:

1. Capturing video input from camera:

Webcam with desirable quality is used for capturing input. We have

written code using OpenCV library to capture images from camera. Here Dev C+
+ editor is used for editing and running program. cvCaptureFromCAM() and
cvQueryFrame() are OpenCV functions used to capture images from camera.

2. Detection of wristband:

User has to use wrist band while using this system. Without wristband it is
impossible to use system. Wrist band of yellow color with width of 4cm is used.
This project first detects wrist band and finds region of interest of wrist band.
Wrist band helps in easily segmenting hand from background image.

SCOE, Information Technology, 2010-11.

14
 Controlling Computer Using Hand Gesture Recognition

3. Segmentation of hand from background image:

Probable area of hand is determined using wrist band. Only probable area
of hand is scanned for next step. cvRectangle() is OpenCv function to determine
region of interest. Skin detection technique is used within region of interest so that
actual hand is detected. Only region of interest is considered and remaining
portion of image is ignored. In this way processing region is reduced and
processing time is saved here. This segmented hand is given for further
processing.

4. Recognize the hand gesture.

In above step actual hand is not detected. Convex hull and convex defects
techniques are used to calculate exact hand. Convex hull technique calculates
contour of hand including finger tips. Convex defect technique used to calculate
defects in between fingers. In this way actual hand is traced out.

5. Implementation of gestures:

After detecting hand gestures, are implemented with the help of

windows.h file of system. Some amount of system level programming is required
to be implemented here. Performing the PC control activities depending upon
gesture recognized such as:

1: Mouse movement on screen according to the hand movement.

2: Right click, left click, double clicks of mouse.
3: Navigation key press which are up, down, left, right.
4: Number recognition from 0 to 5.

2.2.3Requirements:
SCOE, Information Technology, 2010-11.

15
 Controlling Computer Using Hand Gesture Recognition

a) Related Hardware Requirements: Computer and Webcam.

Computer:

Any standard PC running Windows XP is capable of running our project.

The minimum hardware requirements are those of the operating system and the
required software.

Webcam:

We used a standard USB webcam of desired resolution. This viewing

area proved to be sufficient, though a larger area would have advantages as far as
having more room for performing gestures. Webcam with maximum resolution
will be more efficient. You can use webcam of higher resolution. Webcam should
be in front of user that is parallel to screen pointing towards user.

b) Operating Systems:

WindowsXP OS:

This operating system developed by Microsoft is a stable and trustworthy

platform for running applications made with Dev C++.

c) Availability of resources: OpenCV Libraries and DevC++ .

SCOE, Information Technology, 2010-11.

16
 Controlling Computer Using Hand Gesture Recognition

OpenCV Libraries:

It is a C++ based computer vision library developed by Intel. It focuses

mainly on real-time image processing, and gives the user a large amount of tools
that provide extensive functionality on this field. In our project we used OpenCV
for capturing frames from the webcam and processing the images using
morphology Monitor.

cvBlobsLib is a library for OpenCV which performs binary image

connected components labeling. It also has additional functionalities such as
manipulating, filtering, and extracting blob features. This library is used in our
project to determine when a pinch is being made, and to determine the center of
the blob for moving the mouse.

Dev C++:

Dev C++ is the main editor used for coding purpose. In order to use
OpenCV library, we have configured OpenCV with DevC++. Configuration
includes adding library paths for .lib files, .bin files, include files for both C as
well as C++.

Latest version of Dev C++ is used that is Dev C++ 4.9.9.2.

d) Network requirement :

This project is performed on single stand alone machine. So there is no

need of networks.

e)Other Requirements :

SCOE, Information Technology, 2010-11.

17
 Controlling Computer Using Hand Gesture Recognition

Yellow color Wrist Band:

We have used yellow color wrist band of width approximately 4 cm. This
band is helpful for segmenting hand from background image. This band avoids
extra work needed for subtracting background from foreground. In this way it
reduces extra efforts and adds flexibility to our projects.

For best use of this system we must require good camera and user with at
least a hand. Here we have defined some conventions for operations such as left-
click, right-click etc. Depending on gestures recognized specific operations are
performed on computer.

System should easily capture hand images and should give quick response
to the hand gestures to perform respective operations. It should support maximum
requirements of user. It should provide better flexibility to the user. It should have
better efficiency and minimum drawbacks.

System provides following services to user:

1: Mouse movement on screen according to the hand movement.
2: Right click, left click, double clicks of mouse.
3: Navigation key press which are up, down, left, right.
4: Number recognition from 0 to 5.

2.3 System Requirement Specification

2.3.1 Introduction
SCOE, Information Technology, 2010-11.

18
 Controlling Computer Using Hand Gesture Recognition

2.3.1.1.Problem Statement:
Controlling Computer Using Hand Gesture Recognition

2.3.1.2. Describe the Scope of the System:

1. Capturing video input from camera:

Webcam with desirable quality is used for capturing input. We have

written code using OpenCV library to capture images from camera. Here Dev C+
+ editor is used for editing and running program. cvCaptureFromCAM() and
cvQueryFrame() are OpenCV functions used to capture images from camera.

2. Detection of wristband:

User has to use wrist band while using this system. Without wristband it is
impossible to use system. Wrist band of yellow color with width of 4cm is used.
This project detects wrist band and finds region of interest of wrist band. Wrist
band helps in easily segmenting hand from background image.

3. Segmentation of hand from background image:

Probable area of hand is determined using wrist band. Only probable area
of hand is scanned for next step. cvRectangle() is OpenCV function to determine
region of interest. Skin detection technique is used within region of interest so that
actual hand is detected. Only region of interest is considered and remaining
portion of image is ignored. In this way processing region is reduced and
processing time is saved here. This segmented hand is given for further
processing.

4. Recognize the hand gesture.

SCOE, Information Technology, 2010-11.

19
 Controlling Computer Using Hand Gesture Recognition

In above step actual hand is not detected. Convex hull and convex defects
techniques are used to calculate exact hand. Convex hull technique calculates
contour of hand including finger tips. Convex defect technique used to calculate
defects in between fingers. In this way actual hand is traced out.

5. Implementation:

After detecting hand gestures, are implemented with the help of system
level programming which contains windows.h file of system. Some amount of
system level programming is required to be implemented here. Performing the
PC control activities depending upon gesture recognized such as:

1: Mouse movement on screen according to the hand movement.

2: Right click, left click, double clicks of mouse.
3: Navigation key press which are up, down, left, right.
4: Number recognition from 0 to 5.

2.3.2. Proposed System (Project)

2.3.2.1 System Overview :

SCOE, Information Technology, 2010-11.

20
 Controlling Computer Using Hand Gesture Recognition

Project requires stand alone PC with windows XP installed on it. Because

XP performs smooth operations, it is flexible and supports maximum of functions.
Webcam is required for capturing video input. Softwares required are OpenCV
library and Dev C++ editor. OpenCV is computer vision library which provides
all functionality related to vision of computer and image processing. Wrist band is
another requirement of system.

The detailed overview of this project is given below:

1. Capturing the live video input from camera:

Camera used can be a simple webcam. We need to capture the live feed of
camera. One solution to this can be OpenCV. We can use OpenCV library which
is supported on common operating systems like Windows and Linux. Captured
video is given for processing. Program is written using OpenCV libraries.
OpenCV library has lots of functions related to image processing and GUI
programming. Camera quality affects lots of things in processing.

SCOE, Information Technology, 2010-11.

21
 Controlling Computer Using Hand Gesture Recognition

Fig.2.3.2a Captured image through camera

2. Detection of wrist band:

Wrist band detection is crucial step in this project. It has many advantages
over other techniques. A yellow color wristband is used in this project. As name
suggests wrist band is tied with wrist. Captured image is scanned for identifying
band. If there is no band present in image, system is not going to work properly.
Wrist band helps in removing background image from foreground image.
Background subtraction is very difficult task in this project, which is made easier
by wrist band detection technique.

Another advantage of wrist band is background of variant color also has

no effect on system performance. If band gets detected properly, then system will

SCOE, Information Technology, 2010-11.

22
 Controlling Computer Using Hand Gesture Recognition

run properly. But there is a disadvantage of this system, if background contains

color similar to wrist band then there is a problem while processing the image.

Screen shot below shows how wristband is detected, region of interest of

wrist band and probable region of hand above the wrist band.

SCOE, Information Technology, 2010-11.

23
 Controlling Computer Using Hand Gesture Recognition

Fig.2.3.2b) Wristband detection

3. Segmentation of hand:

Before we talk about the segmentation of hand from background we need

to consider the type of background used. It can be

a)Constant: Considering the plane background, segmentation of hand

from background is comparatively easy. It would be far easy to segment hand if
the background color is different from skin color.

b)Varying: In varying background segmenting hand is quite tedious job.

Because you not only need to recognize the hand but have to remove all
background objects which resembles to skin color.

Here in this project, background subtraction becomes easy as we are using

wrist band to detect hand. Using wrist band background of any color can be easily
removed but it fails when background has same color with wristband or skin.

After detecting wrist band, wrist band to hand ratio is calculated for
determining region of interest of hand. In this region of interest (ROI) skin pixel
detection technique is used in order to detect hand. Only region of interest is
scanned in further phases. The R-G-B values of the skin can be used to separate
out pixels of skin from the background.

SCOE, Information Technology, 2010-11.

24
 Controlling Computer Using Hand Gesture Recognition

OpenCV libraries provide number of functions for detecting skin pixels

from image. In this way actual hand is segmented from whole image in order to
reduce processing.

4. Recognizing hand:

There are number of methods to recognize hand, but we have used

wristband technique to recognize hand. Firstly wrist band is detected depending
on its color, region of interest of wrist band is calculated. We have used wrist
band to hand ratio to detect probable hand from image. Skin color separates hand
from background image. Skin color plays important role in detecting hand. The R-
G-B values of the skin can be used to separate out pixels of skin from the
background.

There are two techniques to detect exact hand from ROI. These are convex
hull and convex defect. Convex hull technique joins extreme points of hand
including finger tips. It forms polygon of hand whose points are connected.
Convex defect technique finds actual defects in between fingers to give exact
image of hand. After getting exact image of hand we fill that image with color
which is less used in environment.

Screen shot below shows segmented hand, ROI of wrist band along with
separated hand from whole background.

SCOE, Information Technology, 2010-11.

25
 Controlling Computer Using Hand Gesture Recognition

Fig.2.3.2c Hand detection

5. Identify the gesture:

Gesture identification may or may not need the gesture database,

depending upon the algorithm used for identification. Here in this project we have
used convex hull and convex defect techniques to segment and detect hand
correctly. Convex hull and convex defect techniques determine exact portion of
hand which required for us to perform operations. In this way these techniques
helps in identifying hand gesture.

If current hand gesture is different from previous then gesture is detected.

We have predefined some gestures in the program.

SCOE, Information Technology, 2010-11.

26
 Controlling Computer Using Hand Gesture Recognition

Following figure shows how gestures are identified? In figure identified

gesture is number 4 as four fingers are straight.

Fig.2.3.2d Gesture identification

SCOE, Information Technology, 2010-11.

27
 Controlling Computer Using Hand Gesture Recognition

6. Implementing the gesture:

After recognizing the gesture we can evaluate its meaning and do the
action which is meant to be done. Some amount of system level programming is
required to be implemented here. Performing the PC control activities depending
upon gesture recognized such as cursor movements like up, down, left and right
etc. left click, right click, double click and numbers counted from 0 to 5.

1. Moving Mouse Cursor:

We used the wrist band center as a cursor controller to control mouse

cursor. We used the following approach for moving the mouse cursor. The
method is mapping cursor control. It means that the center of wrist band on a
camera screen can position maps to a desktop screen position. In other words, the
mouse cursor is placed on the desktop window along with the center of wrist band
position displayed on the camera screen position.

This method has a problem. If the resolution of the desktop window has a
higher resolution than the camera resolution, then the cursor position cannot be
accurate because when the camera resolution converts to the desktop window
resolution we lose intermediate value. We expect the ratio of jumping pixel is up
to 4 pixels. Another problem is when any one of the finger goes outside the
camera boundary then gesture is not identified properly. Cursor gets moved
according to center of wrist band only if index finger is straight and all other
fingers are curled. This method gives more accurate results.

2. Left Clicking and Double-Clicking:

SCOE, Information Technology, 2010-11.

28
 Controlling Computer Using Hand Gesture Recognition

To call system event for left clicking, at least two convex hull vertexes
have to be off the palm area which was computed in the previous part. In addition,
the x position of one of two vertexes should be lower than the other to restrict
detection of other finger tips. When the degree of the index finger and thumb is 70
to 90 degree then we can recognize that the gesture is left clicking. Actually, if the
thumb is placed outside the circle of the hand region, then the gesture is left
clicking. The double-clicking occurs when the thumb moves 0 to 90 degree and
back two times fast.

3. Right Clicking:

We simply implemented this part using two fingers as index finger and
middle finger. If both fingers are straight and separated by small space then
system performs right click operation. Left click and right click are performed if
count of fingers is two. But how does system differentiate between them? We
used a method in which distance between tip of fingers and angle between them is
calculated. If distance is large and angle is acute then left click is performed
otherwise right click is performed.

4. Recognizing numbers from 0 to 5:

We have implemented this with the help of number of straight fingers. If

there are two straight fingers then count is 2, three straight fingers then count is 3
and so on.

Figure given below shows, how numbers are recognized? As there are
three fingers straight so count is 3. In this way numbers are recognized from 0 to
5 using hand gesture recognition.

SCOE, Information Technology, 2010-11.

29
 Controlling Computer Using Hand Gesture Recognition

Fig.2.3.2e Recognition of numbers

2.3.3. Project Status

A) Classification :

The project can be classified into modules as:

SCOE, Information Technology, 2010-11.

30
 Controlling Computer Using Hand Gesture Recognition

1. Wrist band detection

2. Hand segmentation
3. Hand Detection
4. Gesture identification and implementation.

B) Definition :

Basic purpose is to develop system such that we are able to operate

computer with the help of hand gestures without using mouse. This project helps
us in introducing new computer interface, provided interface will be more natural
and user friendly. It also reduces physical contact with devices and minimizes
hardware required for interfacing.

In this project wrist band of single color is used. We first detect wristband
using camera, and calculates region of interest. Wrist band helps in removing
background. Camera detects skin color and segment hand from background
image. Finally hand gesture is identified and respective operations are performed
on computer.

C) Responsibilities:

For best use of this system we must require good camera, wrist band and
user with at least a hand. Here we have defined some conventions for operations
such as left-click, right-click etc. depending on gestures recognized specific
operations are performed on computer.

System should easily capture hand images and should give quick response
to the hand gestures to perform respective operations. It should support maximum
requirements of user. It should provide better flexibility to the user. It should have
better efficiency and minimum drawbacks.
SCOE, Information Technology, 2010-11.

31
 Controlling Computer Using Hand Gesture Recognition

System provides following services to user:

1: Mouse movement on screen according to the hand movement.

2: Right click, left click, double clicks of mouse.
3: Navigation key press which are up, down, left, right.
4: Number recognition from 0 to 5.

D) Composition

1. Design [ UML Diagrams ]:

Use case Diagram:

SCOE, Information Technology, 2010-11.

32
 Controlling Computer Using Hand Gesture Recognition

System

Switch on/off web cam

<<include>>
Valid guestures

Makes guestures

<<include>>
User
Valid operations
initiates different operations

System

Checks whether cam is on

Detect Wrist Band

Detect Region of Interest

Machine

Detect Hand

Identify gesture

Perform Operation

Fig.2.3.3a Use case Diagram

Class Diagram:

SCOE, Information Technology, 2010-11.

33
 Controlling Computer Using Hand Gesture Recognition

User Machine

+char name +int cam

+int hand +int fun
+int temp1 +int temp
gives input to
+int cam() +int check()
+void makegesture() +int WBdetect(int color)
+int hdetect()
+void improcess()
+pform_operation()

Fig.2.3.3b Class Diagram

Sequence Diagram:

SCOE, Information Technology, 2010-11.

34
 Controlling Computer Using Hand Gesture Recognition

User Machine

1 : power on cam()

2 : Cam on()
3 : Moves hand to cam()
4 : Detect Wrist Band()

5 : Detect Region Of Interest()

6 : Detect Hand()

7 : Make gestures()

8 : Capture image()

9 : Process image()

10 : Identify Gesture()

11 : Perform operation()

sd ALt
12 : Error()

13 : Displays error message()

Fig.2.3.3c Sequence Diagram

Activity Diagram:

SCOE, Information Technology, 2010-11.

35
 Controlling Computer Using Hand Gesture Recognition

User Machine

Switch on cam Cam switched on

Moves hand to cam Detects Wrist Band

Detect region of Interest

Detect hand

Makes gestures Captures image

processes image

Identify gesture

[gesture identified] [gesture not identified]

Perform Operation Report error

Fig.2.3.3d Activity Diagram

DFD:

SCOE, Information Technology, 2010-11.

36
 Controlling Computer Using Hand Gesture Recognition

DFD Level 0:

Mouse
Operations

K/B
Operations
Hand gestures Machine/OS

Processing

Fig.2.3.3e DFD Level 0

DFD Level 1:

SCOE, Information Technology, 2010-11.

37
 Controlling Computer Using Hand Gesture Recognition

Perform Operations

User Machine

Hand Gestures

Identified Gestures
Capture
Gestures
Through
WEBCAM

Provide Image input

Processing

Fig.2.3.3f DFD Level 1

CHAPTER 3
PROJECT ANALYSIS

SCOE, Information Technology, 2010-11.

38
 Controlling Computer Using Hand Gesture Recognition

3.1 Use-case Model :

System

Switch on/off web cam

<<include>>
Valid guestures

Makes guestures

<<include>>
User
Valid operations
initiates different operations

System

Checks whether cam is on

Detect Wrist Band

Detect Region of Interest

Machine

Detect Hand

Identify gesture

Perform Operation

Fig.3.1 Use case Diagram

3.2 Use case Specification/Document-Scenario description:

3.2.1 Use case specification: Make gesture

1. Description
Use case allows user to make gestures.
SCOE, Information Technology, 2010-11.

39
 Controlling Computer Using Hand Gesture Recognition

2. Actor
User

3. Flow of event
3.1Basic flow-
3.1.1 Start the application
3.1.2 Make the gestures
3.2Alternate flow
3.2.1 Make different gesture

4. Special Requirements:
Camera is installed properly.

5. Precondition
5.1 Camera is on
5.2 Wrist band must be used.

6. Post condition
6.1 Gestures made successfully.

3.2.2 Use case specification: Detect hand

1. Description
Use case allows machine to detect hand.

2. Actor

SCOE, Information Technology, 2010-11.

40
 Controlling Computer Using Hand Gesture Recognition

Machine

3. Flow of event
3.1Basic flow-
3.1.1 Switch on the camera
3.1.2 Capture gesture image
3.1.3 Detect wrist band.
3.1.4 Detect Hand.

4. Special Requirements:
Camera is installed properly.

5. Precondition
5.1 Camera is on
5.2 Wrist band must be used.

6. Post condition
6.1 Hand detected successfully.

3.2.3 Use case specification: Identify Gestures

1. Description
Use case allows machine to recognize gestures.

2. Actor
Machine
SCOE, Information Technology, 2010-11.

41
 Controlling Computer Using Hand Gesture Recognition

3. Flow of event
3.1Basic flow-
3.1.1 Switch on the camera
3.1.2 Capture gesture image
3.1.3 Detect wrist band.
3.1.4 Detect Hand.
3.1.4 Identify gesture.

4. Special Requirements:
Camera is installed properly.

5. Precondition
5.1 Camera is on
5.2 Wrist band must be used.

6. Post condition
6.1 Gesture identified successfully.

3.2.4 Use case specification: Perform Operations

1. Description
Use case allows machine to perform operations depending on gestures.

2. Actor
Machine

SCOE, Information Technology, 2010-11.

42
 Controlling Computer Using Hand Gesture Recognition

3. Flow of event
3.1Basic flow-
3.1.1 Switch on the camera
3.1.2 Capture gesture image
3.1.3 Detect wrist band.
3.1.4 Detect Hand.
3.1.4 Identify gesture.
3.1.5 Perform operation depending on gesture.

4. Special Requirements:
Camera is installed properly.

5. Precondition
5.1 Camera is on
5.2 Wrist band must be used.

6. Post condition
6.1 Operation performed successfully depending on gesture made.

3.3 Use case Analysis:

3.3.1 Activity Diagram:

SCOE, Information Technology, 2010-11.

43
 Controlling Computer Using Hand Gesture Recognition

User Machine

Switch on cam Cam switched on

Moves hand to cam Detects Wrist Band

Detect region of Interest

Detect hand

Makes gestures Captures image

processes image

Identify gesture

[gesture identified] [gesture not identified]

Perform Operation Report error

Fig.3.3.1 Activity Diagram

3.3.2 Sequence Diagram:

SCOE, Information Technology, 2010-11.

44
 Controlling Computer Using Hand Gesture Recognition

User Machine

1 : power on cam()

2 : Cam on()
3 : Moves hand to cam()
4 : Detect Wrist Band()

5 : Detect Region Of Interest()

6 : Detect Hand()

7 : Make gestures()

8 : Capture image()

9 : Process image()

10 : Identify Gesture()

11 : Perform operation()

sd ALt
12 : Error()

13 : Displays error message()

Fig.3.3.2 Sequence Diagram

CHAPTER 4
PROJECT DESIGN

SCOE, Information Technology, 2010-11.

45
 Controlling Computer Using Hand Gesture Recognition

4.1 Design Model:

4.1.1 Class Diagram:

User Machine

+char name +int cam

+int hand +int fun
+int temp1 +int temp
gives input to
+int cam() +int check()
+void makegesture() +int WBdetect(int color)
+int hdetect()
+void improcess()
+pform_operation()

Fig.4.1.1 Class Diagram

4.1.2Data Flow Diagrams:

DFD Level 0:

SCOE, Information Technology, 2010-11.

46
 Controlling Computer Using Hand Gesture Recognition

Mouse
Operations

K/B
Operations
Hand gestures Machine/OS

Processing

Fig.4.1.2a DFD Level 0

DFD Level 1:

SCOE, Information Technology, 2010-11.

47
 Controlling Computer Using Hand Gesture Recognition

Perform Operations

User Machine

Hand Gestures

Identified Gestures
Capture
Gestures
Through
WEBCAM

Provide Image input

Processing

Fig.4.1.2b DFD Level 1

4.2 Implementation Model:

4.2.1 Component Diagram:

SCOE, Information Technology, 2010-11.

48
 Controlling Computer Using Hand Gesture Recognition

Webcam

captured gestures

Computer
Hand movements

Users

Fig.4.2.1 Component Diagram

CHAPTER 5
TESTING

SCOE, Information Technology, 2010-11.

49
 Controlling Computer Using Hand Gesture Recognition

5.1Test case Design- Unit/Integration:

Manual Testing:-

Manual testing is the process of manually testing software for defects. It requires
a tester to play the role of an end user, and use most of all features of the application to
ensure correct behavior. To ensure completeness of testing, the tester often follows a
written test plan that leads them through a set of important test cases. It is the oldest and
most rigorous type of software testing. Manual testing requires a tester to perform manual
test operations on the test software without the help of Test automation. Manual testing is
a laborious activity that requires the tester to possess a certain set of qualities; to be
patient, observant, speculative, creative, innovative, open-minded, resourceful,
unopinionated, and skillful.

Unit Testing:-

This initial stage in testing normally carried out by the developer who wrote the
code and sometimes by a peer using the white box testing technique. The most 'micro'
scale of testing; to test particular functions or code modules. Typically it is done by the
programmer and not by testers, as it requires detailed knowledge of the internal program
design and code. Not always easily done unless the application has a well-designed
architecture with tight code; may require developing test driver modules or test harnesses.

Here in our project we have tested each module uniquely by using unit testing.
We have provided facility to recover project when error occurs.

Integration testing:-

SCOE, Information Technology, 2010-11.

50
 Controlling Computer Using Hand Gesture Recognition

It is a testing of combined parts of an application to determine if they function

together correctly. The 'parts' can be code modules, individual applications, client and
server applications on a network, etc. This type of testing is especially relevant to
client/server and distributed systems. This stage is carried out in two modes? viz?. As a
complete package or as a increment to the earlier package. Most of the time black box
testing technique is used. However, sometimes a combination of Black and White box
testing is also used in this stage.

Test cases:

Table 5.1.1 Wristband detection test case

Project: Controlling Computer

Using Hand Gesture
SCOE, Information Technology, 2010-11.

51
 Controlling Computer Using Hand Gesture Recognition

Recognition.
Module: Wrist band detection
Form REF: Main
Test Case 1
No:
Functional Wrist band detection
Specification
Test Date: 9/04/11 1min
Test To check whether user has
Objective: put wrist band on or not.
Test Data: Wrist band
Precondition Camera should be on
Step No Steps Data Expected Results Actual
Results
1 Use hand without wrist Camera input Show message as YES,
band. put wrist band on message is
displayed
2 Use wrist band with color Camera input Show message as YES,
other than yellow. put wrist band on, message is
as wrist band other displayed
than yellow color
is not detected.
3 Use wrist band with Camera input Wrist band is YES, wrist
yellow color. detected and ROI band is
of wrist band is detected
calculated. properly.

Table 5.1.2 Hand segmentation test case

Project: Controlling Computer

Using Hand Gesture
Recognition.
Module: Hand segmentation
Form REF: Main
SCOE, Information Technology, 2010-11.

52
 Controlling Computer Using Hand Gesture Recognition

Test Case 2
No:
Functional Segmentation of hand
Specification
Test Date: 9/04/11 1min
Test To check whether hand is
Objective: segmented properly or not.
Test Data: Hand with wrist band
Precondition Camera should be on.
Step No Steps Data Expected Results Actual
Results
1 Use hand without wrist Camera input Show message as YES,
band in front of camera. put wrist band on. message is
displayed and
hand is not
segmented.
2 Use hand with wrist band Camera input Show message as YES,
with color other than put wrist band on, message is
yellow in front of camera. as wrist band other displayed and
than yellow color hand is not
is not detected. segmented.
3 Use hand with wrist band Camera input Wrist band is YES, wrist
of yellow color in front of detected and ROI band is
camera.. of hand is detected and
calculated and hand is
hand is segmented. segmented
properly.

Table 5.1.3 Gesture identification test case

Project: Controlling Computer

Using Hand Gesture
Recognition.
Module: Gesture identification
Form REF: Main

SCOE, Information Technology, 2010-11.

53
 Controlling Computer Using Hand Gesture Recognition

Test Case 3
No:
Functional Identification of hand
Specification
Test Date: 9/04/11 1min
Test To check whether gesture
Objective: is identified properly or
not.
Test Data: Hand gestures
Precondition 1) Camera should be
on.
2) Hand with yellow
wrist band.
Step No Steps Data Expected Results Actual
Results
1 Hand with unspecified Camera input Wait for specified Do nothing.
gestures. gestures.
2 Hand with specified Camera input Gesture is YES, gesture
gestures. identified properly. is identified
and message
is displayed
(e.g. left
click, right
click etc.)

Table 5.1.4 Perform operations test case

Project: Controlling Computer

Using Hand Gesture
Recognition.
Module: Perform operations
Form REF: Main
Test Case 4
SCOE, Information Technology, 2010-11.

54
 Controlling Computer Using Hand Gesture Recognition

No:
Functional Implementations of
Specification operations
Test Date: 9/04/11 1min
Test To check whether
Objective: operations are performed
correctly or not.
Test Data: Hand gestures
Precondition 1) Camera should be
on.
2) Hand with yellow
wrist band.
Step No Steps Data Expected Results Actual
Results
1 Hand with unspecified Camera input Wait for specified Do nothing.
gestures. gestures and no
operations
performed.
2 Hand with specified Camera input Operations are YES,
gestures. performed operations are
depending on performed
gestures. properly.

Table 5.1.5 Left click test case

Project: Controlling Computer

Using Hand Gesture
Recognition.
Module: Perform operations
SCOE, Information Technology, 2010-11.

55
 Controlling Computer Using Hand Gesture Recognition

Form REF: Main

Test Case 5
No:
Functional Implementations of left
Specification click
Test Date: 9/04/11 1min
Test To check whether left
Objective: click is performed
correctly or not.
Test Data: Hand gestures
Precondition 1) Camera should be
on.
2) Hand with yellow
wrist band.
Step No Steps Data Expected Results Actual
Results
1 Hand with unspecified Camera input Wait for specified Do nothing.
gestures. gestures and no
operations
performed.
2 Hand with only index Camera input Left click operation YES, left
finger and thumb are is performed. click is
straight and angle between performed
them is acute. properly.

Table 5.1.6 Cursor movement test case

Project: Controlling Computer

Using Hand Gesture
Recognition.
Module: Perform operations
Form REF: Main
SCOE, Information Technology, 2010-11.

56
 Controlling Computer Using Hand Gesture Recognition

Test Case 6
No:
Functional Implementations of cursor
Specification movement
Test Date: 9/04/11 1min
Test To check whether cursor
Objective: movement operation
performed correctly or not.
Test Data: Hand gestures
Precondition 1) Camera should be
on.
2) Hand with yellow
wrist band.
Step No Steps Data Expected Results Actual
Results
1 Hand with unspecified Camera input Wait for specified Do nothing.
gestures. gestures and no
operations
performed.
2 Hand with only index Camera input Cursor moves YES, cursor
finger is straight and all according to hand movement
other fingers are curled movements. takes place.
and movement of hand
takes place.

Table 5.1.7 Right click test case

Project: Controlling Computer

Using Hand Gesture
Recognition.
Module: Perform operations
Form REF: Main
Test Case 7
No:
SCOE, Information Technology, 2010-11.

57
 Controlling Computer Using Hand Gesture Recognition

Functional Implementations of right

Specification click
Test Date: 9/04/11 1min
Test To check whether right
Objective: click is performed
correctly or not.
Test Data: Hand gestures
Precondition 1) Camera should be
on.
2) Hand with yellow
wrist band.
Step No Steps Data Expected Results Actual
Results
1 Hand with unspecified Camera input Wait for specified Do nothing.
gestures. gestures and no
operations
performed.
2 Hand with only index Camera input Right click YES, right
finger and middle fingers operation is click
are straight and all other performed. operation is
fingers are curled. performed.

CHAPTER 6
CONCLUSIONS

The goal of this project is to create a system that will recognize the hand gestures
and control Computer according to those gestures. The project will benefit the mobile
systems where using pointing devices like mouse is difficult. In this project video camera

SCOE, Information Technology, 2010-11.

58
 Controlling Computer Using Hand Gesture Recognition

captures 25 fps. It is considered that a modern computer system would therefore allow
the project goals to be exceeded.

Before actual implementing operations, wrist band detection, determining region

of interest, hand segmentation and identifying gestures from stored frames need to be
done. After detecting wristband, band to hand ratio is calculated, which is helpful in
segmenting hand from background image. Convex hull and Convex defect techniques are
used to find actual hand image and depending on that gestures identified. After gesture
identification we need to implement those gestures to system level mouse and keyboard
instructions.

In future project can be used for multimedia applications like playing games
without using mouse also all keyboard operations can be performed using this project.
But enhancing this project is quite complex and difficult, so lots of hard work is required.

However, it is difficult to get stable results because of the variety of lighting and
skin colors of human races. Most vision algorithms have illumination issues. From the
results, we can expect that if the vision algorithms can work in all environments then our
system will work more efficiently. This system could be useful in presentations and to
reduce work space.

CHAPTER 7
BIBLIOGRAPHY

[1].[Bauer & Hienz, 2000] Relevant feature for video-based continuous sign language
recognition. Department of Technical Computer Science, Aachen University of
Technology, Aachen, Germany, 2000.
SCOE, Information Technology, 2010-11.

59
 Controlling Computer Using Hand Gesture Recognition

[2].[Davis & Shah, 1994] Visual gesture recognition. In proceedings IEEE Visual
Image Signal Process, 1994, vol.141, No.2, pages 101-106.

[3].J. Brand and J. Mason, "A Comparative Assessment of Three Approaches to

Pixel-level Human Skin Detection", IEEE International Conference on Pattern
Recognition, 2000.

[4].Mark Tabb and Narendra Ahuja, “Multiscale Image Segmentation by Integrated

Edge and Region Detection”, IEEE Transactions on Image Processing, vol. 6, no.
5, May 1997.

[5].R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice-Hall, 2nd

edition, 2002.

[6].Zhi Li, Ray Jarvis, “Real time Hand Gesture Recognition using a Range Camera”.

SCOE, Information Technology, 2010-11.

60