GYAN GANGA COLLEGE OF TECHNOLOGY A Project Report on
GESTURE RECOGNITION USING MATLAB Submi tted tted in par par tial ti al f ul f i l l ment ment f or th e awar awar d of of the degre degree e Of
BACHELOR OF ENGINEERING
In
ELECTRONICS & COMMUNICATION COMMUNICATION ENGINEERING Submitted By: AMEE VISHWAKARMA, APOORVA SRIVASTAVA, DEBOLINA SUR, MOMITA SAHA & MONALISA HAZRA Enrollment No.: 0208EC101017, 0208EC101028, 0208EC101036, 0208EC101055, 0208EC101056
Guided by: Mr. Rajender Yadav
Designation: Asst. Professor Deptt.of Electronics & Communication Engineering Gyan Ganga College of Technology, Jabalpur
CERTIFICATE This is to certify that the Minor Project entitled RECOGNITION
USING
VISHWAKARMA,
APOORVA
MATLAB ” ”
submitted
SRIVASTAVA,
“
G ESTURE GESTURE
by
AMEE
DEBOLINA
SUR,
M OM I TA SAHA SAHA & M ON ONALI ALI SA HAZRA HAZRA has been carried out under my
guidance & supervision. The project is approved for submission towards BACH H ELOR OF partial fulfillment as required for the award of degree of BAC ENGINEERI NG in ELECTRO ELECTRONI NI CS & CO COM M M UNI CAT CATII ON from Gyan Ganga
College
of
Technology,
Jabalpur
under RAJIV
GANDHI
PROUDYO PR OUDYOGIKI GIKI VI SH WAVI DYALAYA, BHOPAL (M .P. P.))
SIGNATURE
SIGNATURE
Mrs. Papiya Dutta
Mr. Rajender Yadav
HEAD OF THE DEPARTMENT
GUIDE
CERTIFICATE This is to certify that the Minor Project entitled RECOGNITION
USING
VISHWAKARMA,
APOORVA
MATLAB ” ”
submitted
SRIVASTAVA,
“
G ESTURE GESTURE
by
AMEE
DEBOLINA
SUR,
M OM I TA SAHA SAHA & M ON ONALI ALI SA HAZRA HAZRA has been carried out under my
guidance & supervision. The project is approved for submission towards BACH H ELOR OF partial fulfillment as required for the award of degree of BAC ENGINEERI NG in ELECTRO ELECTRONI NI CS & CO COM M M UNI CAT CATII ON from Gyan Ganga
College
of
Technology,
Jabalpur
under RAJIV
GANDHI
PROUDYO PR OUDYOGIKI GIKI VI SH WAVI DYALAYA, BHOPAL (M .P. P.))
SIGNATURE
SIGNATURE
Mrs. Papiya Dutta
Mr. Rajender Yadav
HEAD OF THE DEPARTMENT
GUIDE
GYAN GANGA COLLEGE OF TECHNOLOGY CERTIFICATE This is to certify that the Minor Project report entitled
“
G ESTURE GESTURE
RECGNITION USING MATLAB has been submitted by A M E E ” ”
VISHWAKARMA,
APOORVA
SRIVASTAVA,
DEBOLINA
SUR,
M OMI TA SAHA SAHA & M ONALI SA HAZRA HAZRA for partial fulfillment of the BACH H ELOR OF OF ENGINEERI NG required for the award of degree of BAC in E L ECTRO ECTRONI NI CS & CO COM M M UNI CATI ON at Gyan Gyan Ganga Ganga Co Coll ll ege of Technology
under
RAJIV
GANDHI
PROUDYOGIKI
VI SH WAVI DYALAYA, BH OP OPAL AL (M .P. P.) )
Internal Examiner Date:
External Examiner Examiner Date:
ACKNOWLEDGEMENT
We would like to express my sincere gratitude to Dr. R.K Ranjan, Principal and Mrs. Papiya Dutta, H.O.D of Electronics and Communication Department of Gyan Ganga College Of Technology for providing me with an opportunity to do our minor project on “GESTURE RECOGNI TI ON USI NG M ATL AB ”. This project bears on imprint of many people. We sincerely thank our project guide Mr. RAJENDER YADAV, Assistant Professor, Department of Electronics & Communication, Gyan Ganga College of Technology, Jabalpur whose help, stimulating suggestions and encou ragement, helped to coordinate the project especially in writing this report. We would also like to acknowledge with much appreciation the crucial role of the officials and other staff members of the institute who rendered the ir help during the period of project work. Last but not the least we wish to avail this opportunity to appreciate and give a special thanks to every team member for their supportive contribution, project enhancing comments and tips that improved the presentation skills, report writing and brought about clarity in the software work.
Place: Jabalpur
Date:
TABLE OF CONTENTS
Index
Page No.
Chapter 1:- Introduction to Hand Gesture Recognition
1.1 1.2 1.3
Introduction……………………………………………... Motivation………………………………………………. Gesture Analysis………………………………………….
Chapter 2:- Objectives & Tools
1 1 1 1 3
2.1
Introduction………………………………………………
3
2.2
Objectives………………………………………………...
3
2.3
Tools……………………………………………………..
3
Chapter 3:- Literature Review & Algorithm
4
3.1
MATLAB Overview……………………………………
4
3.2
Literature Review on Gesture Recognition……………..
8
3.3
Neural Networks………………………………………...
9
3.4
Neuron Model…………………………………………... 10
3.5
Perceptron……………………………………………….. 11
3.6
Image Database………………………………………….. 12
3.7
Orientation Histogram…………………………………… 13
3.8
Operation…………………………………………………. 15
3.9
Algorithm………………………………………………… 17
Chapter 4:- Results & Discussion
19
APPENDIX I: - Commands
21
APPENDIX II: - Coding
23
References
28
CHAPTER 1 INTRODUCTION TO HAND GESTURE RECOGNITION
1.1 INTRODUCTION: This project is to create a method to recognize hand gestures, based on a pattern recognition technique developed by McConnell ; employing histograms of local orientation. The orientation histogram will be used as a feature vector for gesture classification and interpolation. Computer recognition of hand gestures may provide a more natural-computer interface. Hand gesture recognition is an important area o f computer vision and pattern recognition field. Gestures are the way by which one can communicate non-verbally. Gesture recognition is a field, in which there is large number of innovations. Gestures can be defined as a physical action, which is used to convey the information. There are various input – output devices for interacting with the computer, but now days emphasis is given ,how to make human – computer interaction more easy going, and for that purpose hand gesture recognition comes in light. Hand can be used as an input device, by making its gesture understandable to computer, and for this purpose, this project aims at recognizing the various hand gestures.
1.2 MOTIVATION: Hand gesture recognition is done in this project by aiming basic shapes made by hand. Communication in our daily life is generally vocal, but body language has its own significance, like hand gestures, facial expressions and sometimes they play an important role in conveying the information. Hand gesture would be an ideal option for expressing the feelings, or in order to convey something, like representing a number. It has many areas of application like sign languages are used for various purposes and in case of people who are deaf and dumb, sign language plays an important role. Gestures are the very first form of communication. So this area influenced us very much to carry on the further work related to hand gesture recognition.
1.3 GESTURE ANALYSIS: Gestures are defined as a physical activity that conveys some message, whether it can be facial expressions, body language, hand movements etc. Gesture can be defined as the motion of the body in order to communicate with others. 1
Hand gestures can be classified in two categories: static and dynamic . A static gesture is a particular hand configuration and pose, represented by a single image. A dynamic gesture is a moving gesture, represented by a sequence of images. We will focus on the recognition of static images . In gesture analysis, there are mainly three steps to follow
Hand localization
Hand feature extraction
Hand model parameter computation of features
There are two types of classification
Rule Based Classification
Learning Based Classification
We have our project based on LE ARNIN G BASED CLA SSI FI CATION . Under this classification, H idden approach has been used. An HMM consists of a number of hidden states, each with a M arkov M odels (HM M ) probability of transitioning from itself to another state. The transitioning probabilities are modeled as nth order Markov processes. An important feature of the topolog y is that states are allowed to transition themselves. Alternatively, models based on finite state machines have been used to capture the sequential nature of gestures by requiring a series of states estimated from visual data to match in order, to a learned model of ordered states. The advantage of this representation over HMMs is that it does not require a large set of data in order to train the models. Finally, temporal extensions to neural networks, (time-delay neural networks), have been used to learn mappings between training exemplars (2D or 3D features) and gestures. Also, much care must be taken during the training stage; otherwise the network may over fit on the training gesture set and not generalizes well to variations of gestures outside the training set.
2
CHAPTER 2 OBJECTIVES & TOOLS
2.1 INTRODUCTION: This project on “Gesture Recognition Using MATLAB” emphasizes o n easy & swift communication using minimum tools which can be easily accessible. The main objectives & tools used for this project are discussed in the nest sections.
2.2 OBJECTIVES: The main objectives of our project are:
Response time should be very fast.
Computer vision algorithms should be reliable and work for different people.
The vision-based interfaces will be replacing existing ones, which are often of low cost.
User-friendlier man-machine interaction.
Reduce time consumption.
Reduce error scope.
Easy operation for operator of the system.
2.3 TOOLS: This project is developed using the following tools: HARDWARE:
Processor Memory Processor Speed RAM memory
: Intel Core i3 : 3 GB : 2.30GHz : 2.00 GB
SOFTWARE:
Windows XP/7 MATLAB 7.01 3
CHAPTER 3 LITERATURE REVIEW & ALGORITHM
3.1 MATLAB Overview The name MATLAB stands for matrix laborator y. MATLAB is a high-performance language for technical computing. It integrates computation, visualization, and programming in an easy-to-use environment where problems and solutions are expressed in familiar mathematical notation. Typical uses include:
Math and computation Algorithm development Modeling, simulation, and prototyping Data analysis, exploration, and visualization
Scientific and engineering graphics
Application development, including Graphical User Interface building
MATLAB is an interactive system whose basic data element is an array that does not require dimensioning. This allows solving many technical computing problems, especially those with matrix and vector formulations, in a fraction of the time it would take to write a program in a scalar non-interactive language such as C or FORTRAN. The reason we have decided to use MATLAB for the development of this project is its toolboxes. Toolboxes allow learning and applying specialized technology. Toolboxes are comprehensive collections of MATLAB functions (M-files) that extend the MATLAB environment to solve particular classes of problems. It includes among others image processing and neural networks toolboxes. In 2004, MATLAB had around one million users across industry and academia. MATLAB users come from various backgrounds of engineering, science, and economics. MATLAB is widely used in academic and research institutions as well as industrial enterprises.
4
GRAPHICS AND GRAPHICAL USER INTERFACE PROGRAMMING
MATLAB supports developing applications with graphical user interface features. MATLAB includes GUIDE (GUI development environment) for graphically designing GUIs. It also has tightly integrated graph plotting features. For example the function plot can be used to produce a graph from two vectors x and y. The code: x = 0:pi/100:2*pi; y = sin(x); plot(x,y) produces the following figure of the sin e function :
Figure 1: sine function
A MATLAB program can produce three-dimensional graphics using the functions surf , plot3 or mesh. [X,Y] = meshgrid(-10:0.25:10,-10:0.25:10); f = sinc(sqrt((X/pi).^2+(Y/pi).^2)); mesh(X,Y,f); axis([-10 10 -10 10 -0.3 1]) xlabel('{\bfx}') ylabel('{\bfy}') zlabel('{\bfsinc} ({\bfR})') hidden off 5
This code produces a wireframe 3D plot of the two-dimensional unnormalize sinc function:
Figure 2: Wireframe 3D plot of 2D sinc function
[X,Y] = meshgrid(-10:0.25:10,-10:0.25:10); f = sinc(sqrt((X/pi).^2+(Y/pi).^2)); surf(X,Y,f); axis([-10 10 -10 10 -0.3 1]) xlabel('{\bfx}') ylabel('{\bfy}') zlabel('{\bfsinc} ({\bfR})')
This code produces a surface 3D plot of the two-dimensional unnormalized sinc function: 6
Figure 3: Surface 3D plot of 2D sinc function
In MATLAB, graphical user interfaces can be programmed with the GUI design environment (GU IDE) tool.
INTERFACING WITH OTHER LANGUAGES:
MATLAB can call functions and subroutines written in the C programming language or FORTRAN. A wrapper function is created allowing MATLAB data types to be passed and returned. The dynamically loadable object files created by compiling such functions are termed "MEX-files" (for MATLAB executable). Libraries written in Perl, Java, ActiveX or .NET can be directly called from MATLAB, and many MATLAB libraries (for example XML or SQL support) are implemented as wrappers around Java or ActiveX libraries. Calling MATLAB from Java is more complicated, but can be done with a MATLAB toolbox which is sold separately by MathWorks, or using an undocumented mechanism called JMI (Java-to-MATLAB Interface), (which should not be confused with the unrelated Java Metadata Interface that is also called JMI). As alternatives to the MuPAD based Symbolic Math Toolbox available from MathWorks, MATLAB can be connected to Maple or Mathematica. Libraries also exist to import and export MathML. 7
3.2
Literature Review on Gesture Recognition:
Research on hand gestures can be classified into three categories.
The first category, glove based analysis, employs sensors (mechanical or optical) attached to a glove that transduce finger flexions into electrical signals for determining the hand posture.
The second category, vision based analysis, is based on the way human beings perceive information about their surroundings.
The third category, analysis of drawing gestures, usually involves the use of a stylus as an input device. Analysis of drawing gestures can also lead to recognition of written text.
Our project is based on anal ysis of drawi ng gestur es . OBJECT RECOGNITION: It is classified into two categories –
OBJECT RECOGNITION
LARGE OBJECT
SHAPE
TRACKING
RECOGNITION
Figure 4
LARGE OBJECT TRACKING -
The large-object-tracking method makes use of a low-cost
detector/processor to quickly calculate moments. This is called the artificial retina chip. This chip combines image detection with some low-level image processing. The chip can compute various functions useful in the fast algorithms for interactive graphics applications. 8
SHAPE RECOGNITION - If the hand signals fell in a predetermined set, and the camera views a close-up of
the hand, example-based approach may be used, combined with a simple method top analyze hand signals called orientation histograms. These example-based applications involve two phases; training and running . In the training phase, the user shows the system one or more examples of a specific hand shape. The computer forms and stores the corresponding orientation histograms. In the run phase, the computer compares the orientation histogram of the current image with each of the stored templates and selects the category of the closest match, or interpolates between templates, as appropriate. This method should be robust against small differences in the size of the hand but probably would be sensitive to changes in hand orientation.
3.3 NEURAL NETWORKS: Neural networks are composed of simple elements operating in parallel. Neural networks are models that are capable of machine learning and pattern recognition. They are usually presented as systems of interconnected “neurons” that can compute values from inputs by feeding information through the network. Commonly neural networks are adjusted, or trained, so that a particular input leads to a specific target output. There, the network is adjusted, based on a comparison of the output and the target, until the network output matches the target. Typically many such input/target pairs are used, i n this supervised learning, to train a network. Neural networks have been trained to perform complex functions in various fields of application including pattern recognition, identification, classification, speech and vision and control systems.
Figure 5: Neural Net block diagram
9
There are two modes of learning: Super vised an d u nsupervised .
SUPERVISED LEARNING: Supervised learning is based on the system trying to predict outcomes for
known examples and is a commonly used for training method. It compares its predictions to the target answer and "learns" from its mistakes. The data start as inputs to the input layer neurons. The neurons pass the inputs along to the next nodes. As inputs are passed along, the weighting, or connection, is applied and when the inputs reach the next node, the weightings are summed and either intensified or weakened. This continues until the data reach the output layer where the model predicts an outcome. In a supervised learning system, the predicted output is compared to the actual output for that case. If the predicted output is equal to the actual output, no change is made to the weights in the system. But, if the predicted output is higher or lower than the actual outcome in the data, the error is propagated back through the system and the weights are adjusted accordingly. This feeding error backwards through the network is called "back-propagation."
UNSUPERVISED LEARNING : Neural networks which use unsupervised learning are most effective for
describing data rather than predicting it. The neural network is not shown any outputs or answers as part of the training process--in fact, there is no concept of output fields in this type of system. The advantage of the neural network for this type of analysis is that it requires no initial assumptions about what constitutes a group or how many groups are there. The system starts with a clean slate and is not biased about which factors should be most important.
3.4 NEURON MODEL:
A class of statistical models will be called “neuron” if they:
Consist of sets of adaptive weights (numerical parameters that are tuned b y a learning algorithm
Are capable of approximating non-linear functions of their inputs
10
Figure 6: Neuron
The scalar input p is transmitted through a connection that multiplies its strength by the scalar weight w, to form the product w p, again a scalar. Here the weighted input wp is the only argument of the transfer function f, which produces the scalar output a. The neuron on the right has a scalar bias, b. The bias is much like a weight, except that it has a constant input of 1. The transfer function net input n, again a scalar, is the sum of the weighted input wp and the bias b. This sum is the argument of the transfer function f. Here f is a transfer function, typically a step function or a sigmoid function, that takes the argument n and produces the output a. w and b are both adjustable scalar parameters of the neuron. The central idea of neural networks is that such parameters can be adjusted so that the network exhibits some desired or interesting behavior. Thus, the network can be trained to do a particular job by adjusting the weight or bias parameters, or perhaps the network itself will adjust these parameters to achieve some desired end. All of the neurons in the program written in MATLAB have a bias.. A
3.5 PERCEPTRON:
The perceptron is a program that learns concepts , i.e. it can learn to respond with True (1) or False (0) for inputs presented to it, by repeatedly "studying" examples presented to it. The structure of a single perceptron is very simple. There are two inputs, a bias, and an output. Both the inputs and outputs of a perceptron are binary - that is they can only be 0 or 1. Each of the inputs and the bias is connected to the main perceptron by a weight. A weight is generally a real number between 0 and 1. When the input number is fed into the perceptron, it is multiplied by the corresponding weight. After this, the weights are all summed up and fed through a hard-limiter. Basically, a hard-limiter is a function that defines the threshold values for 'firing' the perceptron. For example, the limiter could be: 11
For example - If the sum of the input multiplied by the weights is -2, the limiting function would return 0. Or
if the sum was 3, the function would return 1.
Figure 7: Perceptron block diagram
3.6 IMAGE DATABASE: The starting point of the project was the creation of a database with all the images that would be used for training and testing. The image database can have different formats. Images can be either hand drawn, digitized photographs or a 3D dimensional hand. Photographs were used, as they are the most realistic approach. Images came from two main sources. Various ASL databases on the Internet and Photographs took with a digital camera. This meant that they have different sizes, different resolutions and sometimes almost completely different angles of shooting. Two operations were carried out in all of the images. They were converted to grayscale and the background was made uniform. The internet databases already had uniform backgrounds. Camera had to be processed in Adobe Photoshop. Drawn images can still simulate translational variances with the help of an editing program (e.g. Adobe Photoshop). The database itself was constantly changing throughout the completion of the project as that would decide the robustness of the algorithm. Therefore, it had to be done in such way that different situations could be tested and thresholds above which the algorithm didn‟t classify correct would be decided. 12
An example is shown below. In the first row are the training images. In the second , the testing images.
Train image 1
Test Image 1
Train image 2
Train image 3
Test Image 2
Test Image 3
Figure 8
3.7 ORIENTATION HISTOGRAMS: To have the gestures same regardless of where they occur with the images boarders, position is ignored altogether, and a histogram is tabulated of how often each orientation element occurred in the image. Clearly, this throws out information and some distinct images will be confused by their orientation histograms. In practice, however, one can choose a set of training gestures with substantially different orientation histograms from each other. One can calculate the local orientation using image gradients. In this project two 3 – tap x and y derivative filters have been used. The outputs of the x and y derivative operators will be dx and dy. Then the gradient direction is atan (dx, dy). The edge orientation is used as the only feature that will be presented to the neural network. The reason for this is that if the edge detector was good enough it would have allowed testing the network with images from different databases. 13
Another feature that could have been extracted from the image would be the gradient magnitude using the formula below –
√ Where, a= dx b= dy This would lead though to testing the algorithm with only similar images. Apart from this the images before resized should be of approximately the same size. This is the size of the hand itself in the canvas and not the size of the canvas. Once the image has been processed the output will be a single vector containing a number of elements equal to the number of bins of the orientation histogram. Figure shows the orientation histogram calculation for a simple image. Blurring can be used to allow neighboring orientations to sense each other.
Figure 9: Orientation histogram 14
3.8 OPERATION: The program can be „divided‟ in 6 steps.
Step1
The first thing for the program to do is to read the image database. A for loop is used to read an entire folder of images and store them in MATLAB‟s memory. The folder is selected by the user from menus. A menu will firstly pop-up asking whether to run the algorithm on test or train sets. Then, a second menu will pop-up for the user to choose which ASL sign he wants to use.
Step2
Resize all the images that were read in Step1 to 150x140 pixels. This size seems the optimal for offering enough detail while keeping the processing time low.
Step3.
Next thing to do is to find the edges. As mentioned before 2 filters were used. For the x direction x = [0 -1 1]
For the y direction y= which is the same as x but transposed and multiplied by – 1.
Step 4
Divide the two resulting matrices (images) dx and dy element by element and then take the atan ( tan−1 ). This will give the gradient orientation.
Step 5
Then the MATLAB function im2col is called to rearrange the image blocks into columns. This is not a necessary step but it has to be done to display the orientation histogram. Rose creates an angle histogram, which is a polar plot showing the distribution of values grouped according to their numeric range. Each group is shown as one bin. Below are some examples. While developing the algorithm those histograms are the fastest way of getting a good idea how good the detection is done. 15
Orientation histogram of a_1
Orientation histogram of a_2 Figure 10
Here we can see the original images that generated the histograms above in the same order –
Original image a_1
Original image a_2 Figure 11
16
3.9 Algorithm
17
1. Read Text Files from Disk:
This is the first step where we input image to the system. 2. Determine Number of Neurons & Targets:
For multidimensional arrays RGB image is being considered. To access a sub-image, the syntax is, subimage=RGB(20:40,50:80,:85); For optimal number determination of neurons, always the highest among the input and output is taken. So number of neurons determined are 85. Since we have taken 5 test images in our project, number of targets to be matched are 5. 3. Initialize Pre-Processing Layer:
Pre-processing of image is done for enhancement of image and also for getting results with minimum error. In the proposed algorithm the image is p re-processed using RGB color model. Three primary colors red(R), green (G), and blue (B) are used. The main advantage of this color space is its simplicity.. 4. Initialize Learning Layer: This is a method for initialization of weights of neural networks to reduce the training time. 5. Train Perceptron:
A perceptron learns to distinguish patterns through modifying its weights. In the perceptron, the most common form of learning is by adjusting the weights by the difference between the desired output and the actual output . 6. Plot Error: This step plots the graph of vector with error bars. The error is calculated by subtracting the output A from target T. 7. Select Test Set:
Test images are selected so that they can be matched with the trained images to obtain the desired output. 8. Display Output:
Finally, the output is displayed showing the similarity or difference between trained & test images about their orientation histogram.
18
CHAPTER 4 RESULTS & DISCUSSION 4.1 RESULT:
4.2 DISCUSSION:
CONCLUSION:
We proposed a simple hand gesture recognition algorithm, followed by various steps like pre-processing, image converted into RGB , so that varying lightening conditions will not cause any problem. Then smudge elimination is done in order to get the finest image. These pre-processing steps are as important as any other step. After performing the pre-processing on the image, the second step is to determine the orientation of the image, only horizontal and vertical orientation is considered here and images with uniform background is taken. The strength of this approach includes its simplicity, ease of implementation, and it does not required any significant amount of training or post processing as rule based learning is used. It provides the higher recognition rate with minimum computational time. The weakness of this method is that certain parameters and threshold values are taken experimentally that is it does not follow any systematic approach for gesture recognition, and many parameters taken in this algorithm are based on assumption made after testing number of images. In this system we have only considered the static gesture, but in real time we need to extract the gesture form the video or moving scene. To realize the ultimate goal of humans interfacing with machines on their own natural terms gestures are expressive, meaningful body motions involving physical movements of the fingers, hands, arms, head, face, or body with the intent of: 1) Conveying meaningful information interacting with the environment 2) Gesture recognition is an extensively developed technology available designed to identify human position, action, and manipulation. Gestures are used to facilitate communication with digital applications. 3)Among the various ways of gesture recognition like Hand, Face and Body Gesture Recognition, Hand Gesture Recognition is efficient technique to o recognize human gestures due to its simple and greater accuracy features. 19
FUTURE SCOPE:
The future scope lies in making this algorithm applicable for various orientations of hand gestures, also different classification scheme can be applied. Gesture recognition co uld be used in many settings in the future.
The algorithm can be improved so that images with non uniform background can also be used, this will enhance the human computer interaction.
Visually impaired people can make use of hand gestures for human computer interaction like controlling television, in games and also in gesture to speech conversion.
Georgia Institute of Technology researchers have created the Gesture Panel System to replace traditional vehicle dashboard controls. Drivers would change, for example, the temperature or soundsystem volume by maneuvering their hand in various ways over a designated area. This could increase safety by eliminating driver‟s current need to take their eyes off the road to search for controls.
During the next few years, according to Gartner's Fenn, gesture recognition will probably be used primarily in niche applications because making mainstream applications work with the technology will take more effort than it's worth.
Hand recognition system can be useful in many fields like robotics, computer human interaction and so make hand gesture recognition offline system for real time will be future work to do.
Support Vector Machine can be modified for reduction of complexity. Reduced complexity provides us less computation time so we can make system to work real time.
Facial Gesture Recognition Method could be used in vehicles to alert drivers who are about to fall asleep.
20
APPENDIX I COMMANDS: 1. echo on: The commands in a script M-file will not automatically be displayed in the Command Window. To display the commands along with results we use echo. 2. clc: Clear command window clears all input and output from the command window display, giving a clear screen. 3. pause: Each time MATLAB reaches a pause statement, it stops executing the M-file until the user presses a key. Pauses should be placed after important comments, after each graph, and after critical points where your script generates numerical outputs. Pauses allow viewer to read and understand results. 4. fid: file ID=> An integer file identifier obtained from fopen. 5. fopen: Opens file. 6. f id=fopen(‘train.txt’,’rt’) : Opens the file with the type of access specified by permission. „rt‟ means Read Text (t-text mode). 7. fscanf (file scan format): Read data from device, and format as text. 8. P1=fscanf(fid,’%f’,[19,inf]); : A=fscanf(str,’format’,sizeA) Read data from string and converts it according to format. Format is a C language conversion specification. Conversion specification involves the % character and the conversion characters d, i, o, u, x, X, f, e, E, g, G, c and s. Size A can be integer or can have the form [m,n]. [m,n]: m by n matrix filled in column order, n can be inf , but m cannot. n: Column vector with n elements. inf: Column vector with the number of elements in the file (default). str: Character string. 21
%f: It denotes floating point numbers. Floating point fields can obtain any of the following (inf, -inf, NaN or NaN). [19]: Neural network interface. (Face segmentation using a Gaussian mixture model of skin or deformable models of body / face parts) 9. TS1: Test vector. 10. T: Target vector. 11. T=fscanf(fid,’%f’,[8,inf]); : [8] means 8 bytes (since set A is always double). 12. Determine optimal number of neurons: For multidimensional arrays RGB image is being considered. subimage=RGB(20:40,50:80,:85); For optimal number determination of neurons, always the highest among the input and output is taken. So we have, S1=85 13. initp: This command initializes weights and biases. 14.Pre-processing: Image pre-processing is the very first step, pre-processing of image is done for enhancement of image and also for getting results with minimum error. Reduction in the dimensionality of the input raw data is probably the most important reason for pre-processing. 15. A1=simup(P,W1,b1); : The network takes P as the input data processes and transforms it to A1. A1 is 85x5 matrix. A1 will then be the input to the hidden layer that will train the network. A data sheet of 5 imagesafter the first layer pre-processing contains 85x5=425 elements. 16. simup: It simulates perception/preprocessing layer. 17. TP=[1 500]; : Creates vector of 500 equally spaced frequencies. 18. clf reset: Clears current figure and resets the window. 19. figure(gcf): Sets the units of current figure. gcf: get current figure. 20. epochs: Presentation of the set of training (input/target) vectors to a network and the calculation of new weights and biases. 21. ploterr (errors): Plots the graph of vector with error bars. The error is calculated by subtracting the output A from target T. 22. close(W): Closes the waiting bar. 22
APPENDIX II CODING:
echo on clc pause clc %Store the training information in a test file
fid=fopen(„train.txt‟,‟rt‟); P1=fscanf(fid,‟%f‟,[19,inf]); P=P1; %%Open some text file using code to writ e and fetch the required information about image fid=fopen(„testA.txt‟,‟rt‟); TS1=fscanf(fid,‟%f‟,[19,inf]); %(As here we are only testing alphabet ‘A’ ) Fid=fopen(„target8.txt‟,‟rt‟); T=fscanf(fid,‟%f‟,[8,inf]); %%It has been found that the optimal number of neurons for the hidden layer is 85 S1=85; S2=5; %%Now we have to initialize pre-processing layer [W1,b1]=initp(P,S1); %%We also have to initialize learning layer [W2,b2]=initp(S1,T); Pause %%Now train the network 23
A1=simup(P,W1,b1);
%First layer is used to preprocess the input vectors
TP=[1:500]; pause clf reset figure(gcf) %%Resize the frame size Setfsize(600,300); [W2,b2,epochs,errors]=trainp(W2,b2,A1,T,TP); pause clc ploterr(errors); pause M=MENU(„Choose a file resolution,‟Test A‟); If M==1 TS=TS1; else disp(„Wrong Input’); a1=simup(TS,W1,b1); a2=simup(a1,W2,b2); echo off %%Create a Menu clc F=MENU(„Choose a database set,‟Test Set‟,‟Train Set‟); If F==1 K=MENU(„Choose a file,‟Test A‟); %%For testing a Datasheet 24
If K==1 loop=5 for i=1:loop string=[„test\A\‟num2str(i)‟.tif‟]; Rimages{i}=imread(string); end end end end; %%For training If F==2 loop=3
%%Set loop to 3 considering all train sets have 3 images
L=MENU(„Choose a file‟,‟Train A‟); If L==1 for i=1:loop string=[„train\A\‟num2str(i)‟.tif‟]; Rimages{i}=imread(string); end end end T{i}=imresize(Timages{i},[150,140]); x=[0 -1 1];
%x-derivative filter
y=[0 1 -1];
%y-derivative filter
dx{i}=convn(T{i},x,‟same‟); dy{i}=convn(T{i},y,‟same‟); gradient{i}=dy{i}./dx{i}; 25
theta{i}=atan(gradient{i}); cl{i}=im2col(theta{i},[1,1],‟distinct‟); N{i}=(cl{i}*180)/3.14159265359; C1{i}=(N{i}>0)&(N{i}<10); S1{i}=sum(C1{i}); C2{i}=(N{i}>10.0001)&(N{i}<20); S2{i}=sum(C2{i}); C3{i}=(N{i}>20.0001)&(N{i}<30); S3{i}=sum(C3{i}); C4{i}=(N{i}>30.0001)&(N{i}<40); S4{i}=sum(C4{i}); C5{i}=(N{i}>40.0001)&(N{i}<50); S5{i}=sum(C5{i}); C6{i }=(N{i}>50.0001)&(N{i}<60); S6{i}=sum(C6{i}); C7{i }=(N{i}>60.0001)&(N{i}<70); S7{i}=sum(C7{i}); C8{i }=(N{i}>70.0001)&(N{i}<80); S8{i}=sum(C8{i}); C9{i }=(N{i}>80.0001)&(N{i}<90); S9{i}=sum(C9{i}); C10{i }=(N{i}>90.0001)&(N{i}<100); S10{i}=sum(C10{i}); C11{i }=(N{i}>-89.9)&(N{i}<-80); S11{i}=sum(C11{i}); C12{i }=(N{i}>-80.0001)&(N{i}<-70); 26
S12{i}=sum(C12{i}); C13{i }=(N{i}>-70.0001)&(N{i}<-60); S13{i}=sum(C13{i}); C14{i }=(N{i}>-60.0001)&(N{i}<-50); S14{i}=sum(C14{i}); C15{i }=(N{i}>-50.0001)&(N{i}<-40); S15{i}=sum(C15{i}); C16{i }=(N{i}>-40.0001)&(N{i}<-30); S16{i}=sum(C16{i}); C17{i }=(N{i}>-30.0001)&(N{i}<-20); S17{i}=sum(C17{i}); C18{i }=(N{i}>-20.0001)&(N{i}<-10); S18{i}=sum(C18{i}); C19{i }=(N{i}>-10.0001)&(N{i}<-0.001); S19{i}=sum(C19{i}); D{i}=[S1{i} S2{i} S3{i} S4{i} S5{i} S6{i} S7{i} S8{i} S9{i} S10{i} S11{i} S12{i} S13{i} S14{i} S15{i} S16{i} S17{i} S18{i} S19{i}]; close(W);
27
