2015 International Conference on Pervasive Computing (ICPC)

Human hand tracking using MATLAB to control

Arduino based robotic arm

Pankaj S Lengare Milind E Rane

Department of Electronics and Telecommunication Department of Electronics and Telecommunication
Vishwkarma Institute of Technology, Vishwkarma Institute of Technology,
Pune, India Pune, India
Pankaj.lengare13@vit.edu Me_rane@yahoo.com

Abstract— Hand tracking system gets so much attention in in any direction. This paper is very important since it gives an
recent time because of its great applications. This method can be intuitive way to develop human centered forms of human
implemented by so many techniques. Here we show a straight
forward technique of tracking the human hand using the robotic machine interaction. At the same time it is difficult to control,
arm. This paper shows the interfacing of human hand using since it needs different identification schemes of hands such as
robot arm. With this method the robotic hand can be controlled pattern recognition, tracking, color identification and giving X
using human hand. Its demonstration is done by using image
and Y axis to the points [3].
processing technique to detect different colors at different axis of
human hand. This technique is very useful since it takes real time Different techniques are used to interact with humans and the
video of hand and tracks it to get interface with robotic arm. A robot i.e. by using a PIC micro-controller in which a C-MOS
laptop camera will get the video and track the RGB (red, green camera [4] is used to determine the position of the object.
and blue) colures at different axis of hand denoting X and Y axis.
Tracking of such hand will interface the controller with robotic
Recently many robotic arm is used in industries which will
arm. The main aim behind this approach to program a robotic work in accordance to the program, such as lifting the object,
arm, so that it should be controlled by human hand and will placing the object, drilling etc. but these robots are trained, any
reach the locations where human will not be able to reach and do change in the working of robots will change the program, but
the given task by direct interfacing with human hand. In this we
can see the real time movement of robotic arm. with this idea we can function whatever we want to do without
much change.
In this study we are implementing it using MATLAB to track
Keywords—Color coding, human hand tracking, interfacing, the human arm using different X and Y axis to control the
controller, robotic arm. robotic arm. For each position of hand there is a color to detect
its positions that is from shoulder to elbow the color let us say
Green, from elbow to wrist red color and then ahead of wrist a
I. INTRODUCTION blue color is used. These colors will be detected by MATLAB
and gives position to each and every color depending upon the
The main aim of making human hand tracking system is to axis defined for the video screen. These axis locations will be
create interaction between human hand and robotic arm. Man – captured and interfaced with the controller. The controller will
machine interaction gives the relation between human and gives the command to the robot that, at movement of particular
computer [1], [2]. The idea shows the creation of a robotic arm Location the particular robotic servo motor will move which is
which is as similar to the human arm as possible and not to placed in the axis of robotic arm.
limit the arm to one set of task. Also, the arm is to be
controlled using Image Processing technique. Similarity of
human hand with the robotic arm is the basic aim of the project II. LITERATURE SURVEY
but, there is an underlying aim which is to expand the
functionality of the arm once the basic model is created like as At present there is enormous amount of work is done in order
example hammering a nail. to identify the motion of the hand. Lots of articles have been
This paper introduces the technology used to track the arm and used in surveying the detection for the motion of hand. The
interface it to control the robotic arm for different activities. different fields using this technique contain automatic hand
The motions can be controlled by the user by moving our hand language, automatic sketching, computer graphics and also

978-1-4799-6272-3/15/$31.00(c)2015 IEEE
industrial robots used to work instead of humans. This paper A. Tracking of colors
gives the most successful technique to use robotics.
Tracking of colours contain three types of colors RGB,
In these area two types of techniques [5], [6] are used:
assigning to the human hand as shown in figure 2(A) and
1) contact type
2(B). These colors are chosen because every image in this
2) non-contact type
world is made up of these three colours. The colours are
assigned in such a way that, shoulder to elbow will get one
1) Contact type: Contact type of devices contain data
color, elbow to wrist second and ahead of wrist the third one.
gloves, electromagnetic tracking system, exoskeleton etc.
For conversion of image from normal image to gray image
2) Non-contact type: This type contains vision based firstly we have to convert the 3D image to 2D image.
system, speech recognition, camera based etc.

Our technique comes under non-contact type since it uses a

camera to track the hand. Traditionally these systems were
implemented using CCTV cameras, based on computer vision
algorithm, using embedded system. [7] These techniques
mainly contain video analysis and are used for tracking the
people. Some non-contact techniques involve skin color
detection in which user will directly tracks the hand and
detects the hand. Whereas some uses notation techniques for
different positions of hand like as first finger denotes the
motion in forward position, second will denotes the backward
position etc.
Tremendous amount of work is done by the use of single
cameras. Nowadays different methods and techniques are
using to detect the by the use filters, suppurating the
background, using 3D cameras etc.
For the extraction of features for Hand tracking we are
using color coding technique which is based on the colors,
which is given on different positions of hand. Where the
movement of hand will shows the different axis at different
locations.
Hand tracking system is very useful in situation, where
human cannot reach or reaching of them may become
dangerous. Such places like as hazardous chemical reactions,
bomb diffusion, high power electrical connections.

III. ALGORITHM

The sequence wise procedure of human hand tracking for

controlling robotic arm is as shown in figure 1. FIGURE 1: Sequence wise procedure of human hand tracking
Input is taken as a video signal of the human hand and analyses for controlling robotic arm.
it to track the different colors stick on the hand at different
locations. Video signal may be taken by a laptop camera or a These colors are then assigns with their respective X and Y
camera build on robotic arm itself, which will be make it axis. This axis will be taken as the position of that colored
portable and efficient for handling. pixel from the zero position.

B. Angle positions
Angle position from the axis is calculated using the simple
trigonometric equations as shown in figure 3. These angles
will tell how much our hand is rotated and is in which
direction. The three axial positions give the rotation of arm in
any direction to give the direction for the robotic arm and to
perform the functions.

FIGURE 3: Calculation of angles by the use axis on the

colors.

IV. RESULT AND OBSERVATION

Tracking of human hand gives the tracking of different axis at

different location of human hand. Figure 4, shows the different
FIGURE 2(A): Detection of colors and its axis with straight axis location at different position of hand. As the axis location
hand position. will going to change the angle between the each joint of hand
will change. Depending upon these angles the robotic arm will
move.

FIGURE 2(B): Detection of colors and its axis with 90

degrees hand position.

Several methods has been used to trace the angles of human

hand like gesture recognition in which a particular shape of
palm or number of fingers has been shown to the camera, to FIGURE 4: Detection of X and Y axis at different positions of
track the angles and the voice recognition, whatever we speak hand.
will write in the MATLAB in the form of code and then it is
interfaced with the robotic arm to trace the angle [7]. By serially interfacing the angles recorded by the axis of hand
Depending on the axis location the angle can be trace by the will be send to the Arduino and then it controls the robotic
camera to give signal to robotic arm. For the movement of the arm in the real time.
robotic arm the angle signal is transmitted from MATLAB to Here we may get error during the recording of video but it can
the micro controller and process the signal. The processed be eliminated by taking white color background.
signal is the transmitted to the robotic arm to perform various
actions.
 V. CONCLUSION communication engineering(IOSR-JECE), ISSN: 22782834.ISBN:
2278-8735, PP :31-35.
3) Markossigalas, HarisBaitzakis and panostrahanias,”Foundation of
In this paper the implementation of human hand tracking using
research and technology” Hellas (FOURTH)
robotic arm is shown. The arm is controlled by the human 4) Yoshishigesato, “study of intelligent control of an arm robot
hand and it will listen all the instructions of the human hand. equipped with ac-mos camera using a new type of image
This technique can be used even if the user is dumb or deaf processing”, tsuruoka national college of technology, Yamagata
prefecture, PP: 997-8511.
that is it listen the instructions of hand only. This is a very
5) Euijungjung, Jachoon lee, Byungjuyi, Member ,IEEE, Jooyoung
easily implemented straight forward method to coupled man- Park, Shin’ichiyuta, fellow,IEEE, and Si tac nah, ”IEEE/ASME
machine interface. To control the robotic arm, the human hand transactionon mechatronics , “Development of a laser range finder
must be recorded in the camera in proper lightning conditions, based human tracking and control algorithm for a marathoner
service robot”, PP: 1083-4435 vol 19. No.6, December 2013
with a black color background. So that error will be less while
6) Guanglong Du, Ping Zhang, and Di li, “human manipulator
recording the video. To control it I have used the angle interface based on multisensory process via kalman filters”, IEEE
measuring technique. The angles are measured by the use of transaction on industrial electronics, PP: 0278-0046 vol.61,no.10,
simple trigonometric equations of sine, cosine, tan etc. October 2013.
7) Aishwaryapatil, AnillumarDharwal, Ankurharshe, Monish Gakare
To detect the hand we should not need any database for this
and Monika sajjanwar, “Haptic Robotic Arm Using Voice &
technique. Since this is a real time processing with this the Gesture Recognition, International Journal of Advanced Research
memory required for the storage of database will not require. I in Computer and Communication Engineering, Vol. 2, Issue 3,
have proposed a simple algorithm to track the human hand March 2013, PP:2278-1021
8) Jagdishlalraheja, Radheyshyam, G. ArunRajshekhar and P.
using RGB colors to detect the axis of the color depending
BhanuPrasad, “council of scientific & industrial research pilani,
upon these axes, the angle between the joints of the hand can rajasthan”
be recognized. Above figures shows the different algorithms 9) Sui bo,Dudong,Chenqiang,Hanxiangyu, Wang li and Zhang
to detect the hand and at the same time the robotic arm can be hua,”tsinghua science and technology ISSN 1007-0214 23/24, PP:
356362”, vol 9, no3, june2004
controlled.
10) Harsh Kumar Kaura,Vipulhonarao,SayaliPatil and
Pravishshetty,”international journal of advance research in
VI. FUTURE SCOPE
artificial intelligence”, vol. 2, no. 5, 2013
11) X.zabulist, H. baltzakist and A. argyrosit,“Vision based hand
This technique contain color detection technique, which need gesture recognition for human computer interaction”, institute of
more filtering options such that no other colors should be computer science, Foundation of research and technology.

present at the background of recording. This problem we have

to eliminate. This method can be used for one degree of
accuracy which can be improved up to 0.1degree or even less
than that.
For the controlling of the robotic arm it is connected to the
Arduino micro controller. Since it contain serial interfacing of
the robotic arm and the camera or our laptop. We have to
make serial interfacing program which will be interface the
laptop camera and the robotic arm according to the movement
of the human hand. Depending upon the angle we have taken
from the axis location of the colors the interfacing program
will interface the angle and the rotation of the servo motor.
This angle is send to the servo motor by serial interfacing and
the rotation of the servo motor by same degree will takes
place.

REFERENCES

1) RafiqulZaman Khan and Noor Adnan Ibraheem,”international

journal of artificial intelligence and applications (IJALA)”, vol.3,
no.4, july 2012.
2) Mr.Raskar D S, Mrs. Belagali P P, “An interactive robot control
using image processing method”, IOSR journal of electronics &