Real Time Finger Counting and Virtual Drawing Using Color Detection and Shape Recognition

Dr. Ghous Buksh Narejo

Department of Electronic Engineering NED University of Engineering and Technology Karachi, Pakistan ghousnarejo@gmail.com

Shahyan Pervez Bharucha

Department of Electronic Engineering NED University of Engineering and Technology Karachi, Pakistan Shahyan30@gmail.com

Abstract In this paper we have provided a cheap method for counting open fingers and virtual drawing in real time using color detection and shape detection algorithms. These algorithms dont require training and database. Our method for finger counting detects the circular markers having a specified color visible on the inner side of the fingers by using color detection and shape recognition operations on the real time captured image and then counts the markers visible. The count can be further used as an input for other medium using communication methods. Different schemes are also defined using the combination of colors and hands used to generate desired output signals. The method devised for virtual drawing detects the circular specific colored marker on a stylus or finger using color detection and shape recognition. The position of the marker is marked and then plotted using the points marked in the previous frames which results in the pattern tracking the markers movement. The development of both applications is done using Matlab as the developing environment using a normal A4-Tech webcam and run on Dell latitude D630 laptop with Intel Core2 Duo processor, Ram 2.00GB. KeywordsImage Processing; Color Detection; Counting; Virtual Drawing; Shape Detection Finger

interactive medium. The applications are developed using Matlab [1] and normal webcam on a dell latitude D630 laptop with a 2GHZ Intel Core2 Duo processor. The key features of this algorithm are that it doesnt require training and data base.

II.

Background

I.

Introduction

Real time color detection and shape recognition are the basic algorithms which have provided the basis for other complex algorithms [9] [13]. Most of the applications in color detection and shape recognition are derived employing these algorithms. These algorithms can be used individually or as a part of some other algorithms. This allows us to make an accurate detection with very low probability of background interference in the real world environment. The applications we are proposing are real-time finger tracking, counting and virtual drawing. Hands acts as a natural interaction device for everyday interaction with our environment therefore researchers are trying to use hands as an input medium for digital applications [15] .Many companies are now days using human machine interaction in their products such as Kinect [5] by Microsoft to be used for playing interactive games on Xbox 360, Eye- Toy[6] is a product by Sony used to play interactive games on play station 3, multi-touch devices like Microsoft Surface, iphone [7] and Samsung galaxy [8] series utilizes human touch as an

The finger tracking and counting [2] [10] plays a significant part in the field of human-machine interaction [19]. Hand is the most employed in gesture as shown in the survey [22] which can display nonverbal human messages which have a major role in our human society. Vision based [18] hand gesture recognition is performed either by 3d model based or appearance based [25]. This task has been achieved using hand detection techniques and applying convex hull effects on it [2]. Using the human hand as an input-device is the region of great interest after the first Virtual/Augmented Reality (VR/AR) application was developed [13]. Many hand gesture input devices have also been developed in these years such as Cyber Glove 2 [23] and IGS-190 [24]. Cyber Glove 2 [23] utilized proprietary resistive bend-sensing technology to accurately transform hand and finger motions into real-time digital joint-angle data. Ultrasonic sensors with other sensors are also used to recognize gestures by computing propagation, reection, speed of time and triangulation [25]. The system used for detecting hands and fingertip in [14] utilized infrared cameras to segment skin region from background pixels using a template matching approach was unable to provide precise fingertip position data. In [15] hand tracking is done using a multicolored glove which covered the whole hand with color patterns using two Point Grey Research Dragony camera with a 4mm lens that captures 640 480 video at 30 Hz. Various challenges [22] regarding gesture recognition are faced such as cost, speed, reliability and the presence of intuitiveness in hand gestures. In [16] Marker based motion capture required reflective markers or LEDs and high cost cameras. The system provided accuracy at high system cost. In [20] more than 100 research papers on the topic of HCI (Human Computer Interaction) were reviewed and suggested the use of many sources of gesture to be incorporated

cohesively for effective HCI. Summarized review of 130 research papers have been given in [21] which is discusses techniques such as tracking, pose estimation and recognition. The virtual drawing is the method of writing or drawing objects without touching any input devices. This application gives an opportunity to the person to express his feelings without using any hard medium. It can help in generating a better flow in human writing. Virtual drawing [4] has picked fame recently by the introduction of high tech input devices like the Kinect [5]. [11] Does hand tracking for vision based drawing using Kalman filtering least squares fingertip fits but it requires a solid surface to be drawn. In [12] virtual drawing is being performed without need of any solid background but the drawing itself is not done by tracking the fingers instead it locates hand position as a human's upper arm and lower arm have consistent length and approximate locations of finger and elbow form two spheres with constant radii. Then from the previously tracked body points spheres are constructed computing the 3D position of the elbow and fingers and are plotted together to form lines III.

then replacing all pixels in the groups with the median of the respective groups. Step 4: Image Conversion Then we convert the filtered image into binary image at a specific threshold found by experimentation. This increases the accuracy and the speed of the system by giving is it less and appropriate information to process. Step 5: Binary Area Operation All the pixel groups fewer than 300 pixels are removed by merging them with their neighboring groups so the unwanted blue components are irradiated. Step 6: Image Labeling Group pixel components are then being labeled so they can be individually processed for required properties extraction. Step 7: Information Extraction The bounding box, eccentricity and the centroid information of each of the labeled group pixel component are calculated. The visual marking and further processing can be easily done. Step 8: Circle Detection Image eccentricity is multiplied by 100 for better range and then compared with a fix value so that is if the value of eccentricity is less than 90 then it is considered as a circle else the group is ignored. Step 9: Visual Marking Visual marking is done for visual aid on the selected groups using the previous calculated coordinate and marked on the original RGB image. Step 10: Finger Counting The number of pixel groups is counted giving an interpretation of number of fingers kept open. Step 11: Display The finger count is given as the output on screen which can be used for further application using serial or parallel communication.

Real Time Finger Counting

A. Methadology
The task is done using color detection as its base. The experimental setup requires a webcam, pc and circular shape color markers. The markers are placed ingeniously in top center of each finger so that the closing and opening action of the fingers can be interpreted accurately. When a finger is open the marker is visible and gets detected by color detection algorithm [3]. The total number of detected markers gives the finger count. When a finger is closed the marker gets hidden inside the palm and there is no detection. The markers can also be round shaped color stickers which temporarily stick or custom made gloves having the circular color markers printed on the inner finger sections.

B. Algorithm
It detects and count blue color markers visible on the finger tips as under. The algorithm contains the following detailed steps: Step 1: Image Frame Capture The image is capture via a standard A4 tech webcam. The webcam keeps on functioning until 200 frames are captured because of the memory buffer capacity. Step 2: Color Subtraction The specified blue components are subtracted from the gray scale component of the captured image. This forms the basis of color detection by giving us an image containing only blue color components. Step 3: Filtering Median filter is applied on the blue components. Median filter is used to filter unwanted noise components found in nature e.g. salt and pepper noise it works on dividing the image into 3-by-3 group pixels. It works by taking median of every pixel group and

C. Usage Scheme
This system can be used as a sub system for many hardware and software systems for control signal generation. It can generate five control signals if used as single hand scheme and can generate ten and twenty-five signals if used as single color pair hands scheme and dual color pair hand scheme respectively. 1) Single Hand Scheme In single hand scheme the user only uses single hand and single color for the markers are used. Maximum five control signals can be generated using this scheme. Single hand scheme is comparatively more accurate and fast than pair hand scheme as the less processing power is required.

Initialization A Image Frame Acquisition Image Frame Subtraction Median Filtering Image Binarization Binary Area Open Image Binarization Pixel Group Labeling Pixel Group Properties Extraction Eccentricity of Pixel Group * 100 Yes Give Finger Count

Initialization

Image Frame Acquisition Image Frame Subtraction Median Filtering Image Binarization Binary Area Open Image Binarization Pixel Group Labeling Pixel Group Properties Extraction Eccentricity of Pixel Group * 100

Is Eccentricity of Pixel Group < 90?

No

Is Eccentricity of Pixel Group < 90?

No

Discard Group Yes Keep Pixel Group Finger Count = 0 Pixel Group Visual Marking Finger Count +=1 Are There No Pixel Group? Yes Keep Pixel Group

Discard Group

Yes

Incorrect Position

No Mark Centroid Point

Is Every Pixel Groups Marked?

No Join With Centroid Point Found In Previous Frame

A 1a 1b

Fig.1. The Flow Charts. (a) Finger counting (b) Virtual Drawing

2)

Single Color Pair Hands Scheme

In single color pair hands scheme both hands are given the same color for the markers. We can use the counting technique to generate 10 signals. 3) Dual Color Pair Hands Scheme In dual color pair hands scheme both hands are given the different colors for the markers. We can use the counting technique to generate 35 signals that is 5 signals by are generated for individual hands and 25 signals are generated when both hands are used. Dual color pair hand scheme though generates comparatively higher amount of control signals but its also comparatively slow and inaccurate than the above two methods as more processing power is required.

IV.

Real Time Virtual Drawing

A. Methadology The setup requires a webcam, pc and color marker circular in shape which is placed ingeniously on the index finger so that the finger can be used as a pointer for drawing and by closing the finger the pen mode can be deactivated. When the marker is visible it gets detected by using color and shape detection and other elements of the same color are ignored. The markers can be of round shape color stickers which temporarily stick on the fingers or a custom made gloves. The circular color marker is printed on its inner finger section or a stylus can be used with a circular tip. B. Algorithm The algorithm contains of the following steps with details. It is used to detect the blue round marker and to draw with respect to its movement. Step 1: Image Frame Capture The image is capture via a standard A4 tech webcam. The webcam keeps on functioning until 200 frames are captured because of the memory buffer capacity. Step 2: Color Subtraction The specified blue components are subtracted from the gray scale component of the captured image. This forms the basis of color detection by giving us an image containing only blue color components. Step 3: Filtering Median filter is applied on the blue components. Median filter is used to filter unwanted noise components found in nature e.g. salt and pepper noise it works on dividing the image into 3-by-3 group pixels and then taking median of every pixel groups and then replacing all pixels in the groups with the median of the respective groups. Step 4: Image Conversion Then we convert the filtered image into binary image at a specific threshold found by experimentation. This increases the accuracy and the speed of the system by giving is less and appropriate information to process. Step 5: Binary Area Operation The pixel groups fewer than 300 pixels are removed by merging them with their neighboring groups so the unwanted blue components are irradiated. Step 6: Image Labeling Group pixel components are then being labeled so they can be individually be processed for required properties extraction.

2(a)

2(b)

2(c) Fig. 2. The usage schemes. (a)Single hand scheme. (b) single color pair hand scheme. (c)Dual color pair hand scheme.

Step 7: Information n Extraction E an nd the centroid d information of o each labeled d The Eccentricity group p pixel compo onent are calcu ulated so the visual v marking g and further fu processi ing can be don ne. Step 8: Circle Detec ction Eccen ntricity is mul ltiplied by 100 for better range r and then n comp pared with a fix x value that is if the value of f eccentricity is s less than t 90 then it ts considered as a circle els se the group is s ignor red. Step 9: Plotting The point p location is marked on n the white screen using the e centro oid informatio on and it is joining with the point found in n the previous p frame e this way draw wing is done by joining the e point t of current fram me with the po oints of previou us frames V.

2(b)

Ex xperime ental Set tup

2(c)

In n the experime ent an A4 tech webcam was connected to a Lapto op via USB ca able. The algor rithm was run under u dark and d light backgrounds which w were at t 40 inches fro om the camera a. t were obs served at different distance of o the markers s The tests from the camera. It was obse erved that the e accuracy of f ction decreased d as the distan nce between th he camera and d detec the marker m increase ed. A. So oftware Implem mentation Resu ults for Finger Counting & V Virtual Drawing g The algorithms a wer re implemente ed using MAT TLAB [1] for a video o stream of 200 frames at a video frame grab g interval of f five frames. f The al lgorithm failed d completely after a a distance e of 20 0 inches from th he camera in both b backgroun nds. It was also o obser rved that the e speed of detection d depe ended on the e proce essing device speed. s The acc curacy in dete ection was also o affect ted when the hand was at an a angle or th he fingers were e joined d together. In virtual drawi ing it was obs served that the e range e of detection can be increas sed by increasi ing the marker r size but b the contro ol of drawing was limited causing rough h edges s in drawing which w can be smoothen by increasing the e proce essing device speed.

2(d)

2(e)

2(f)

2 (a)

120% A c c u r a c y 100% 80% 60% 40% 20% 0% 7 11 16 20 Distancefromcamera(inches) 2(g) Fig.2. Experimentation Results. (a) Finger count from one to five. (b) Finger detection at different distances from the camera. (c) Errors caused by joining fingers or keeping the hand at an angle. (d) Stylus being detected. (e) GUI for Virtual Drawing. (f) Drawings made from Virtual Drawing. (e) Graph between Accuracy and Distance from the camera. Background1 Background2

[8] [9]

[10] [11]

[12] [13]

[14]

[15]

[16]

VI.

Conclusion

[17]

In this paper we have used color detection and shape recognition to perform finger counting and virtual drawing showing average result in return of simple algorithm. Color and hand schemes can provide an extension of output signals. It has been found that if a non rare color like light green is use so the probability of color matching in background becomes very less. The system is cheap and to run requires only one camera with a normal use laptop. . The system can be used as a subsystem in many applications and can be connected to external hardware using communication mediums.

[18]

[19]

[20]

[21]

References
[1] [2] [3] [4] [5] [6] [7] http://www.mathworks.com/products/matlab Chapter 1 Hand Tracking and Gesture Recognition for HumanComputer Interaction by Cristina Manresa-Yee, Javier Varona, Ramon Mas and F.J. Perales World Scientic Review Volume - 9in x 6in Color-Based Road Sign Detection and Tracking Luis David Lopez and Olac Fuentes Computer Science Department University of Texas, El Paso 79902, USA fldlopez Using Kinect to emulate an Interactive Whiteboard final thesis UNIVERSIT DEGLI STUDI DI TRENTO www.xbox.com/en-US/KINECT http://us.playstation.com/ps2/accessories/eyetoy-usb-camera-ps2.html http://www.apple.com/iphone-5s/

[22]

[23]

[24] [25]

http://www.samsung.com/pk/consumer/mobile-devices/mobilephones/smartphones/GT-I9500ZKAPAK Inspection of Bottles Crates in the Beer Industry through Computer Vision by Mrio Campos, Manuel Ferreira, Teresa Martins, Cristina Santos,Department of Industrial Electronics,University of Minho. 11 http://en.wikipedia.org/wiki/Finger_tracking 13 HAND GESTURE RECOGNITION: A LITERATURE REVIEW 1 Rafiqul Zaman Khan and 2 Noor Adnan Ibraheem 1,2 Department of Computer Science, A.M.U. Aligarh, India, International Journal of Artificial Intelligence & Applications (IJAIA), Vol.3, No.4, July 2012 DOI : 10.5121/ijaia.2012.3412 161, (19) A Virtual 3D Blackboard : 3D Finger Tracking Using a Single Camera 20 Real-Time Markerless Tracking the Human Hands for 3D Interaction, Erlangung des Doktorgrades (Dr. rer. nat.), Universitt Bonn Institut fr Informatik II Rmerstrae 164, D-53117 Bonn 22 Y. Sato, Y. Kobayashi, H. Koike. Fast tracking of hands and fingertips in infrared images for augmented desk interface. In Proceedings of IEEEInternational Conference on Automatic Face and Gesture Recognition (FG),2000. pp. 462-467 23 Real-Time Hand-Tracking with a Color Glove Robert Y. Wang,Jovan Popovic http://people.csail.mit.edu/rywang/handtracking/s09-handtracking.pdf 24 PARK, J., AND YOON, Y. 2006. LED-glove based interactions in multi-modal displays for teleconferencing. In International Conference on Articial Reality and TelexistenceWorkshops (ICAT), 395399. 25 D.J. Sturman and D. Zeltzer, A Survey of Glove-Based Input, IEEE Computer Graphics and Applications, vol. 14, pp. 30-39, Jan. 1994 Stephen C. Crampton Counting fingers in real time using computer vision techniques Boston Univetsity Graduate school of arts and computer sciences Chaudhary, A., Raheja, J.L., Das, K., Raheja, S.,"A Survey on Hand Gesture Recognition in International Journal of Embedded Systems and Computer Engineering Local Copy Vol. 3, No. 2, July-December 2011, pp 85-88 context of Soft Computing", Published as Book Chapter in Advanced Computing CCIS, Springer Berlin Heidelberg, Vol. 133, pp. 2-3, 2010. Sharma R, Huang TS (1997) Visual interpretation of hand gestures for humancomputer interaction: a review. Trans Pattern Anal Mach Intell 19(7):677695 Moeslund T, Granum E (2001) A survey of computer vision based human motion capture. Comput Vis Image Underst 81:231268 Siddharth S. Rautaray Anupam Agrawal (2012) Vision based hand gesture recognition for human computer interaction: a survey Springer Science+Business Media Dordrecht 2012 pg 2-3 Kevin NYY, Ranganath S, Ghosh D (2004) Trajectory modeling in gesture recognition using cybergloves and magnetic trackers. In: TENCON 2004. IEEE region 10 conference, pp 571574 http://www.metamotion.com/gypsy/gypsy-gyro.htm# Berci N, Szolgay P (2007) Vision based humanmachine interface via hand gestures. In: 18th European conference on circuit theory and design (ECCTD 2007), pp 496499