You are on page 1of 2

- We propose finger gesture-based user interaction on

mobile devices, focusing on mobile phones equipped with rear-
facing cameras. A user interacts with a mobile application by
using finger gestures with the hand that holds the mobile phone,
which is detected by video taken with the rear-facing camera. A
preprocessing method uses skin-color segmentation, morpho-
logical operation and skeletonization for finger tracking. Our
system generates mouse events and other commands reacting to
finger gestures.
I. INTRODUCTION
In recent years, the rapid growth oI mobile devices has made
people increasingly use multimedia content and web services in
their daily lives. Thus the importance oI the user interIace (UI)
has increased. Mobile devices currently support a variety oI UIs
Irom keypads to multi-touch screens and sensors. Accordingly,
there have been researches on new types oI input interIace
systems. Sensors such as accelerometers could be added to
mobile devices to support new types oI UIs. However, these can
be diIIicult to integrate into existing small mobile devices at the
hardware level |1|.
Thus, vision-based gesture recognition can serve as an
important way oI using camera-equipped mobile devices
because it does not require any other space on a device. It is an
active area oI research in the Iield oI human computer
interaction (HCI). A gesture-based UI has an advantage in that
the direct use oI hands and Iingers is a natural means Ior humans
to communicate with devices. ThereIore people can interact
more intuitively with their devices using a gesture-based UI than
with traditional interIaces.
We propose a Iinger gesture-based mobile user interIace using
a rear-Iacing camera. In previous research oI vision-based
gesture input interIace Ior mobile devices, two-handed
interaction is required with a Iront-Iacing camera |2|. But in the
proposed system, users can operate the device with Iinger
gestures Irom the hand that is holding the device. Using Iinger
gesture images through the rear-Iacing camera, the system
estimates the movement and gesture oI the Iinger to control
mobile applications.
II. FINGER GESTURE ESTIMATION
A. Finger Tracking
The proposed Iinger tracking algorithm uses skin-color
segmentation, morphological operation and skeletonization to

This research is Iinancially supported by the Ministry oI Education,
Science and Technology (MEST), the Ministry oI Knowledge
Economy (MKE) through the Iostering project oI the Industrial-
Academics Cooperation Centered University
accurately estimate the Iinger gestures using the images captured by
a low-resolution rear-Iacing camera.


(a)

(b)

(c)

(d)

(e)
Fig 1. Finger Tracking : (a) Capture images by a rear-Iacing camera (b) The original
image (c) Skin-color segmentation (d) Morphological operation (Erosion and Dilation)
(e) Skeletonization and Estimated Finger Point

We use YCbCr space instead oI RGB space to separate the
Iinger region eIIiciently Irom an image with a varying
background |3|. Each pixel is classiIied as either skin or non-
skin and together they become a new binary image as in Fig.
1(c) with our threshold value, which is deIined as Iollows:

otherwise
C C if
y x SkinColor
r b
0
) 178 133 ( ) 127 77 ( 1
¦ ) , (
s s s s
=

(1)

Morphological operation oI a 3 by 3 structuring element is
applied to the binary image as shown in Fig. 1(d). The erosion
and dilation operations are used in order to remove small noise
holes and to restore the shape oI the Iinger.
Fig. 1(e) shows the result oI skeletonization and a peak
Iinger point. Actually skeletonization is the process oI peeling
oII oI as many pixels as possible Irom a pattern without
aIIecting the general shape oI the pattern, which needs a lot oI
computation time. ThereIore it is not appropriate Ior a real-
time gesture recognition system on mobile devices. Thus we
compute the centroid oI each rows oI the binary Iinger image
to decrease computation time. At last, the skeletonized line is
used Ior the estimation oI the Iinger region orientation and the
peak point oI the line is applied to Iinger gesture commands.
B. Finger Gesture Commanas
In the proposed system, Iinger gestures can be perIormed Ior
the operations oI click, up, down, leIt and right. They can play
the same role as a directional keypad and mouse. All oI the
directional commands except click are deIined by chessboard
distance. The moved distance, direction and the instant speed oI
the Iinger point between two Irames determines the directional

Finger Gesture-Based Mobile User InterIace
Using a Rear-Iacing Camera
Jun-ho An, Stuaent member, IEEE and Kwang-Seok Hong, member ,IEEE
School oI InIormation and Communication Engineering, Sungkyunkwan University, South Korea
2011 IEEE International Conference on Consumer Electronics (ICCE)
978-1-4244-8712-7/11/$26.00©2011 IEEE 303

commands.
Assume that we have an M N × digital image I with
1 0 , 1 0 | , | ÷ s s ÷ s s e M f N i f i I and the current pixel
position oI the Iinger point is at ) , (
2 2
f i , while the previous
position was at ) , (
1 1
f i . The chessboard distance is deIined as:

) , max(
1 2 1 2
f f i i a
chess
÷ ÷ = (2)

TABLE I
Directional Commands by Finger Gesture Recognition
Directional Commands Conditions
Up
1 2 1 2
i i f f ÷ > ÷ ,
1 2
f f > , o / M a
chess
>
Down
1 2 1 2
i i f f ÷ > ÷ ,
1 2
f f < , o / M a
chess
>
LeIt
1 2 1 2
f f i i ÷ > ÷ ,
1 2
i i < , o / N a
chess
>
Right
1 2 1 2
f f i i ÷ > ÷ ,
1 2
i i > , o / N a
chess
>

Table.1 shows each oI the directional commands. In the
table, the variable o can be changed according to the Irame
rate oI the camera properly. In our system 4 = o is used with
the Irame rate oI 10Ips, 24 bit color 320 x 240 resolution.
The instantaneous rate oI change oI the Iingertip area is
applied to the click command by Iinger gestures and it is deIined
as the Iingertip area oI the current Irame over the Iingertip area
oI the previous Irame. The click command occurs when a user
moves his Iinger back and Iorth by clicking a mouse towards the
camera.


Fig 2. Example oI click action
III. EXPERIMENT AND APPLICATION
In the developed system, the Irame rate is 10Ips on our test
mobile device, an HP ipaq h4700 PDA with an HP photo-
smart mobile 1.3MP camera that is inserted into the SD slot.
We used the camera at a Irame rate oI 10Ips and a resolution
oI 320 x 240 pixels.
An accuracy assessment, with the developed Iinger gesture
test program in Fig. 3, shows an overall accuracy better than
88° under uniIorm lighting conditions when the background
image does not contain colors similar to skin color. However,
the accuracy drops considerably under low lighting conditions.


(a)

(b)

(c)
Fig 3. Examples oI Iinger gesture command : (a) Click command (b) Down
command (c) Right command

We implemented a Iew test applications to demonstrate the
proposed system`s strengths and limitations. One oI the
applications, a simple game called as Arkanoid, is shown in Fig.
4(b). Users can move the small paddle easily and hit the ball
using Iinger gestures. As a result, we demonstrated that our
system can be applied to any other mobile application such as a
web-browser, mobile game, or photo viewer that is controlled
by mouse and keypad,


(a)

(b)
Fig 4. Finger Gesture-based Mobile User InterIace : (a) Developed System
(b) Simple game application
IV. CONCLUSION
In this paper, we proposed a Iinger gesture-based mobile
user interIace using a rear-Iacing camera. A rear-Iacing camera
was chosen because it is now widely available on mobile
devices and can be used without introducing new sensors.
Furthermore, previous research on vision-based gesture input
interIace systems Ior mobile devices uses a Iront-Iacing
camera. But those systems require a two-handed user interIace
which is a little inconvenient Ior a mobile device. On the other
hand, our system can support an interIace using only one
Iinger. In the Iuture, robust skin-color segmentation under
various illumination conditions, the reduction oI computation
time Ior improved real-time perIormance, and the method oI a
more natural Iinger gesture command should be studied.
REFERENCE
|1| Eunjin Koh, Jongho Won, Changseok Bae, 'On-premise skin color
modeling method Ior vision-based hand tracking,¨ Consumer
Electronics, 2009. ISCE `09 IEEE 13
th
international Symposium on, vol.,
no., pp.908-909, 25~28 May, 2009.
|2| A. Haro, K. Mori, T. Capin and S.Wilkinson, 'Mobile camera-based
user interaction.¨ In Proc. oI IEEE Workshop on Human-Computer
Interaction, pages 79-89, 2005.
|3| F. Gasparini, R. Schettini, 'Skin Segmentation using Multiple
Threshoding,¨ SPIE proceedings, pp. 6061-18, 2006.
304

4.

5"'2 .

%.

.

 .

.

.

.

 ' .

 .

A"'.

(.

     ! #""".

.

7! ')&.

*.

  .

#""".

).

.

 .

.

" .

3.

)"&"&! .

 .

)".

7 .

.

                  .

   .

.

                  .

.

                  .

                      .

.

               .

    .

      "    !                   " .

 .

.

.

".

".

.

 .

 ".

.

.

!'".

 ' .

  .

.

#$.

.

#$.

.

.

.

.

.

 .

.

 .

!.

.

.

 .

 .

 .

.

 .

".

" .

.

.

!.

 .

.

.

 .

  .

".

.

 .

.

.

".

 .

# $.

 .

  .

.

%.

 .

".

".

.

.

.

.

.

& .

.

"'".

.

.

 .

.

(.

.

 .

.

 .

.

!.

.

.

".

 .

 .

).

".

.

.

".

.

.

.

.

 .

.

".

!.

.

.

 .

*! .

.

 .

.

".

.

 .

.

+.

 .

.

 .

.

.

 ! .

 .

.-. .

.

".

' .

".

.

 .

 .

.

.

 .

! .

.

".

  '/".

.

 .

 ".

.

.

.

/".

.

.

 .

.

.

  .

.

.

.

 .

 .

.

 .

.

.

.

.

" .

".

 .

#* $ .

(.

"' .

.

 .

.

  .

.

 .

.

.

".

.

 .

.

.

.

.

 " .

 .

.

" .

.

" .

!.

  .

.

.

 .

 .

.

" .

!.

.

 .

".

.

"' .

.

 .

!.

  .

  .

.

0.

.

.

.

"' .

.

".

 .

".

.

 ' .

  .

.

 ".

 .

.

' .

".

".

 .

.

.

 .

!' .

 .

.

/".

!.

.

' .

  .

-1. .

2".

.

.

.

.

".

 .

 .

.

 .

!.

.

".

.

.

 .

 .

.

.

.

  .

.

.

".

 .

".

.

 ' .

  .

.

.

 .

.

 .

.

".

.

.

.

.

.

.

  .

4 53 .

53) 3.

3) %( .

.

.

#$.

#$.

4.

. .

4.

 &.

9.

# $.

".

 .

.

.

 ' .

  .

#$.

.

 .

 .

#$.

)&'.

 .

#$.

% .

 .

#3.

.

 $.

.

#$.

)&6 .

.

.

3 .

4.

:.

# $.

.

0.

".

 ..

 .

 .

.

52.

 .

.

  .

.

.

.

.

.

.

 .

!.

.

.

 &".

-<. .

3 .

+.

.

 .

.

.

&.

.

' &.

.

.

.

.

.

!.

 .

 .

.

.

4 .

.#$.

!.

".

.

".

!.

.

.

.

!9.

.

. .

 #>>    .

<<    . #   $ @ = .1>$ #.

>?$ . .

.

$.#.

.

% .

 .

.

.

<.

.

<.

"".

.

.

.

.

.

 .

 .

.

!.

.

4 .

.#$ .

.

.

.

 .

 .

.

".

.

.

.

 .

 .

.

.

.

.

.

.

 .

.

.

 .

.

4 .

.#$.

!.

.

".

.

&6 .

.

.

 &.

.

 .

(" .

&6 .

.

.

.

.

.

.

.

.

 .

+.

.

.

.

.

 .

!".

.

.

 .

 .

.

.

 .

!.

.

.

.

.

" .

 .

.

.

.

.

 .

.

.

 ' .

".

.

.

.

.

  .

".

!.

".

.

.

.

 .

!.

.

.

 .

.

 .

.

 .

" .

 .

(.

 .

.

&6.

.

.

".

.

.

 .

.

.

.

.

 .

.

.

 &.

.

.

.

.

.

.

.

.

".

  .

 .

       .

.

.

.

.

".

 .

.

.

.

.

 .

.

&.

".

!.

.

.

 .

.

 .

 .

.

 .

.

.

.

 .

& .

.

" .

(.

.

.

.

.

 .

.

  .

".

.

.

%.

.

3" .

 .

 .

+.

&.

.

.

.

 .

).

.

.

#%3) $.

.

%.

.

7!.

  .

.

 .

 .

.

.

.

 .

.

.

3.

#%73$.

".

.

.

8.

.

.

" ' .

.

.

!.

!.

 .

.

.

 .

( .

 .

.

 .

.

   .

 .

.

.

 &.

.

".

&'.

 .

 .

 .

.

&6 .

.

00©2011 IEEE 303 . 978-1-4244-8712-7/11/$26.

.

  .

.

(".

 .

!.

 .

.

% $ .

 .

 .

# !.

# -.

= . & .

= & $ . . % .

.

".

+.

.

.

.

.

.

.

 #.

1  & 1 $ .

!.

.

 ".

.

! .

.

#.

$ .  &..

.

 .

 .

.

.

9.

.

    +# .

1 .

$ ..  &1 &.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

#1$.

.

(2B3.

.

 .

 .

.

4.

5".

.

 .

 .

.

.

!.

B.

.

.

.

# $.

#$.

#$.

4.

< .

3+ .

.

.

".

 .

9.

# $.

&.

 .

#$.

!.

 .

#$.

.

 .

.

.

&1 &1 &. . &.

1 .

. .

1 .

    $ C .  &1 &. &. &1 &1 &..

    $ C .  &1 &.

.

.

1 .

..

    % C .

.

.

1 .

..

    % C .

.

1 .

. .

1 .

..

 .

!.

 .

.

.

 .

  .

.

.

 .

.

 .

 .

.

 .

.

.

.

 .

 .

.

.

  .

 .

.

".

.

D .

".

!.

.

 .

 .

.

=..

1D.

.

.

<1=.

+.

1D=.

" .

.

.

  ".

 .

.

 .

.

.

.

 .

.

.

.

.

&.

 .

.

.

".

.

.

.

.

.

.

.

 .

.

.

".

 .

 .

.

.

 .

.

.

 ".

  .

.

&.

 .

".

!.

.

".

 .

.

.

 &.

.

.

.

&.

.

".

! .

.

  .

.

.

0.

.

.

!.

.

 .

.

 .

.

.

G.

.

.

  .

.

.

.

 .

.

.

 .

 .

.

(& .

.

!.

.

4 .

D#$ .

.

 .

 .

.

 .

 .

 .

.

.

.

 .

".

.

" .

(.

.

".

!.

 .

 .

".

.

 .

.

.

.

.

.

.

 .

".

.

.

!'!.

.

 .

.

.

!.

 .

.

.

.

".

.

& .

.

.

.

# $.

#$.

4.

D .

4.

5"' .

%.

.

 .

9.

# $.

 .

).

.

#$.

).

 .

 .

.

H B ) .

4.

1 .

.

3+ .

.

&.

.

.

( .

3E:3 %3 .

( .

(::B .

.

 .

.

.

 .

 .

.

=..

.

".

.

.

 .

.

*:.

 /.

D>==.

: (.

!.

.

*:.

'  .

.

<%:. .

  .

 .

.

.

.

.

) .

 .

0.

".

.

  .

.

.

 .

 .

.

.=.

.

.

".

.

<1=.

+.

1D=.

+ .

(.

" .

.

!.

.

 .

.

".

.

 .

.

4 .

<.

!.

.

  .

" .

.

 .

??F.

".

".

.

.

!.

.

 &".

 .

.

.

 .

.

 .

.

&.

 .

*! .

.

" .

.

 .

".

!.

.

 .

.

.

.

.

 .

!.

.

.

.

"' .

.

".

 .

".

.

 ' .

  .

(.

 ' .

  .

! .

.

 ".

.

.

!.

!.

 .

.

.

 .

.

 .

.

".

!".

".

!.

 .

4".

 ".

 .

.

' .

".

".

 .

.

.

.

 .

".

.

' .

  .

2".

.

.

/".

.

!' .

".

 .

!.

.

.

.

 .

.

.

.

  .

.

.

.

 .

".

.

 .

".

.

 .

".

.

.

 .

.

.

"".

".

&'.

 .

".

".

" .

.

.

".

.

" .

.

.

 .

 '.

 .

.

.

.

.

.

.

 " .

.

".

 .

".

.

" .

. 3. 343 3 -.

-1. -<. 3"8.

7.

A.

0.

 &.

2 .

I '.

&.

.

.

.

.

' .

 .

 &J.

".

3.

1==K .

) 3.

G=K.

333.

.<.

  .

)".

.

 .

 .

 K=?'K=K.

1LM1?.

% .

1==K .

.

( .

* .

7 .

%.

.

.

.

) 0&.

I%.

  ' .

".

  J.

.

: .

.

333.

0&.

.

*" ' ".

 .

 .

>K'?K.

1==L .

4 .

5  .

.

).

I)&.

) .

".

%".

J.

): 3.

.

 .

'.N=N.?.

1==N .

304 .