Journal of Neuroscience Methods 190 (2010) 289–298

Contents lists available at ScienceDirect

Journal of Neuroscience Methods

journal homepage: www.elsevier.com/locate/jneumeth

A brain–computer interface method combined with eye tracking for 3D

interaction
Eui Chul Lee a , Jin Cheol Woo b , Jong Hwa Kim c , Mincheol Whang d,∗ , Kang Ryoung Park e
a
Division of Fusion and Convergence of Mathematical Sciences, National Institute for Mathematical Sciences (NIMS), 628, Daeduk Boulevard,
3F Tower Koreana, Yuseong-gu, Daejeon 305-340, Republic of Korea
b
Culture Technology Institute, Sangmyung University, 7 Hongji-dong, Jongro-Ku, Seoul 110-743, Republic of Korea
c
Department of Emotion Engineering, Sangmyung University, 7 Hongji-dong, Jongro-Ku, Seoul 110-743, Republic of Korea
d
Division of Digital Media Technology, Sangmyung University, 7 Hongji-dong, Jongro-Ku, Seoul 110-743, Republic of Korea
e
Division of Electronics and Electrical Engineering, Dongguk University, 26 Pil-dong 3-ga, Jung-gu, Seoul 110-715, Republic of Korea

a r t i c l e i n f o a b s t r a c t

Article history: With the recent increase in the number of three-dimensional (3D) applications, the need for interfaces to
Received 22 December 2009 these applications has increased. Although the eye tracking method has been widely used as an interaction
Received in revised form 6 May 2010 interface for hand-disabled persons, this approach cannot be used for depth directional navigation. To
Accepted 11 May 2010
solve this problem, we propose a new brain computer interface (BCI) method in which the BCI and eye
tracking are combined to analyze depth navigation, including selection and two-dimensional (2D) gaze
Keywords:
direction, respectively.
BCI (brain computer interface)
The proposed method is novel in the following five ways compared to previous works. First, a device to
Eye tracking
3D interaction
measure both the gaze direction and an electroencephalogram (EEG) pattern is proposed with the sensors
needed to measure the EEG attached to a head-mounted eye tracking device. Second, the reliability
of the BCI interface is verified by demonstrating that there is no difference between the real and the
imaginary movements for the same work in terms of the EEG power spectrum. Third, depth control for
the 3D interaction interface is implemented by an imaginary arm reaching movement. Fourth, a selection
method is implemented by an imaginary hand grabbing movement. Finally, for the independent operation
of gazing and the BCI, a mode selection method is proposed that measures a user’s concentration by
analyzing the pupil accommodation speed, which is not affected by the operation of gazing and the BCI.
According to experimental results, we confirmed the feasibility of the proposed 3D interaction method
using eye tracking and a BCI.
© 2010 Elsevier B.V. All rights reserved.

1. Introduction
With the recent increase in the number of three-dimensional
(3D) applications, the need for interfaces with these applications
has increased. For example, there are many kinds of next-
generation interfaces, such as a 3D mouse having six degrees
of freedom (DOFs), gesture interaction, and full body interaction
(Bowman et al., 2005). Among these new types of interfaces, a
brain computer interface (BCI) and eye tracking can acquire a
user’s intention through electroencephalography (EEG) and gaze
direction, respectively. However, a BCI is commonly used for
one-dimensional (1D) controlling or object selection including
emulations of arm reaching or grabbing (Santhanam et al., 2006;
∗ Corresponding author. Tel.: +82 2 2287 5473.
E-mail addresses: eclee@nims.re.kr (E.C. Lee), mcun@naver.com
(J.C. Woo), rmx2003@naver.com (J.H. Kim), whang@smu.ac.kr (M. Whang),
parkgr@dongguk.edu (K.R. Park).
Lebedev and Nicolelis, 2006). Also, most eye tracking methods
obtain only gaze positions on a plane, such as a computer mon-
itor or a 2D image (Lee and Park, 2009a; Gonzalez and Woods,
2002; Yoo et al., 2002; Shih and Liu, 2004; Lee et al., 2009a). Con-
sequently, current eye tracking methods cannot acquire a depth
directional gaze position but only an X–Y plane, since they calculate
gaze position by using a single eye. In general, a human recog-
nizes the depth of an object using information from both eyes.
To determine the depth direction gaze position, both eye images
need to be analyzed. A study of 3D eye tracking analyzing the
relationship between the inter-distance of both eye centers and
the Z distance is described in Essig et al. (2004). However, since
both eye regions had low resolution and were extracted and ana-
lyzed from one face image, the accuracy of the depth directional
gaze tracking was low. To solve this problem, two high resolu-
tion cameras need to capture each eye image at a close distance,
which increases the weight and the cost of device. In addition, a
complex 3D calculation is required to evaluate the eye and facial
movements. Consequently, combining a gaze tracking method with

0165-0270/$ – see front matter © 2010 Elsevier B.V. All rights reserved.
doi:10.1016/j.jneumeth.2010.05.008
290 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298
depth-controllable interaction methods offers a good solution for
3D interaction.
An arm reaching approach is an alternative solution for depth
directional control (Bowman et al., 2005). Although the required
information can be easily measured by attaching acceleration or
magnetic sensors to several positions on a glove or arm, this method
can be inconvenient for the user. Therefore, in previous studies
measuring arm reaching, a brain computer interface (BCI) was
adopted (Santhanam et al., 2006; Lebedev and Nicolelis, 2006;
Pistohla et al., 2008). In these studies, arm reaching and hand grab-
bing were characterized by measuring and analyzing EEG patterns.
Reaching and grabbing motions were triggered based on the imag-
inary movement of the user and a pre-defined threshold, which
was obtained by a user-dependent calibration. This BCI method has
been used to control artificial arms and hands for the disabled (Gao
et al., 2005) consequently, an imaginary-based BCI method com-
bined with an eye tracking method could be an important solution
as an interface for 3D interaction.
In this study, a head-mounted device was implemented to sup-
port an eye capturing camera and EEG measuring sensors. From the
captured eye images, the pupil position was used to determine the
gaze positions, and the pupil size was used to measure the user’s
concentration. By measuring the pupil accommodation speed from
sequential eye images, two modes (a gaze mode and an arm reach-
ing mode) were automatically switched in order to independently
operate two navigation methods using the eye and arm. Imaginary
movement of the grabbing was used for a selection method. Con-
sequently, by combining these methods, we implemented the new
3D interaction method using eye tracking and imaginary-based BCI.
The structure of this paper is as follows: explanations of each
interface, such as eye tracking, pupil measurement, and EEG anal-
ysis are described in Section 2. In Section 3, several experimental
results used to verify the performance and the reliability of the pro-
posed methods are shown. Discussions about experimental results,
conclusions, and future research plans are presented in Section 4.
2. The proposed method
A diagram of the proposed 3D interaction method is shown in
Fig. 1. First, imaginary movement of arm reaching was performed
for depth directional navigation, as shown in Fig. 1(a). The imagery
Fig. 1. The conceptual diagram of the proposed method (a) the depth directional
navigation based on the imaginary arm reaching movement, (b) the 2D navigation
using eye tracking and (c) the object selection based on the imaginary grabbing
movement.
from the user was analyzed from the acquired EEG patterns. Depth
directional navigation is based on the dwell time of the imagery.
For example, the object of the second layer (A in Fig. 1) was con-
sidered as the selection in the case of imagining about the arm
reaching for 2 s. In addition, for imaginary movements of the same
activity for 4 s, the object of the third layer (B in Fig. 1) was con-
sidered. The user then freely carried out 2D directional navigation
using the eye tracking system, as shown in Fig. 1(b). If the user
gazes at one position on the X–Y plane, the operation mode of the
system is changed to the arm reaching mode based on the change
of pupil accommodation speed, which was analyzed from the cap-
tured eye images. After gazing at the destination, the final selection
of the object is performed by the imaginary movement of grabbing
motion, as shown in Fig. 1(c). Consequently, the proposed method
can provide navigation in the depth direction and in the 2D plane,
and also the selection of the object.
Actually, the optimal scheme of our proposed method is the
concurrent operation of the BCI and gaze tracking. However, it is
requires a great deal of time for the user-dependent training proce-
dure in order to separate the three BCI statuses of idle, grabbing, and
arm reaching. To reduce the duration of the user-dependent train-
ing procedure, we adopted the mode switching method using the
pupil accommodation speed. That is to say, the problem of triple
decisions such as idle, grabbing and arm reaching is divided into
two binary decisions by separating the two BCI stages (depth direc-
tional navigation, and selection). Through using this method, the
BCI training time was significantly reduced from about 5 min to
about 30 s.
The overall procedure of the proposed method is shown in Fig. 2.
Each interface is described in the following sub-sections. The
measurement and analysis of the EEG patterns are explained in
Section 2.1. The eye tracking device and method are explained in
Section 2.2. The explanation of the measurement method of the
pupil accommodation speed is presented in Section 2.3.
2.1. Measuring the EEG signal
2.1.1. The device
The proposed head-mounted system includes a device to mea-
sure EEGs, which consists of eight head-attached electrodes
for measuring the EEG signal and an amplifier (EEG100C,
BIOPAC Systems Inc., USA) (http://www.biopac.com/eeg-
electroencephalogram-amplifier, accessed on 15th Mar., 2010);
a data acquisition module using an analog-to-digital converter
(ADC); and an analysis system(NI PXI-1042Q, National Instruments
Corporation, USA) for the EEG signal (http://www.ni.com, accessed
on 15th Mar., 2010).
As shown in Fig. 3, the eight shielded electrodes (F-E5GH,
An Astro-Med Inc., USA) (http://www.astro-med.com, accessed
on 15th Mar., 2010) were attached using conductive gel (Ten20,
Weaver and Company, USA) (http://www.doweaver.com, accessed
on 15th Mar., 2010) at locations: F3, F4, C3, C4, P3, P4, O1, and O2 on
the top of the subjects’ heads. Each location was calculated on the
basis of the international 10-20 system after having measured each
participant’s head size (Herwig et al., 2003). The reference locations
were A1 and A2, located on both earlobes. These measurement loca-
tions were selected to measure brain activities in most areas (the F3
and F4 locations originated at the prefrontal cortex, the C3 and C4
locations originated from the lateral premotor cortex, the P3 and
P4 locations originated at the posterior parietal cortex, and the O3
and O4 locations originated at the occipital cortex) by using min-
imum electrodes (Babiloni et al., 2002). The discrimination of the
imaginary and realistic movement was successfully done by using
extended C3 and C4 locations from these locations (Neuper et al.,
2009). The EEG signals were sampled at 400 Hz. The 50 Hz notch
interference filter was activated to shield the subject and electrodes
 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298 291

Fig. 2. The overall procedure of the proposed method (a), (b), and (c) corresponding to the operations shown in Fig. 1(a), (b), and (c), respectively.

from electrical disturbances. By using the headset device of Fig. 3 2.1.2. The analyzing method
and monitoring the wearing status of the device by the adminis- With the acquired EEG data, the signal powers were calculated
trator during the experiment, the locations, according to the 10-20 for two frequency bands: alpha (8–13 Hz) and beta (14–34 Hz). The
system, were confirmed for each individual. acquired data from the eight electrodes in the time domain were

Fig. 3. The configurations of the EEG measurement device and electrodes (a) the EEG acquisition system and (b) an example of the eight fixed electrodes.
292 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298

transformed into the frequency domain using a fast Fourier trans-

form (FFT). Then, the changing patterns of each mode (gaze, arm
reaching, and grabbing) for real and imaginary movements were
analyzed based on the power of each time frequency block (Michel
et al., 1991).
In addition, we performed coherence analyses of the two fre-
quency bands in order to find mutual relations between two signal
patterns from different electrodes. A coherence-based analyzing
method can confirm a brain processing pattern by defining the rela-
tionship between the sub-regions of the brain through quantifying
the correlation between the two signal patterns. Coherence was
calculated using following equations:


∞
 
xy (ω) = Cxy exp(−2irω), ω ∈ −1/2, . . . , 1/2 (1)
Fig. 4. The determination of the ERD/ERS thresholds used for the discrimination
r=−∞
between movements and idle.
2 2
Axy (ω)2 = Re(xy (ω)) + Im(xy (ω)) (2)

Axy (ω)2
kxy (ω) = (3) et al., 2009; Lee and Park, 2009a,b; Lee et al., 2009b; Daugman,
xx (ω)yy (ω)
2004):
Eq. (1) was used to determine coherence by calculating the
  
cross-spectrum, which is defined by covariance Cxy . As shown in (2), ∂ I(x, y) 
max(r,x0 ,y0 )  ds (4)
the power spectrum Axy can be acquired by summing the squares  ∂r 2r
of the real and the imaginary terms of cross-spectrum xy . Eq. (3) r,x0 ,y0

was used to calculate the coherence kxy . The calculated coherence

value is represented in the range from 0 to 1. The coherence values
of 0 and 1 represent no correlation and the strongest correlation,
respectively (Ruchkin, 2005).
The following process was the analysis method of the move-
ment cognition. First, EEG training data, which consist of repeatedly
imaginary movements and idle was measured over 30 s. Next, sig-
nal powers were calculated for the alpha and beta bands using the
EEG training data. As shown in Fig. 4, the thresholds that extract
the event-related desynchronization/event-related synchroniza-
tion (ERD/ERS) patterns are determined.
The ERD/ERS thresholds were defined by the Bayesian rule based
method for determining among ERD/ERS and idle distributions. To
avoid excessive switching between ERD/ERS and idle status, the
thresholds were flexibly applied by considering the 2 sigma. Con-
sequently, these two thresholds are applied to recognize imagery
based movements such as arm reaching and grabbing.
2.2. Eye tracking
Eye tracking was used to determine the 2D gaze direction. The
device and algorithm used for the eye tracking are explained in
Sections 2.2.1 and 2.2.2, respectively.
2.2.1. The eye tracking device
There are two components in the proposed eye tracking sys-
tem (Cho et al., 2009): a universal serial bus (USB) camera, and a
near-infrared light-emitting diode (NIR LED), as shown in Fig. 4.
The captured image from the USB camera has a spatial resolu-
tion of 640 × 480 pixels and the capturing speed is 15 frames/s
(http://www.logitech.com, accessed on 15th Mar., 2010). The illu-
mination angle and wavelength of the NIR LED are about 23 degrees
(horizontally and vertically) and 850 nm, respectively. The eye cap-
turing camera was attached to the end of the flexible frame from
the head mounting device in order to user independently capture
NIR light images of the eye, as shown in Fig. 3(b).
2.2.2. The eye tracking method
In this section, the procedure for detecting the pupil center is
described. To detect the initial pupil center, we used a circular edge
detection (CED) algorithm based on (4) and shown in Fig. 4(a) (Cho
where I(x, y) represents a gray level at the (x, y) position, and
(x0 , y0 ) and r represent the center position and the radius of a
pupil to be detected, respectively. However, the CED algorithm
may not determine an accurate pupil center, as shown in Fig. 5(b),
because the actual pupil shape can be distorted or become ellip-
tical as a result of gazing. Therefore, the radius value estimated
from the CED algorithm cannot be used to represent the size of the
pupil.
To solve this problem, we used a local thresholding method
using an adaptive thresholding of the local area based on the initial
pupil center, as shown in Fig. 5(c). Through this method and mor-
phological operations, an accurate pupil region was acquired, as
shown in Fig. 5(e). The geometric center of the pupil area was used
to calculate the gaze position, and the acquired pupil sizes from the
image sequences were used to calculate the pupil accommodation
speed (Lee and Park, 2009b; Lee et al., 2009b).
User-dependent calibrations were performed to obtain an accu-
rate eye gaze position. According to the wearing position of the
proposed device from Fig. 4 and the sitting position of the user, the
movements of the pupil can vary even when gazing at the same
position. Thus, in the initial stages, user-dependent calibrations
must be performed. To achieve this, a user was asked to look at
four positions in the 2D virtual plane, as shown in Fig. 6. The four
pupil centers of (x1 , y1 ), (x2 , y2 ), (x3 , y3 ), and (x4 , y4 ) in the captured
images were obtained, as shown in Fig. 7, and were used to obtain
the eye gaze position.
Fig. 7 shows the mappings between the pupil movable area
defined by the four pupil centers at the calibration stage and the
virtual 2D plane shaped by the four gazed positions. The mapping
function was defined as a transform matrix T between the pupil
movable area ((x1 , y1 ), (x2 , y2 ), (x3 , y3 ), and (x4 , y4 )) and the vir-
tual 2D plane ((x1 , y1 ), (x2 , y2 ), (x3 , y3 ) and (x4 , y4 )), as shown in
Fig. 7. Here, we defined the virtual 2D plane in a monitor region.
To determine the transform between these two distorted squares,
a geometric transform method was used based on the following
equations (Lee and Park, 2009a; Gonzalez and Woods, 2002; Cho
et al., 2009):
x1 = ax1 + by1 + cx1 y1 + d
(5)
y1 = ex1 + fy1 + gx1 y1 + h
 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298 293

Fig. 5. The procedure of the pupil detection method (Lee and Park, 2009b): (a) detecting the coarse position of the pupil by using circular edge detection, (b) the defined
local area based on the coarse pupil center, (c) the histogram of the detected pupil area, (d) the binarized pupil area and (e) the geometric center of the pupil after region
filling by using a morphological operation.

To obtain the eight unknown parameters (a through h), a matrix

representation was used as follows:
P = T P1
⎡ ⎤ ⎡ ⎤⎡ ⎤
x1 x2 x3 x4 a b c d x1 x2 x3 x4
⎢ y1 y2 y3 y4 ⎥ ⎢ e f g h ⎥ ⎢ y1 y2 y3 y4 ⎥
⎣ ⎦=⎣0 0 0 0 ⎦ ⎣ x1 y1 x2 y2 x3 y3 x4 y4 ⎦
0 0 0 0
0 0 0 0 0 0 0 0 1 1 1 1

(6)

Since the transformation matrix T was obtained using the

inverse matrix of P1 , the eight parameters (a through h) can be cal-
culated. Based on the eight parameters, the gaze position (gx , gy ) in
a virtual 2D plane (monitor region) was calculated using the pupil
center (cx , cy ) in a current image and (7):

Fig. 6. The procedure for the user-dependent calibration for eye tracking by gazing ⎡ = T⎤ C ⎡
G ⎤⎡ ⎤
at four points of a virtual 2D plane.
gx a b c d cx
⎢ gy ⎥ ⎢ e f g h ⎥ ⎢ cy ⎥ (7)
⎣ 0 ⎦ = ⎣0 0 0 0⎦⎣ 1 ⎦
0 0 0 0 0 1

Fig. 7. The mapping relationship between the pupil movable area defined by calibration and the virtual 2D plane.
294 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298
Fig. 8. Examples of the same person’s pupil accommodation: (a) the expanded pupil
and (b) the contracted pupil.
2.3. Measuring the pupil accommodation speed
From the captured eye image sequences, we measured the pupil
accommodation speed in order to switch modes by estimating the
user’s concentration. The pupil accommodation speed was calcu-
lated based on the following digital image processing methods.
In our study, pupil accommodation means the natural change
of pupil size which cannot be controlled by human intention or
environmental illumination. The pupil accommodation speed was
calculated based on the extracted pupil sizes found in sequen-
tial eye images. Fig. 8 shows the example images of when pupil
accommodation occurred. In previous studies, it was shown that
pupil accommodation speed is related to eyestrain. If the amount
of eyestrain becomes high, the average pupil accommodation speed
decreased (Lee and Park, 2009b; Lee et al., 2009b). In addition, pupil
size reflects the concentration (Kramer et al., 1997). However, since
pupil size can be affected by other factors such as environmental
lighting conditions or emotions, pupil accommodation speed was
used in the proposed system to estimate the concentration.
The experimental results used to validate the proposed scheme
are shown in Section 3.4. The average pupil accommodation speed
was measured using the following equation (Lee and Park, 2009b):

14
 
p= Sn+1 − Sn 
n=1
In (8), Sn and Sn+1 are the sizes of pupil in the nth and (n + 1)th
images, respectively. In (8), the size changes in the pupil are
summed over frame numbers 1–14, which correspond to the pupil
Fig. 9. The experimental setting of (a) the experiment device and of (b) the experimental procedure.
accommodation speed for 1 s because the image acquisition speed
of our system is 15 frames/s (http://www.logitech.com, accessed on
15th Mar., 2010). Because the scale of the pupil size has individual
variation, we normalized it to the range from 0 to 1.
3. The experimental results
Twenty undergraduate students (age: 22.4 ± 1.7) participated in
the experiment. They did not have physical and cognitive problem
at hand grabbing. Visuo-motor tasks were done by the participants.
Visual light was presented in each stage of the visuo-motor task
indicating idle, attention, gaze, and arm reaching using an LCD
Monitor (24 in., Samsung, Korea). Participants tried to grab two ten-
nis ball attached to both horizontal ends of the monitor, as shown
Fig. 9(a).
The movements were asked to be both real and imaginary in
this study. Fig. 9(b) shows the experimental procedure, which
was repeated ten times, five times with real movement and five
times with imaginary movement. This procedure consisted of idle,
attention, gaze, arm reaching and grabbing. This procedure had
five stages of sequential movements. The idle was non-movement
under the light-off condition for 5 s. The attention was performed
under a light-on for 5 s. The gaze was the stage of gazing at the ball
under light for 5 s. The next stage was to reach the arm to the ball
for 5 s; its moment starting the movement was indicated by light-
on for 1 s. The final stage was to grab the ball for 10 s at the velocity
of one grab per 2 s.
The ANOVA (analysis of variance) was performed to find the sig-
nificant difference among stages in the EEG coherence and the EEG
power spectrum of the alpha and beta. Significant difference was
verified by Gabriel Post hoc. The Bayesian rule was adopted in order
to determine the threshold of the pupil accommodation speed for
switching the modes from gaze to BCI, alternatively. The exper-
imental results of the EEG power spectrum, the EEG coherence,
the eye tracking accuracy, and the pupil accommodation speed are
shown in the following sections.
3.1. The EEG pattern analysis in terms of average power
In this section, we show the experimental results of the EEG
(8) power according to user tasks such as arm reaching and grab-
bing. Performing three stages of gaze, arm reaching, and grabbing,
twenty participants repeated two times by both real and imaginary
movements.
As shown in the results of real movement in Fig. 10(a), the power
of the alpha frequency band in the grabbing stage was significantly
 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298 295

Fig. 10. The average EEG power of the alpha and beta bands showing significant difference among movements (*, **, and *** indicate significance at <0.1, <0.05, and <0.01,
respectively) (a) the results for real movements and (b) the results for imaginary movements.

greater than that of the arm reaching stage, except for the EEG pat- areas. This may be due to the change from rest to grabbing caus-
tern from the C3 and O1 areas. The power of the beta frequency band ing event-related synchronization (ERS). Also, the EEG power of the
in the grabbing stage was significantly greater than that of the arm idle stage was significantly greater than that of the arm reaching
reaching stage, except for the EEG pattern from the C3, P4 and O1 stage in alpha frequency band at O1 area and in beta frequency at

Fig. 11. The EEG coherences of the alpha and beta bands showing significant difference between movements. Broken line showed significant different at p < 0.05 and solid
line at p < 0.01 and showing (a) the results for real movement and (b) the results for imaginary movement.
296 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298

Fig. 12. The average normalized pupil accommodation speed at each stage.

F4, O1 and O2 areas. The difference between idle and arm reaching
was caused from event-related desynchronization (ERD).
As shown in the imaginary movement results found in Fig. 10(b),
the power of the alpha and beta bands in the arm reaching and grab-
bing stages was significantly greater than in the idle stage except
for the EEG pattern from the C3, P3, and O1 areas. These results
showed that the EEG pattern of imaginary movement was different
from that of real movement. Therefore, based on the results shown
in Fig. 10, we found that the power of the EEG could be used for
discriminating between the three stages of the idle, arm reaching,
and grabbing on imaginary movements.
3.2. Analyzing the EEG patterns in terms of coherence
As shown in Fig. 11(a), the EEG coherence of the alpha and beta
bands for the arm reaching stage was significantly greater than for
the grabbing and idle stages. The EEG coherence increased when
performed movement of the idle to arm reaching and grabbing
to arm reaching. These results were the case of real movement.
In contrast, during the imaginary movement, the EEG coherence
decreased when moved from the idle to reaching stage and grab-
bing stage as shown Fig. 11(b). Therefore, three movements of the
idle, arm reaching, and grabbing can be discriminated by strength
of the EEG coherence in real movement. However, in imaginary
movement, the EEG coherence of the alpha and beta bands for
the idle stage was significantly greater than for the arm reaching
and grabbing stages meaning coherence decrease during move-
ment progress. Therefore, EEG coherence can discriminate between
movement and non-movement but not the three stages in the case
of imaginary movement.
experiment, we analyzed the gaze detection accuracy by calculat-
ing the successful detection rate among 600 trials (20 subjects × 30
trials). Based on a gaze detection accuracy rate of 97.2% (583/600),
we confirmed that the eye tracking system could be used to val-
idate the feasibility of the proposed interface. In our experiments
for measuring eye tracking accuracy, head movements after the
user-dependent calibration were not allowed.
3.4. Analyzing the pupil accommodation speed for mode selection
In this section, we describe the experimental results of the
pupil accommodation speed during eye gazing, arm reaching, and
grabbing. One experimental set was organized into idle, gaze, arm
reaching, and grabbing stages. This set was performed two times
by real and imaginary movements for both the arm reaching and
grabbing. 20 persons participated in this experiment and the exper-
imental results are shown in Fig. 12. Here, we used the normalized
pupil accommodation speed to reduce individual variations. The
average pupil accommodation speed was calculated by using aver-
age sliding window. The slowest and fastest normalization speeds
were measured from the training videos which were obtained
from 10 users for 300 s during pre-experiment. The slowest and
the fastest speeds were obtained as 78 pixels/s and 6315 pixels/s,
respectively.

3.3. The accuracy of eye tracking

In this section, the experimental results for the eye tracking are
described. To measure the accuracy of the proposed eye tracker,
two kinds of tests were performed. First, subjects sequentially
gazed at the pre-defined 25 points on the monitor plane. As a
result, the root mean square (RMS) pixel errors of the eye tracker
were 23.4 pixels (X-axis: 18.7 pixels, Y-axis: 14.0 pixels) which cor-
responded to the RMS angle error of 1.08 degrees (X-axis: 0.87
degrees, Y-axis: 0.65 degrees) when the user was positioned at the Z
distance of 75 cm from a 24 in. display having a 800 × 600 pixel spa-
tial resolution. Second, 20 subjects were asked to gaze at one of nine
pre-defined regions on the monitor plane (the right side of Fig. 7),
which was done by the random indication of the highlighted region.
And each subject repeated this procedure thirty times. After the Fig. 13. An example of selecting an object by using the proposed method.
 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298
Table 1
The success rate of the feasibility test.
Depth navigation using
imaginary movement
of arm reaching
Success rate (%) 77.5% (155/200)
(success/trial)
Fig. 14. The sequence of selecting a yellow highlighted object: (a) selecting the 2nd layer by depth directional navigation using the dwell time of the imaginary movement
of arm reaching, (b) navigating to a mid-right object by using eye tracking and (c) selecting the mid-right object by using the imaginary movement of grabbing.
As shown in Fig. 12, we confirmed that there was no tendency
difference in real and imaginary movements. Surprisingly, a low
pupil accommodation speed was clearly observed in the gaze stage
compared to the other stages. Therefore, we can easily determine
the threshold of imaginary movement for switching between the
two modes (eye gazing, and arm reaching including grabbing).
Here, the threshold was determined based on the equal error rate
(EER) of switching between the two modes. When we determined
the threshold as 0.384, as shown in Fig. 12, the corresponding EER
was 5.97%. This means that the probability of falsely recognizing the
gaze stage as the arm reaching stage including grabbing is 5.97%.
3.5. The feasibility test using an integrated system
In this section, we show the experimental results that par-
ticipants randomly selected highlighted objects by using the
proposed system. Experiments were performed in a desktop mon-
itor environment, as shown in Fig. 13. Virtual cylindrical objects
of 3 × 3 × 3 were arranged in horizontal, vertical and depth direc-
tions, respectively, as shown in Fig. 14. 20 subjects participated in
this experiment, and each subject performed 10 trials. We asked
each subject to select the highlighted virtual object in the monitor
plane.
When all the procedures including the depth directional naviga-
tion, the imaginary movement of arm reaching, 2D navigation using
eye gazing, and object selection using the imaginary movement of
grabbing were successful, we regarded it as the case of final correct
selection. The rate of final correct selections is shown in Table 1.
All the failure cases of the first (depth navigation using the
imaginary movement of arm reaching) and the third stages (object
selection using the imaginary movement of grabbing) were caused
by the errors of the interpretation of BCI signal. Among the 45 fail-
ure cases of the first stage (depth navigation), the 4 cases were
occurred by user’s missing of his intended level of depth. And the
other failure cases were caused by the errors in the engagement of
the system about the depth navigation.
The 3 failure cases of the second stage (2D navigation by eye
gaze) were occurred by the errors of eye tracking. The 37 failure
cases of the third stage (object selection) were definitely occurred
by the errors of the interpretation of BCI signal. The 3 failure cases
297
2D navigation by eye Object selection using Final correct selection
gaze imaginary movement
of grabbing
98.5% (197/200) 81.5% (163/200) 74.5% (149/200)
were caused by mode switching error among the 51 failure cases
of the final selection.
4. Conclusion
We presented a 3D interaction method in which depth direc-
tional navigation, 2D navigation, and selection were performed by
the imagery of arm reaching, eye tracking, and the imagery of grab-
bing, respectively. Since a user’s concentration is estimated by the
pupil accommodation speed, two modes (the gaze mode and the
BCI mode) can be alternately switched. As demonstrated by the
experimental results, the proposed interface showed an acceptable
performance and we confirmed the possibility of 3D interaction
using eye tracking and the BCI.
Although previous eye tracking methods showed good perfor-
mance in terms of navigation, there were no appropriate selection
schemes. To perform the selection or triggering, eye blinking or
dwell time of gazing was adopted (Sibert and Jacob, 2000). How-
ever, natural blinks or dwell time of gazing having no intention
with respect to the selection can result in erroneous selections. A
previous method combining eye tracking and a BCI used the BCI
as the only selection method (Vilimek and Zander, 2009). Although
using a BCI as a binary decision is acceptable in terms of the stability
of the system, the scheme is too simple, considering the complex
configuration involved in using a BCI interface. Therefore, in our
proposed method, a BCI was used to perform both the selection
and the depth directional navigation. In previous researches, it was
reported that the EEG patterns of the real and imaginary move-
ments are similar (Decety and Michel, 1989; Decety and Ingvar,
1990). In contrast, in our experimental results, we confirmed that
the EEG power spectrum of the real movement and its correspond-
ing imaginary movement were different. This result could be due
to our experiments being performed using open eye and training
of imaginary movement, which may be not enough to successfully
perform the imaginary movement as well as the real movement.
We expect that our mode selection method using pupil accom-
modation speed is applicable to other situations. We confirmed
that the pupil accommodation speeds showed significant differ-
ences before and after determining one position, i.e., concentration
as a physical phenomenon can be quantitatively measured since
298 E.C. Lee et al. / Journal of Neuroscience Methods 190 (2010) 289–298
the pupil accommodation speed seems to reflect the degree of
concentration.
By using the proposed interface, we expect that a hand-disabled
person can perform a 3D interaction procedure by navigating sur-
roundings through eye tracking, approach a destination by the
imagery of arm reaching, and select an object by the imagery of
hand grabbing. Also, the proposed method can be adapted to the
control of an artificial arm or hand.
In future works, we will increase the usability of the device by
decreasing the number of sensors used to measure the EEG and the
weight of the eye tracker. By adopting the improved eye tracker to
allow for facial movements, we plan to increase the resolution of
2D navigation. Also, we would ultimately consider the user’s con-
venience by using a concurrent operation of BCI and gaze tracking
through the research for reducing the BCI training procedure to
triple decision.
Acknowledgements
This research was supported by R&D Program of MKE (Ministry
of Knowledge Economy) [10032108, Development of Multiverse
Game Platform Technology with Multimodal Interaction] and in
part by Basic Science Research Program through the National
Research Foundation of Korea (NRF) funded by the Ministry of Edu-
cation, Science and Technology (2009-0073315).
References
Babiloni C, Babiloni F, Carducci F, Cincotti F, Cocozza G, Percio C, et al. Human cor-
tical electroencephalography (EEG) rhythms during the observation of simple
aimless movements: a high-resolution EEG study. Neuroimage 2002;17:559–72.
Bowman DA, Kruijff E, LaViola JJ, Poupyrev I. 3D user interfaces theory and practice.
Redwood City, CA, USA: Addison Wesley Longman Publishing Co., Inc; 2005.
Cho CW, Lee JW, Lee EC, Park KR. A robust gaze tracking method by using frontal
viewing and eye tracking cameras. Opt Eng 2009;48:127202.
Daugman J. How iris recognition works. IEEE Trans Circuits Syst Video Technol
2004;14:21–30.
Decety J, Ingvar DH. Brain structures participating in mental simulation of motor
behavior: a neuropsychological interpretation. Acta Psychol 1990;73:13–34.
Decety J, Michel F. Comparative analysis of actual and mental movement times in
two graphic tasks. Brain Cogn 1989;11:87–97.
Essig K, Pomplun M, Ritter H. Application of a novel neural approach to 3D gaze
tracking: by vergence eye-movements in autostereograms. In: Proceedings of
the 26th annual meeting of the cognitive science society; 2004.
Gao X, Xu D, Cheng M, Gao S. A BCI-based environmental controller for the motion-
disabled. IEEE Trans Neural Syst Rehabil Eng 2005;11:137–40.
Gonzalez RC, Woods ARE. Digital image processing. second ed. Upper Saddle River,
NJ, USA: Prentice-Hall; 2002.
Herwig U, Satrapi P, Schonfeldt-Lecuona C. Using the international 10–20 EEG
system for positioning of transcranial magnetic stimulation. Brain Topogr
2003;16:95–9.
Kramer SE, Kapteyn TS, Festen JM, Kuik DJ. Assessing aspects of auditory handicap
by means of pupil dilatation. Int J Audiol 1997;36:155–64.
Lebedev MA, Nicolelis MA. Brain–machine interfaces: past, present and future.
Trends Neurosci 2006;29:536–46.
Lee EC, Ko YJ, Park KR. Gaze detection based on AAM and multiple SVR on mobile
device. Opt Eng 2009a;48:077002.
Lee EC, Park KR. A robust eye gaze tracking method based on a virtual eyeball model.
Mach Vis Appl 2009a;20:319–37.
Lee EC, Park KR. Measuring eyestrain on LCD TV according to adjustment factors of
image. IEEE Trans Consum Electron 2009b;55:1447–52.
Lee EC, Park KR, Whang M, Min KH. Measuring the degree of eyestrain caused
by watching LCD and PDP devices. Int J Ind Ergonom 2009b;39:798–
806.
Michel CM, Lehmann D, Henggeler B, Brandeis D. Localization of the sources of EEG
delta, theta, alpha and beta frequency bands using the FFT dipole approximation.
Electroencephalogr Clin Neurophysiol 1991;82:38–44.
Neuper C, Scherer R, Wriessnegger S, Pfurtscheller G. Motor imagery and
action observation: modulation of sensorimotor brain rhythms during men-
tal control of a brain–computer interface. Clin Neurophysiol 2009;120:
239–47.
Pistohla T, Balla T, Schulze-Bonhagea A, Aertsena AD, Mehringa C. Prediction of arm
movement trajectories from ECoG-recordings in humans. J Neurosci Methods
2008;167:105–14.
Ruchkin D. EEG coherence. Int J Psychophysiol 2005;57:83–5.
Santhanam G, Ryu SI, Yu BM, Afshar A, Shenoy KV. A high-performance
brain–computer interface. Nat Lett 2006;442:195–8.
Shih SW, Liu J. A novel approach to 3-D gaze tracking using stereo cameras. IEEE
Trans Syst Man Cybernet B 2004;34:234–45.
Sibert LE, Jacob RJ. Evaluation of eye gaze interaction. In: Proceedings of the SIGCHI
conference on human factors in computing systems; 2000. p. 281–8.
Vilimek R, Zander TO. BC(eye): combining eye-gaze input with brain–computer
interaction. Lect Notes Comput Sci 2009;5615:593–602.
Yoo DH, Kim JH, Lee BR, Chung MJ. Non-contact eye gaze tracking system by mapping
of corneal reflections. In: Proceedings of the 5th IEEE int. conf. on automatic face
and gesture recognition; 2002. p. 101–6.