Home Search Collections Journals About Contact us My IOPscience

A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor

imagery of both force and speed of hand clenching

This content has been downloaded from IOPscience. Please scroll down to see the full text.

2015 J. Neural Eng. 12 036004

(http://iopscience.iop.org/1741-2552/12/3/036004)

View the table of contents for this issue, or go to the journal homepage for more

Download details:

IP Address: 169.230.243.252
This content was downloaded on 07/04/2015 at 02:18

Please note that terms and conditions apply.

 Journal of Neural Engineering

J. Neural Eng. 12 (2015) 036004 (12pp) doi:10.1088/1741-2560/12/3/036004

A hybrid BCI based on EEG and fNIRS

signals improves the performance of
decoding motor imagery of both force and
speed of hand clenching
Xuxian Yin1,2,7, Baolei Xu1,7, Changhao Jiang3,7, Yunfa Fu4,7,
Zhidong Wang1,5, Hongyi Li1,6 and Gang Shi1
1
State Key Laboratory of Robotics, Shenyang Institute of Automation (SIA), Chinese Academy of
Sciences (CAS), Shenyang 110016, People’s Republic of China
2
University of Chinese Academy of Sciences, Beijing 100049, People’s Republic of China
3
Key Laboratory of Motor and Brain imaging, Capital Institute of Physical Education, Beijing 100088,
People’s Republic of China
4
School of Automation and Information Engineering, Kunming University of Science and Technology,
Kunming 650500, People’s Republic of China
5
Dept. of Advanced Robotics, Chiba Institute of Technology, Chiba 2750016, Japan
6
School of Mechanical Engineering & Automation, Northeastern University, Shenyang 110004, People’s
Republic of China

E-mail: blxu@sina.cn

Received 28 November 2014, revised 11 January 2015

Accepted for publication 19 February 2015
Published 2 April 2015

Abstract
Objective. In order to increase the number of states classified by a brain–computer interface
(BCI), we utilized a motor imagery task where subjects imagined both force and speed of hand
clenching. Approach. The BCI utilized simultaneously recorded electroencephalographic (EEG)
and functional near-infrared spectroscopy (fNIRS) signals. The time-phase-frequency feature
was extracted from EEG, whereas the HbD [the difference of oxy-hemoglobin (HbO) and deoxy-
hemoglobin (Hb)] feature was used to improve the classification accuracy of fNIRS. The EEG
and fNIRS features were combined and optimized using the joint mutual information (JMI)
feature selection criterion; then the extracted features were classified with the extreme learning
machines (ELMs). Main results. In this study, the averaged classification accuracy of EEG
signals achieved by the time-phase-frequency feature improved by 7%, to 18%, more than the
single-type feature, and improved by 15% more than common spatial pattern (CSP) feature. The
HbD feature of fNIRS signals improved the accuracy by 1%, to 4%, more than Hb, HbO, or HbT
(total hemoglobin). The EEG–fNIRS feature for decoding motor imagery of both force and
speed of hand clenching achieved an accuracy of 89% ± 2%, and improved the accuracy by 1%
to 5% more than the sole EEG or fNIRS feature. Significance. Our novel motor imagery
paradigm improves BCI performance by increasing the number of extracted commands. Both the
time-phase-frequency and the HbD feature improve the classification accuracy of EEG and
fNIRS signals, respectively, and the hybrid EEG–fNIRS technique achieves a higher decoding
accuracy for two-class motor imagery, which may provide the framework for future multi-modal
online BCI systems.

Keywords: EEG-fNIRS, hand clenching force and speed, motor imagery, joint mutual
information (JMI), extreme learning machines (ELMs)
7
These authors contributed equally to the study.

1741-2560/15/036004+12$33.00 1 © 2015 IOP Publishing Ltd Printed in the UK

J. Neural Eng. 12 (2015) 036004
(Some figures may appear in colour only in the online journal)
1. Introduction
The brain–computer interface (BCI) is a type of technology
that decodes user intentions and directly realizes device
control using brain signals without the participation of the
peripheral nervous system and muscles [1]. First proposed by
Vidal in 1973 [2], BCI has acquired great developments in the
past two decades [3–6]. This technology is used for the
communication of locked-in patients and neurological reha-
bilitation of stroke patients [7, 8] as well as for the enter-
tainment of healthy users [9, 10]. In the literature, several
techniques have been used to analyze brain activities,
including electroencephalography (EEG) and functional near-
infrared spectroscopy (fNIRS).
EEG is the most used non-invasive brain signal for BCI
systems because of its high temporal resolution and low cost.
Here, the key step to achieve a high decoding accuracy is to
extract task-relevant features. Many researchers have inves-
tigated the features of time, spatial, or frequency domains for
EEG decoding, such as power spectral, band power, event-
related de-synchronization/synchronization (ERD/ERS), and
spatial patterns. However, few studies have focused on phase-
domain features. Lachaux et al [11] introduced phase locking
value (PLV) as a statistical measure to evaluate the phase
synchrony phenomenon between two EEG channels, whereas
Gysels et al [12] proposed that PLV is useful for spontaneous
EEG classification during mental tasks. Moreover, Li et al
[13] compared PLV with the phase interval value (PIV) for
classification of motor imagery for BCI applications, and their
results demonstrated that PIV performed better than PLV. In
this study, time-, phase-, and frequency-domain features are
combined to classify different motor imageries for BCI.
Results show that the time -phase-frequency feature can sig-
nificantly improve the classification accuracy.
fNIRS is another promising non-invasive approach for
BCI systems since Coyle et al first applied it in BCI research
in 2004 [14]. This technology measures de-oxyhemoglobin
(Hb) and oxyhemoglobin (HbO) concentration changes by
shining near-infrared light into the head (with two or more
wavelengths) and measuring the diffusively reflected light
changes that originate from physiological changes in the skin
and cortex [15–18]. Although fNIRS signals provide an
approach to investigating user intentions, an inherent time
delay of hemodynamic response exists.
Most motor imagery-based paradigms currently use
imagination of different limbs, such as the left and right
hands, feet, and tongue [19, 20]. The ability to produce
multiple control commands, however, may be limited.
Although combining motor imageries with other mental tasks,
such as word association and mental arithmetic, is one way to
increase the command number [21], motor parameter imagery
offers a more efficient approach to achieve this goal.
Jochumsen et al [22] detected ankle dorsiflexion movement
intentions associated with different levels of force and speed
using movement-related cortical potential (MRCP). This
2
X Yin et al
study can detect different movement intentions with limited
latencies, but this design is based on actual movement. Yuan
et al [23] investigated the relationship between hand clench-
ing speed and EEG activity and found that activities in alpha
and beta frequency bands were linearly correlated with the
hand clenching speed. However, it is unknown if hand
clenching force and speed motor imageries can be dis-
tinguished. Consequently, this study investigates motor
parameter imageries associated with both force and speed of
hand clenching. This paradigm is a feasible approach in
generating considerable BCI commands.
The hybrid BCI using the EEG–fNIRS combination can
resolve the particular limitations of single modality and
improve the classification accuracy. Fazli et al [24] demon-
strated that the performance of motor imagery-based BCI can
be enhanced by combining the measurements of EEG and
fNIRS signals. In the current study, the hybrid EEG and
fNIRS signals are recorded simultaneously and then com-
bined to improve the classification accuracy of decoding
motor imagery of both force and speed of hand clenching.
Results demonstrate that the classification of motor imagery
of both force and speed of hand clenching is possible, and that
the EEG–fNIRS features can achieve accuracy levels of
89% ± 2%; further, this method is more accurate compared
with that using single-modality features.
2. Materials and methods
2.1. Experiment paradigm
A paradigm based on motor parameter imagery associated
with hand clenching force and speed is proposed in this study
to increase the BCI command number. Unlike paradigms
using motor imageries of different limbs, the two different
motor imagery tasks in this study come from the same limb
(i.e., the right hand). The hand clenching speed task refers to
different hand clenching frequencies, such as 0.5, 1, and 2 Hz.
Before the experiment, subjects practiced actual movements
of different hand clenching speed levels using a metronome;
they were instructed to become familiar with the feeling of
different speed levels. However, during the experiment pro-
cess, they were requested to perform different hand clenching
speed levels of motor imageries without actual movement.
electromyography activity was monitored to exclude those
unqualified trials. The hand clenching force task refers to the
different hand clenching forces of the right hand (e.g., 20%,
50%, and 80% of the maximum hand clenching force, MF).
The MF of the right hand was measured using an electrical
dynamometer. Subjects were required to increase the hand
clenching force to a target value in 2 s and then maintain the
force (figure 1). Before the experiment, subjects were per-
formed substantial training until they could perform the
movements appropriately.
Six right-handed neurologically healthy volunteers (three
males and three females, average age: 26.8 ± 3.3 years)
J. Neural Eng. 12 (2015) 036004 X Yin et al

Figure 1. (a) The force transformation during hand clenching force

tasks. Before the experiment, participants perform the training of real
hand clenching. Participants are required to achieve their hand
clenching force to the target value in 2 s and then maintain the force
during the other period. During the experiment, subjects imagine to
achieve clenching without actual movement. (b) The experiment
paradigm used in the research.

participated in the experiment. All participants gave written

informed consent to join the experiment. Among these sub-
jects, subjects 1, 2, and 6 were trained with actual and ima-
gery movements for a total of three times before the
experiment, whereas subjects 3, 4, and 5 were only trained
one time before the experiment. The experiment was
approved by the Ethical Committee of the Shenyang Institute
of Automation, Chinese Academy of Sciences.
To acquire considerable task-relevant brain signals for
decoding motor imageries, the EEG and fNIRS signals were
recorded simultaneously. In consideration of the time lag in
the fNIRS response to cerebral activations [25], the time
duration of a single trial is longer than the time used for
traditional EEG paradigms. A single trial consists of four Figure 2. (a) The locations of EEG electrodes and fNIRS probes
parts, i.e., baseline, cue, task, and rest period [figure 1(b)]. used in the experiment. The red ones are light emitters and the blue
Owing to the inherent delay of hemodynamic response, the ones are light detectors. The locations between the two probe types
task duration was set to 10 s to ensure fNIRS signals reached are the measured channels. The emitter-detector distance is fixed at
30 mm. (b) The actual EEG and fNIRS channel locations in the
the maximum values. The rest period lasted from 10 s to 12 s experiment.
to eliminate the effect of periodic system error and make
fNIRS signals return to base value.

The EEG signals were acquired by Neuroscan Synamps2

2.2. Data acquisition
A total of 21 EEG electrodes and 24 fNIRS channels over the
primary motor area and the supplementary motor area were
used in this research [figure 2(a)]. The fNIRS probes con-
tained two types, i.e., the light emitters (the red ones) and the
light detectors (the blue ones). The locations between the
emitters and the detectors were set as the measured channels.
In our research, two 3 × 3 probe sets were used. The EEG
electrodes were arranged in accordance with the 10–20
international system, and the fNIRS probe sets were arranged
in accordance with the C3 and C4 locations of the EEG
electrodes.
at a sampling frequency of 1000 Hz, with A1 as the reference
and Fpz as the ground. The electrode impedance was reduced
to 5 K ohms before the experiment. The electro-oculogram
(EOG) was also recorded to ensure that no EOG artifact
existed during the motor imagery task period. Meanwhile, the
fNIRS signals were acquired by ETG-4000 (Hitachi Medical
Corporation, Japan) at a sampling frequency of 10 Hz. The
illuminators emitted near-infrared light at two different
wavelengths, i.e., 695 and 830 nm. The triggers were simul-
taneously sent to Synamps2 and ETG-4000 through parallel
and serial ports. Although the received triggers of the two
signal types may present a difference of a few milliseconds,
such a difference does not affect the following analysis.

3
J. Neural Eng. 12 (2015) 036004
Every subject participated in three sessions of the
experiment. Every session included 30 trials of the hand
clenching speed motor imagery task and 30 trials of the hand
clenching force motor imagery task. Given that the duration
of a single trial was longer than that of traditional EEG
paradigms, the number of trials in one session was corre-
spondingly smaller.
2.3. Signal preprocessing
2.3.1. EEG. In order to reduce the computational cost, the
EEG data were downsampled to 250 Hz, followed by a
Chebyshve II low-pass filter with five orders at a cutoff
frequency of 125 Hz to eliminate signal distortion. The data
were then band-pass filtered with a band of 5 to 45 Hz. A
small Laplacian filter [26] was used to improve the spatial
resolution of the EEG data. This is expressed by
1 where i is the imaginary unit, A(t) = sqrt(x(t) + y(t) ) is the
Vj Lap = Vj − ∑ Vk ,
N k∈S
j
where Vj is the jth channel, Sj is the surrounding channel set
of Vj, N is the size of Sj, and Vk is the kth channel in Sj.
Previous studies have shown that mu rhythm and beta rhythm
are effective for BCI control [27]. Thus, in this study, signals
of 8 to 12 Hz and 18 to 25 Hz were extracted for
classification.
2.3.2. fNIRS. The original fNIRS signals are optical intensity
signals of two wavelengths. Thus, the concentration changes of
Hb and HbO can be calculated with these optical signals using
the modified Beer–Lambert Law, as shown in equation (2),
where ελ1/λ2,Hb/HbO is the extinction coefficient of Hb/HbO
under the corresponding wavelength, Iλ1/λ2(t1/t2) is the
light intensity at times t1 and t2, and DPF is the ratio of
the actual path length of the optical photon to the illuminator-
detector distance [28]. Values for the differential path
length factors (DPF695 = 6.51, DPF830 = 5.86) and the
extinction coefficients (ε695, Hb = 1.9665, ε695,HbO = 0.3120,
ε830,Hb = 0.7804, ε830, HbO = 1.0507) are obtained from
[29, 30].
⎛ Iλ1 ( t1) ⎞
⎜ log
⎜ Iλ1 ( t2 ) ⎟
⎜ ⎟
⎛ ΔcHb ⎞ ⎛ ελ1, Hb ελ1, HbO ⎞−1 ⎜ DPFλ1 d ⎟
ΔC = ⎜ ⎟=⎜ ⎟
ελ 2, HbO ⎠ ⎜
⎝ ΔcHbO ⎠ ⎝ ελ 2, Hb Iλ 2 ( t1) ⎟
⎜ log
⎜ Iλ 2 ( t2 ) ⎟
⎜ DPF d ⎟
⎝ λ2
The low-frequency drift caused by probe movements in
the concentration data was removed using a linear-detrend
filter. The concentration data were then low-pass filtered
using a Chebyshev II filter with five orders, at a cutoff
frequency of 0.1 Hz to remove high-frequency artifacts, such
as heart and breathing rates. After eliminating the physiolo-
gical noise, the data were downsampled to 1 Hz to decrease
feature space dimensionality.
X Yin et al
2.4. Feature extraction
2.4.1. EEG. In this study, the time-frequency-phase feature
is proposed to improve the classification of different motor
imageries. Hilbert transform and the complex wavelet
convolution are two methods used to obtain the phase
information of a signal [11]. These two methods can obtain
similar results [31]. The Hilbert transform method is adopted
in this study.
The Hilbert transform of a signal x(t) can be obtained by
convoluting with the function h(t) = 1/πt using
∞ ∞
y (t ) = P ∫−∞ x (τ ) h (t − τ )dτ = π1 P ∫−∞ tx−(τ )τ dτ, (3)
where P is the Cauchy principal value. The analytic signal of
x(t) can be obtained using equation (2).
z (t ) = x (t ) + iy (t ) = A (t )eiθ (t ) , (4)
2 2
(1)
instantaneous amplitude (IA), and θ(t) = arctany(t)/x(t) is the
instantaneous phase (IP). The value range of θ(t) is [−π π].
The instantaneous frequency (IF) can be obtained by w(t) = dθ
(t)/d(t). To achieve the correct IF, θ(t) must be unwrapped by
adding multiples of ±2π when absolute jumps of more than π
occur between consecutive elements.
In this study, four feature types were included, i.e.,
power, IA, IP, and IF, which were combined together to
construct the time-phase-frequency feature. The latter three
features can be obtained from Hilbert transform. The
topographies of IP and IF between hand clenching force
and speed imagery are shown in figure 3.
To decrease the feature dimensions and improve the
classification stability, the original features were averaged
using a 0.5 s moving window with a step width of 0.125 s.
After testing four window length and step width combinations
(0.5 s to 0.125 s, 0.5 s to 0.2 s, 1 s to 0.125 s, and 1 s to 0.2 s),
we found that the 0.5 s to 0.125 s combination yielded the
best performance. The 0.5 s window length is reasonable in
consideration of the varied nature of EEG signals, and the
0.125 s step width is enough for achieving fluent BCI device
control. The averaged feature points between the time range
of [−0.5 0.5] of all the channels are combined into a vector,
and the original size of each feature type is 8 × 21 = 168. To
⎟ eliminate the degradation of the classifier performance caused
by feature value range variance and obtain a high classifica-
tion accuracy, each feature vector was normalized to the range
(2)
[−1 1] using equation (5). The four normalized feature vectors
⎟ were then merged into a single vector to construct the time-
phase-frequency feature.
⎠ ⎛ Feature − min(Feature)
Featurenorm = ⎜
⎝ max(Feature) − min(Feature)
− 0.5) × 2, (5)
2.4.2. fNIRS. Three types of concentration change signals
are currently used in fNIRS-based BCI applications, i.e., Hb,
HbO, and total hemoglobin (HbT, i.e., HbT = Hb + HbO),
along with two optical intensity signals. The signal pattern
4
J. Neural Eng. 12 (2015) 036004
Figure 3. (a) The topography of the IP of mu feature between hand
clenching force and speed imageries. Upper: topography of hand
clenching force imagery. Lower: topography of hand clenching
speed imagery. (b) The topography of the IF of mu feature between
hand clenching force and speed imageries.
HbD, i.e., the difference between HbO and Hb
concentrations, was also employed in this study [32]. HbO
and Hb concentrations are typically strongly negatively
correlated with each other under normal circumstances [33].
In this work, we propose that HbD may amplify the amplitude
of concentration changes due to cognitive tasks. The time
Figure 4. The time course of the four preprocessed fNIRS concentration signals activated during hand clenching force tasks.
5
X Yin et al
courses of the four preprocessed fNIRS concentration signals
activated by cognitive tasks are shown in figure 4.
The four concentration signals and two optical intensity
signals of 24 channels from the [0 12] s period were extracted
to construct the feature vectors. The original feature space of
each signal consisted of 12 × 24 = 288 values. The features
extracted from these six signal types were then normalized
using equation (5).
2.5. Feature optimization
The existence of redundant information in the original feature
space can significantly reduce the classification accuracy.
Therefore, feature optimization is essential in improving the
discriminative performance of a classifier. Feature extraction
and feature selection are two typically used feature optimi-
zation methods [34]. Although feature extraction projects the
original feature set to new features on the basis of transfor-
mations (e.g., principal component analysis [35]), feature
selection methods select the best subset of the original feature
set on the basis of some criteria. The disadvantage of the
feature extraction method is that the newly created feature
space is difficult to understand. Hence, the feature selection
method was chosen in our research.
Depending on whether the classifier is included in the
selection process, feature selection methods can be grouped
into ‘wrapper’ and ‘filter’ methods [36]. On the one hand, the
wrapper methods generally consider the classification accu-
racy of the classifier as the feature selection criterion, thus
resulting in a high classification rate. However, its general-
ization capability is poor, and its computational burden is
extensive. On the other hand, the filter method, based on joint
mutual information (JMI) criterion, has been chosen in this
study, because Brown et al [36] showed that this criterion
provides the best tradeoff between accuracy and stability.
In order to score the potential usefulness of a feature, the
JMI criterion uses the equation given by
JJMI ( Xk ) = ∑ I ( Xk X j ; Y ), (6)
Xj ∈ S
J. Neural Eng. 12 (2015) 036004 X Yin et al

Table 1. Decoding accuracies of different EEG feature types and their combinations.

Subject Power IA IP IF Power-IA-IP-IF

subj1 0.72 ± 0.13 0.72 ± 0.14 0.75 ± 0.07 0.81 ± 0.06 0.89 ± 0.07
subj2 0.73 ± 0.08 0.76 ± 0.07 0.78 ± 0.04 0.81 ± 0.02 0.86 ± 0.01
subj3 0.69 ± 0.05 0.69 ± 0.06 0.80 ± 0.02 0.80 ± 0.05 0.87 ± 0.06
subj4 0.62 ± 0.04 0.70 ± 0.08 0.83 ± 0.06 0.77 ± 0.02 0.88 ± 0.01
subj5 0.73 ± 0.03 0.73 ± 0.02 0.87 ± 0.09 0.82 ± 0.06 0.90 ± 0.07
subj6 0.73 ± 0.03 0.74 ± 0.07 0.82 ± 0.02 0.77 ± 0.06 0.88 ± 0.01
trained 0.73 ± 0.00 0.74 ± 0.01 0.78 ± 0.03 0.79 ± 0.02 0.88 ± 0.01
non-trained 0.68 ± 0.05 0.71 ± 0.02 0.83 ± 0.03 0.79 ± 0.02 0.88 ± 0.01
all 0.70 ± 0.04 0.72 ± 0.02 0.81 ± 0.04 0.79 ± 0.02 0.88 ± 0.01

Where Xk is the feature to evaluate, Xj is the feature that has
been selected already, S is the selected feature set, and Y is the
feature label. This equation stands for the information
between a joint random variable XkXj and the targets. The
details of the algorithm can be found in [37].
2.6. Classification
In this study, the extreme learning machines (ELMs) were
used to classify between hand clenching force and speed
motor imageries of the right hand. ELMs are types of single-
hidden layer feedforward neural networks (SLFNs) [38],
which can be used for regression and multiclass classifica-
tions [39]. Unlike other feedforward neural networks that use
gradient-based learning algorithms to tune all the network
parameters iteratively, ELMs use an analytical approach by
choosing the input weights randomly and then determining
the output weights of SLFNs. The advantages of ELMs
include extremely fast learning speed, small training error,
and good generalization performance. Huang et al [40] rig-
orously proved that the input weights and hidden layer biases
of SLFNs with infinitely differentiable activation functions
can be randomly assigned.
{
For a training set ℵ = (xi , ti ) xi ∈ R n , ti ∈ R m , i =
1, … , N , standard SLFNs with activation function g (x ) and a
hidden node number N are mathematically modeled next
N N
∑βi gi ( x j ) = ∑βi g ( wi
)
⋅ x j + bi = t j , j = 1, … , N ,
i=1 i=1
where wi is the input weight vector that connects the input
nodes and ith hidden node, β is the output weight vector that
connects ith hidden node and the output nodes, and bi is the
bias of ith hidden node.
Equation (7) can be written in a compact format as
Hβ = T,
where
⎡ g ( w1 ⋅ x1 + b1) ⋯ g ( w1 ⋅ x1 + bN ) ⎤
⎢ ⎥
H=⎢ ⋮ ⋯ ⋮ ⎥ , trained group. However, when the IP and IF features are used,
⎢⎣ g ( w1 ⋅ xN + b1) ⋮ g ( wN ⋅ xN + bN )⎥⎦
N ×N
⎡ βT⎤ ⎡tT⎤
⎢ 1 ⎥ ⎢ 1 ⎥
β=⎢ ⋮ ⎥ and T = ⎢ ⋮ ⎥ .
⎢ β T⎥ ⎢⎣ tN T ⎥⎦
⎣ N ⎦N × m N ×m
When the input weight wi and the bias bi are given, the
hidden layer output matrix H can be calculated. The output
weight β can be calculated using the equation given by
β = H †T, (9)
where H+ is the Moore–Penrose generalized inverse of matrix
H [41].
Liang [42] showed that although the accuracy of ELMs is
comparable with that of support vector machines, ELMs
require less computational cost and shorter training time.
3. Results
3.1. EEG
The classification accuracies between hand clenching force
and speed motor imageries of the right hand are shown in
table 1 and figure 5. When the power, IA, IP, and IF features
are used alone, their decoding accuracies are 0.70 ± 0.04,
0.72 ± 0.02, 0.81 ± 0.04, and 0.79 ± 0.02, respectively. The
phase feature (IP feature) and its derivative feature (IF fea-
ture) outperform the power and amplitude features. When the
four features are used simultaneously, the classification
(7) accuracy can be increased to 0.88 ± 0.01. The CSP method
performs a weighting of the electrodes to maximize the dif-
ference between two different motor tasks, and the channel
variance of the filtered signal is used for the classification
[43]. In this study, we find that the averaged classification
accuracy using CSP is 0.73 ± 0.03, which is worse than that of
the time-phase-frequency feature. This result demonstrates
(8) that using motor parameter imagery for BCI applications is
possible and the time-phase-frequency method outperforms
the traditional CSP method.
When the power and IA features are used, the average
accuracy for the trained group is better than that for the non-
the average accuracy for the non-trained group is not worse
than that for the trained group. Therefore, the phase feature

6
J. Neural Eng. 12 (2015) 036004
Figure 5. Decoding accuracies of different EEG feature types and their combinations. (a) Results for every subject; (b) Results for different
groups.
may provide significantly different information compared to
the amplitude feature.
3.2. fNIRS
The decoding accuracies using four concentration fNIRS
features and two optical fNIRS features are shown in table 2
and figure 6. It can be seen that figure 6(a) shows that dif-
ferent subjects have different feature types. The HbD feature
can acquire the best accuracy compared with other features.
For all subjects, the averaged accuracies of four concentration
features (Hb, HbO, HbT, and HbD) are 0.73 ± 0.05,
0.73 ± 0.05, 0.70 ± 0.06, and 0.74 ± 0.05, respectively, indi-
cating that the HbD feature can achieve a higher accuracy.
When the two optical features are used, the decoding
accuracies are 0.70 ± 0.05 and 0.70 ± 0.06, respectively,
which are lower than those of the four concentration features.
7
X Yin et al
The accuracy of the concentration-combined feature is higher
than that of the optical-combined feature by 0.03, which
implies that the concentration feature should be chosen
preferentially.
3.3. EEG–fNIRS
To investigate the performance of the EEG–fNIRS merged
feature, the four normalized EEG features and the four nor-
malized fNIRS concentration features were combined into
one feature vector, which was then optimized using the JMI
feature selection criterion. The accuracy of the EEG–fNIRS
merged feature and its comparison with the results of sole
EEG and fNIRS features are shown in table 3 and figure 7,
respectively. Although the accuracy of the merged feature can
only be increased by 0.01 for the average of all the subjects,
the accuracy is increased by 0.03 for subject 5 and by 0.05 for
 J. Neural Eng. 12 (2015) 036004
Table 2. Decoding accuracies of different fNIRS signal types and their combinations.

Subject Hb HbO HbT HbD Lam1 Lam2 Lam1-lam2 Hb-HbO-HbT Hb-HbO-HbT-HbD

subj1 0.76 ± 0.05 0.80 ± 0.09 0.79 ± 0.03 0.76 ± 0.01 0.77 ± 0.08 0.76 ± 0.01 0.79 ± 0.02 0.82 ± 0.06 0.80 ± 0.06
subj2 0.74 ± 0.08 0.77 ± 0.06 0.78 ± 0.02 0.78 ± 0.00 0.74 ± 0.06 0.76 ± 0.07 0.80 ± 0.06 0.77 ± 0.02 0.79 ± 0.04
subj3 0.64 ± 0.02 0.68 ± 0.02 0.63 ± 0.01 0.69 ± 0.01 0.62 ± 0.03 0.60 ± 0.02 0.67 ± 0.07 0.69 ± 0.01 0.69 ± 0.07
subj4 0.69 ± 0.08 0.68 ± 0.03 0.67 ± 0.04 0.67 ± 0.03 0.66 ± 0.08 0.68 ± 0.03 0.67 ± 0.04 0.67 ± 0.05 0.70 ± 0.05
subj5 0.75 ± 0.00 0.69 ± 0.07 0.68 ± 0.03 0.72 ± 0.02 0.72 ± 0.05 0.71 ± 0.06 0.75 ± 0.04 0.77 ± 0.04 0.76 ± 0.04
8

subj6 0.77 ± 0.05 0.79 ± 0.09 0.68 ± 0.06 0.79 ± 0.06 0.67 ± 0.02 0.67 ± 0.09 0.71 ± 0.05 0.77 ± 0.04 0.79 ± 0.03
trained 0.76 ± 0.01 0.78 ± 0.01 0.75 ± 0.05 0.78 ± 0.01 0.73 ± 0.04 0.73 ± 0.04 0.77 ± 0.04 0.79 ± 0.02 0.79 ± 0.00
non-trained 0.69 ± 0.05 0.68 ± 0.01 0.66 ± 0.02 0.69 ± 0.02 0.67 ± 0.04 0.66 ± 0.05 0.69 ± 0.04 0.71 ± 0.04 0.72 ± 0.03
all 0.73 ± 0.05 0.73 ± 0.05 0.70 ± 0.06 0.74 ± 0.05 0.70 ± 0.05 0.70 ± 0.06 0.73 ± 0.05 0.75 ± 0.05 0.76 ± 0.05

X Yin et al
J. Neural Eng. 12 (2015) 036004 X Yin et al

Figure 6. Decoding accuracies of different fNIRS signal types and their combinations. (a) Results for every subject; (b) Results for different
groups.

subject 6. Hence, the EEG and fNIRS features can supple-
ment each other and improve the classification of different
motor imagery tasks.
4. Discussion
4.1. Classification of sole EEG
Amplitude and phase are two aspects of a signal. Most fea-
tures used in EEG-based BCI applications are focused on the
amplitude aspect, such as band power, ERD/ERS, and CSP.
In this study, IP feature and its derivative feature (IF) were
compared with the amplitude feature and its square feature
(power feature). The results show that when used alone, the
phase feature outperforms the amplitude feature by approxi-
mately 10% for the average of all the subjects. When the
amplitude and phase features are used simultaneously, the
classification accuracy between hand clenching force and
speed motor imageries can be improved to 88% for both
trained and non-trained groups. The classification accuracy of
time-phase-frequency also outperforms the results using CSP

9
J. Neural Eng. 12 (2015) 036004 X Yin et al

Table 3. Decoding accuracies of fNIRS, EEG, and fNIRS-EEG by 15%. The time-phase-frequency feature proposed in the
features. current study can also be extended to time-spatial-phase-fre-
Hb-HbO-HbT- quency feature by adopting the CSP method.
Hb-HbO- Power-IA- HbD-&-power-
Subject HbT-HbD IP-IF IA-IP-IF
4.2. Classification of sole fNIRS
subj1 0.80 ± 0.06 0.89 ± 0.07 0.89 ± 0.02
subj2 0.79 ± 0.04 0.86 ± 0.01 0.87 ± 0.02 fNIRS is a new measurement method that can be used for BCI
subj3 0.69 ± 0.07 0.87 ± 0.06 0.87 ± 0.03 applications. Both concentration features and optical features
subj4 0.70 ± 0.05 0.88 ± 0.01 0.88 ± 0.01 were used in current research. After comparing the two fea-
subj5 0.76 ± 0.04 0.90 ± 0.07 0.93 ± 0.03 ture types, the results demonstrate that the concentration
subj6 0.79 ± 0.03 0.88 ± 0.01 0.92 ± 0.03
signal outperforms the optical signal, implying that the con-
trained 0.79 ± 0.00 0.88 ± 0.01 0.89 ± 0.02
centration signal is more effective in reflecting brain activa-
non-trained 0.72 ± 0.03 0.88 ± 0.01 0.89 ± 0.03
all 0.76 ± 0.05 0.88 ± 0.01 0.89 ± 0.02 tions than optical signals. Moreover, compared with the Hb,
HbO, and HbT features, the HbD feature can obtain the best

Figure 7. Decoding accuracies of fNIRS, EEG, and fNIRS-EEG merged feature. (a) Results for every subject; (b) Results for different
groups.

10
J. Neural Eng. 12 (2015) 036004
results when used alone. Finally, although the average accu-
racy of the Hb-HbO-HbT-HbD feature combination outper-
forms that of the Hb-HbO-HbT feature combination by only
1%, the HbD feature provides new information for cerebral
activations.
4.3. Classification of EEG–fNIRS
The results in this paper show that when the EEG–fNIRS
merged feature is used, the decoding accuracy can be
improved by 1% to 5%. The best accuracy using the merged
feature is 93% ± 3% for subject 5, which is high enough,
considering that only one hand is used in the experiment. The
merged feature outperforms the sole EEG feature and the sole
fNIRS feature. Although the classifier-level merged method
has been widely used in previous studies [24], the feature-
level merged method is more fundamental in improving
classification performance. Furthermore, the optimized fea-
tures offer cognitive neuroscience experts a convenient way
of explaining the neuromechanism behind the optimized
features.
EEG presents the neural electrical activity during brain
activation, whereas fNIRS measures hemoglobin oxygenation
information. There are three reasons why the EEG–fNIRS
merged features can achieve better performance than sole
EEG and sole fNIRS. First, when EEG and fNIRS are used
simultaneously, more task-relevant information content can
be enhanced about the motor imagery state of the subject.
Second, these two techniques can complement each other in
presenting cortex activation during motor imagery. Finally,
feature optimization can also extract the most relevant fea-
tures, which can separate hand clenching force and speed
motor imageries.
4.4. Limitations of the study
In our study, we used only a single-distance fNIRS approach,
and the hemodynamic responses caused by skin blood flow
may be present [44–48]. Although a Chebyshev II infinite
impulse response filter at a cutoff frequency of 0.1 Hz was
used to reduce physiological noise, the systemic interference
may inevitably be present in fNIRS measurements. It is
possible that changes in respiration that cause changes of CO2
values in the blood also affect the measured signals [49].
However, our objective was to distinguish two motor ima-
geries and extract considerable useful patterns to improve
accuracy, and not to understand the neural mechanisms of
functional brain activity. Our future work will thus examine a
two-detector approach to eliminating the systemic inter-
ference and checking whether this approach can significantly
improve classification accuracy.
In addition, the hand clenching force and speed motor
imageries in this study both come from the same right hand.
The results show that when the EEG–fNIRS feature is used,
the classification of these two motor imageries reaches 89%.
The usage of motor parameter imageries in BCI applications
is, therefore, feasible. Although the hand clenching force
motor imagery and speed motor imagery both contain three
11
X Yin et al
sub-classes, we only separated hand clenching force from
speed motor imagery. In future work, we will identify dif-
ferent levels of hand clenching force and speed motor
imageries.
In conclusion, the paradigm of hand clenching force and
speed motor imageries proposed in this study is feasible in
increasing control command numbers for BCI applications.
Furthermore, the EEG–fNIRS merged feature can better
improve classification accuracy compared with the single-
signal type approach.
Acknowledgments
This work is supported by the National Natural Science
Foundation (NNSF) of China under Grant Nos. 61203368,
81470084, and 61463024, and by the National High Tech-
nology Research and Development Program of China (863
Program) under Grant No. 2012AA02A605.
References
[1] Wolpaw J R, Birbaumer N, Mcfarland D J, Pfurtscheller G and
Vaughan T M 2002 Brain-computer interfaces for
communication and control Clinical Neurophysiology 113
767–91
[2] Vidal J 1973 Toward direct brain-computer communication
Ann. Rev. Biophys. Bioeng. 2 157–80
[3] Scherer R and Pfurtscheller G 2013 Thought-based interaction
with the physical world Trends Cogn. Sci. 17 490–2
[4] Ortiz-Rosario A and Adeli H 2013 Brain-computer interface
technologies: from signal to action Rev. Neuroscience 24
537–52
[5] Kubler A, Holz E and Botrel L 2013 Brain-computer
interfacing: science fiction has come true Brain 136 2001–4
[6] Donoghue J P 2008 Bridging the brain to the world: a
perspective on neural interface systems Neuron 60 511–21
[7] De Massari D, Ruf C A, Furdea A, Matuz T,
Van Der Heiden L, Halder S, Silvoni S and Birbaumer N
2013 Brain communication in the locked-in state Brain 136
1989–2000
[8] Sun H Y, Xiang Y and Yang M D 2011 Neurological
rehabilitation of stroke patients via motor imaginary-based
brain-computer interface technology Neural Regen. Res. 6
2198–202
[9] Marshall D, Coyle D, Wilson S and Callaghan M 2013 Games,
gameplay, and BCI: the state of the art IEEE T. Comput. Int.
Ai Games 5 82–99
[10] Bonnet L, Lotte F and Lecuyer A 2013 Two brains, one game:
design and evaluation of a multiuser BCI video game based
on motor imagery IEEE T. Comput. Int. Ai Games 5 185–98
[11] Lachaux J-P, Rodriguez E, Martinerie J and Varela F J 1999
Measuring phase synchrony in brain signals Human Brain
Mapping 8 194–208
[12] Gysels E and Celka P 2004 Phase synchronization for the
recognition of mental tasks in a brain-computer interface
IEEE T. Neur. Sys. Reh. 12 406–15
[13] Li J I E and Zhang L 2009 Phase interval value analysis for the
motor imagery task in BCI J.Circuit Sys. Comp. 18 1441–52
[14] Coyle S, Ward T, Markham C and Mcdarby G 2004 On the
suitability of near-infrared (NIR) systems for next-
generation brain–computer interfaces Phys. Measure. 25
815–22
J. Neural Eng. 12 (2015) 036004
[15] Jobsis F F 1977 Noninvasive, infrared monitoring of cerebral
and myocardial oxygen sufficiency and circulatory
parameters Science 198 1264–7
[16] Hoshi Y 2005 Functional near-infrared spectroscopy: potential
and limitations in neuroimaging studies Int. Rev. Neurobiol.
66 237–66
[17] Hoshi Y 2007 Functional near-infrared spectroscopy: current
status and future prospects J. Biomed. Opt. 12 062106
[18] Scholkmann F, Kleiser S, Metz A J, Zimmermann R,
Mata Pavia J, Wolf U and Wolf M 2014 A review on
continuous wave functional near-infrared spectroscopy and
imaging instrumentation and methodology Neuroimage 85
6–27
[19] Sano A and Bakardjian H 2009 Movement-related cortical
evoked potentials using four-limb imagery Int. J. Neurosci.
119 639–63
[20] Hashimoto Y and Ushiba J 2013 EEG-based classification of
imaginary left and right foot movements using beta rebound
Clinical Neurophysiology 124 2153–60
[21] Friedrich E V C, Scherer R and Neuper C 2013 Long-term
evaluation of a 4-class imagery-based brain-computer
interface Clin. Neurophys. 124 916–27
[22] Jochumsen M, Niazi I K, Mrachacz-Kersting N, Farina D and
Dremstrup K 2013 Detection and classification of
movement-related cortical potentials associated with task
force and speed J. Neural Eng. 10 056015
[23] Yuan H, Perdoni C and He B 2010 Relationship between speed
and EEG activity during imagined and executed hand
movements J. Neural Eng. 7 026001
[24] Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A,
Muller K R and Blankertz B 2012 Enhanced performance by
a hybrid NIRS-EEG brain computer interface Neuroimage
59 519–29
[25] Matthews F and Pearlmutter B A 2008 Hemodynamics for
brain-computer interfaces IEEE Signal Proces. Mag. 25
87–94
[26] Mcfarland D J, Mccane L M, David S V and Wolpaw J R 1997
Spatial filter selection for EEG-based communication
Electroen. Clin. Neuro. 103 386–94
[27] Mcfarland D J, Miner L A, Vaughan T M and Wolpaw J R
2000 Mu and beta rhythm topographies during motor
imagery and actual movements Brain Top. 12 177–86
[28] Delpy D T, Cope M, Van Der Zee P, Arridge S, Wray S and
Wyatt J 1988 Estimation of optical pathlength through tissue
from direct time of flight measurement Phys. Med. Biol. 33
1433–42
[29] Cope M 1991 The development of a near infrared spectroscopy
system and its application for non invasive monitoring of
cerebral blood and tissue oxygenation in the newborn infants
PhD dissertation Department of Medical Physics and
Biology, University College London, UK
[30] Duncan A, Meek J H, Clemence M, Elwell C E, Tyszczuk L,
Cope M and Delpy D T 1995 Optical pathlength
measurements on adult head, calf and forearm and the head
of the newborn infant using phase resolved optical
spectroscopy Phys. Medicine Bio. 40 295–304
[31] Le Van Quyen M, Foucher J, Lachaux J, Rodriguez E, Lutz A,
Martinerie J and Varela F J 2001 Comparison of Hilbert
transform and wavelet methods for the analysis of neuronal
synchrony J. Neuroscience Meth. 111 83–98
[32] Yoshino K and Kato T 2012 Vector-based phase classification
of initial dips during word listening using near-infrared
spectroscopy Neuroreport 23 947–51
12
X Yin et al
[33] Cui X, Bray S and Reiss A L 2010 Functional near infrared
spectroscopy (NIRS) signal improvement based on negative
correlation between oxygenated and deoxygenated
hemoglobin dynamics NeuroImage 49 3039–46
[34] Jain A K, Duin R P W and Mao J C 2000 Statistical pattern
recognition: a review IEEE T. Pattern Analysis Machine Int.
22 4–37
[35] Yu X, Chum P and Sim K B 2014 Analysis the effect of PCA
for feature reduction in non-stationary EEG based motor
imagery of BCI system Optik 125 1498–502
[36] Brown G, Pocock A, Zhao M-J and Luján M 2012 Conditional
likelihood maximisation: a unifying framework for
information theoretic feature selection J. Machine Learn.
Res. 13 27–66
[37] Meyer P E, Schretter C and Bontempi G 2008 Information-
theoretic feature selection in microarray data using variable
complementarity IEEE J. Sel. Topics Signal Proces. 2
261–74
[38] Huang G B, Zhu Q Y and Siew C K and Ieee 2004 Extreme
learning machine: A new learning scheme of feedforward
neural networks IEEE Int. Joint Conf. Neural Networks
(New York) (IEEE)
[39] Huang G-B, Zhou H, Ding X and Zhang R 2012 Extreme
learning machine for regression and multiclass classification
IEEE Transactions on Systems, Man, and Cybernetics—Part
B: Cybernetics 42 513–29
[40] Huang G B, Zhu Q Y and Siew C K 2006 Extreme learning
machine: Theory and applications Neurocomputing 70
489–501
[41] Rao C R and Mitra S K 1972 Generalized inverse of a matrix
and its applications Proc. of the Sixth Berkeley Symp. on
Mathematical Statistics and Probability Volume 1: Theory
of Statistics (Berkeley, CA) (University of California Press)
[42] Liang N Y, Saratchandran P, Huang G B and Sundararajan N
2006 Classification of mental tasks from EEG signals using
extreme learning machine Int. J. Neural Sys. 16 29–38
[43] Ramoser H, Muller-Gerking J and Pfurtscheller G 2000
Optimal spatial filtering of single trial EEG during imagined
hand movement Reh. Eng., IEEE T. 8 441–6
[44] Erdoğan S B, Yücel M A and Akın A 2014 Analysis of task-
evoked systemic interference in fNIRS measurements:
insights from fMRI NeuroImage 87 490–504
[45] Gagnon L, Yücel M A, Boas D A and Cooper R J 2013 Further
improvement in reducing superficial contamination in NIRS
using double short separation measurements NeuroImage 85
127–35
[46] Kirilina E, Jelzow A, Heine A, Niessing M, Wabnitz H,
Brühl R, Ittermann B, Jacobs A M and Tachtsidis I 2012 The
physiological origin of task-evoked systemic artefacts in
functional near infrared spectroscopy Neuroimage 61 70–81
[47] Tachtsidis I, Leung T S, Chopra A, Koh P H, Reid C B and
Elwell C E 2009 False Positives in Functional Nearinfrared
Topography, in Oxygen Transport to Tissue XXX (New
York: Springer) p 307–14
[48] Takahashi T, Takikawa Y, Kawagoe R, Shibuya S,
Iwano T and Kitazawa S 2011 Influence of skin blood flow
on near-infrared spectroscopy signals measured on the
forehead during a verbal fluency task Neuroimage 57
991–1002
[49] Scholkmann F, Gerber U, Wolf M and Wolf U 2013 End-tidal
CO2: an important parameter for a correct interpretation in
functional brain studies using speech tasks Neuroimage 66
71–9