Journal of Neural Engineering

PAPER You may also like

- Cooperative dry-electrode sensors for
An investigation of in-ear sensing for motor task multi-lead biopotential and bioimpedance
monitoring
classification M Rapin, M Proença, F Braun et al.

- Ultra-low power signal conditioning system

for effective biopotential signal recording
To cite this article: Xiaoli Wu et al 2020 J. Neural Eng. 17 066010 Diksha Thakur, Kulbhushan Sharma,
Sonal Kapila et al.

- Pulsatile flow conditioning of three-

dimensional bioengineered cardiac
ventricle
View the article online for updates and enhancements. Nikita M Patel and Ravi K Birla

This content was downloaded from IP address 45.249.87.200 on 21/05/2022 at 13:38

 J. Neural Eng. 17 (2020) 066010 https://doi.org/10.1088/1741-2552/abc1b6

Journal of Neural Engineering

PAPER

An investigation of in-ear sensing for motor task classification

RECEIVED
1 July 2020
Xiaoli Wu1, Wenhui Zhang1, Zhibo Fu1, Roy T H Cheung2,3 and Rosa H M Chan4
REVISED
30 September 2020 1
MindAmp Limited, Hong Kong, People’s Republic of China
2
ACCEPTED FOR PUBLICATION Department of Rehabilitation Sciences, The Hong Kong Polytechnic University, Hong Kong, People’s Republic of China
3
15 October 2020 School of Health Sciences, Western Sydney University, Sydney, Australia
4
PUBLISHED Department of Electrical Engineering, City University of Hong Kong, Hong Kong, People’s Republic of China
19 November 2020
E-mail: rosachan@cityu.edu.hk

Keywords: electroencephalography, in-ear sensing, human–computer interface, wearable device

Abstract
Objective. Our study aims to investigate the feasibility of in-ear sensing for human–computer
interface. Approach. We first measured the agreement between in-ear biopotential and
scalp-electroencephalogram (EEG) signals by channel correlation and power spectral density
analysis. Then we applied EEG compact network (EEGNet) for the classification of a two-class
motor task using in-ear electrophysiological signals. Main results. The best performance using
in-ear biopotential with global reference reached an average accuracy of 70.22% (cf 92.61%
accuracy using scalp-EEG signals), but the performance in-ear biopotential with near-ear reference
was poor. Significance. Our results suggest in-ear sensing would be a viable human–computer
interface for movement prediction, but careful consideration should be given to the position of the
reference electrode.

1. Introduction state auditory/visual evoked potentials [10] and alpha

Electroencephalogram (EEG) has high temporal res-
olution and is relatively more tolerant of subject
movements compared to other neuroimaging tech-
niques such as functional magnetic resonance ima-
ging [1, 2]. Nevertheless, its low portability and
complicated system setup limit its applicability to
everyday life. Hence, the applications of EEG are
often constrained to medical or research laborator-
ies [3, 4]. To enhance the applicability of EEG and
human–computer interface in daily life, a more port-
able, wearable, and comfortable electrophysiological
signal monitoring device is necessary.
Ear-EEG was first proposed by Looney et al in
2012 [5], as a method that records EEG signal from
electrodes embedded on a personalized earpiece.
Nguyen et al introduced the in-ear EEG recording
system for sleep staging [6]. Compared with the
scalp-EEG system, in-ear EEG monitoring exhibits a
higher degree of wearable potential and more user-
comfortable property, as well as higher tolerance to
the muscle movements [7]. In-ear sensing for mul-
tiple modalities monitoring improves the measure-
ment accuracy and minimize the artifacts [8, 9].
Some studies have demonstrated that it is sufficient
to extract brain activities such as steady and transient
attenuation [11] from the in-ear EEG.
Although these studies showed the potential of
in-ear sensing in the human–computer interface,
its application for the motor task is still not well
developed, despite that the motor task is of great
importance in human–brain interfaces. Kim et al
used signals from ear-adjacent scalp EEG for motor
imagery task classification and showed promising res-
ults [12]. But the usage of the ear-adjacent scalp EEG
was relatively limited in portability [13–15].
Therefore, we introduced an in-ear electro-
physiological signal based human–computer inter-
face system for a 2-class motor task in this paper.
With both in-ear and ear-around sensors, such
wearable in-ear electrophysiological signal monit-
oring device could facilitate a more practical and
wearable human–computer application, and benefit
healthy and disabled users in daily life [16, 17].
We first investigated the quality of the in-ear elec-
trophysiological signals by calculating the channel
correlation and the power spectral density (PSD) dif-
ferences between ear sensors, the adjacent scalp EEG
channels, and the motor-related scalp EEG chan-
nels. We then evaluated the classification accuracy
of using the in-ear electrophysiological signals or
the scalp EEG signals as the input of the EEGnet

© 2020 IOP Publishing Ltd

J. Neural Eng. 17 (2020) 066010
classifier. Our results shed light on the possibility
of the in-ear electrophysiological signals for motor
task classification as the human–computer inter-
face. The dataset is available in the IEEE DataPort
(http://dx.doi.org/10.21227/j7rq-2p11).
2. Method
The in-ear electrophysiological signal recording
device used in this study comprised two customized
earpieces, each of which had four recording electrodes
embedded as shown in figure 1. The ear-electrodes
were made of Ø 4 mm silver (Ag) flakes printed with
Ag/AgCl ink, and for all data collected in this paper,
the electrode positions were xF, xB, xOU, xOD, where
x marks the left (L) or right (R) ear. The ear canal
electrodes were labeled as xF, xB, contacting with
the front, and back ear canal; the other two concha
electrodes were xOU and xOD, corresponding to
the upper and lower positions of outer ears. The
custom-fit earpieces were hand-crafted in the follow-
ing steps: (1) making ear impressions via injecting
high-viscosity silicone to canals; (2) obtaining the
ear model by solidifying the agar of impressions;
(3) manufacturing earpieces by using ultraviolet ray
irradiated resin; (4) Wire leads from the electrodes
connecting to recording system directly, and a hole at
the end of each earpiece for auditory signal delivery.
2.1. Experimental paradigm and data collection
Six subjects (two males, four females, ages: 22–28)
participated in this study. All were right-handed,
without medication and free from any known cent-
ral nervous system abnormality. Complete informa-
tion about the experimental procedures was given to
the subjects, and an informed consent document was
signed before participation. The study was approved
by the ethics committee at the City University of
Hong Kong (Reference number: 2-25-201602_01).
Before the placement of earpieces, the concha and ear
canals were cleaned with ethanol. A small amount of
high viscosity conductive gel (Ten 20 EEG paste) was
applied on each ear electrodes, and additional con-
ductive gel (Compumedics Quik-Gel Electrolyte) was
inserted to the scalp-EEG electrodes. The impedance
of all scalp-EEG electrodes was measured to ensure
their values were below 10 kΩ. Otherwise, more gel
would be added. As the materials for the scalp EEG
and the ear electrodes were different, the impedance
was controlled within 50 kΩ for the ear electrodes.
During the experiment, participants were seated
in front of a computer monitor with stimuli presen-
ted. Five subjects completed at least ten blocks of tri-
als, the sixth subject finished five blocks. Each block
contained sixteen 10 s long trials. At the beginning,
a fixation cross presented in the center of the mon-
itor for 3 s. A short warning beep was played 2 s after
the cross onset to call the subjects’ attention for the
task. When the fixation cross disappeared, an arrow
2
X Wu et al
displayed on the monitor, pseudo-randomly point-
ing to the right or left. During the 4 s arrow presenta-
tion, subjects were instructed to make a left- or right-
handed fist according to the arrow direction. Blinking
and swallowing were not allowed during the 10 s trial,
and eye movement should be appropriately avoided.
Figure 2 provides a schematic diagram of the trial.
The scalp-EEG was recorded from the 122-
channel cap, using the Neuroscan Quick Cap (Model
C190). The reference electrode from the scalp REF
channel and the ground electrode from the scalp GRD
channel was shared by the ear and scalp electrodes.
Figure 3 represents the position of the electrodes for
the scalp and ear EEG. In-ear electrophysiological sig-
nals and scalp-EEG were recorded simultaneously.
The recording amplifier was SynAmps2, and Curry
7 was used for real-time data monitoring and col-
lecting. The signals were sampled at 1000 Hz then
filtered with a band-pass filter between 0.5 and 100 Hz
together with a notch filter to suppress the line noises.
We evaluated the in-ear electrophysiological sig-
nals by measuring the difference of the channel cor-
relation and the PSD between each ear channel, the
adjacent scalp channels, and the motor-related chan-
nels. All raw EEG signals were offline analyzed, and
details were explained as follows.
Data were downsampled from 1000 to 250 Hz,
then filtered with a 4–38 Hz FIR filter. The FP1 chan-
nel from subject 4 was broken and therefore needed to
be removed and interpolated by the remaining scalp
channels. Electrical signals were re-referenced to ref-
erence electrode standardization technique, as pro-
posed by Yao in 2001 [18]. Segmental extraction was
applied to segment the continuous raw data [−2, 6]
s according to the time point when the arrow star-
ted to display. Epoch was rejected if it contained data
points higher than 6σ for single-channel and 2σ for
all channels. Independent component analysis was
performed on the remaining epochs for artifact elim-
ination by ICLable [19]. All processing was carried
out using EEGLAB version 2019.0.
For measuring the PSD differences, all in-ear elec-
trophysiological signals, the adjacent scalp channels
(T9, T10, figure 3 in blue color) and the motor-related
channels (C3, C4, figure 3 in red color) were selec-
ted. The mean power of the theta (4–8 Hz), alpha
(8–13 Hz), beta (13–30 Hz), low gamma (30–38 Hz)
frequency bands were respectively calculated by the
8 s (−2 ~ 6 s) epoch segments and averaged from
all epochs. Then the mean power of left/right in-
ear biopotential was divided by the adjacent T9/T10
channels to obtain the in-ear biopotential to adja-
cent scalp EEG power ratio. And the mean power
of left/right the in-ear biopotential was divided by
the motor-related C3/C4 channels to obtain the in-
ear biopotential to motor-elated scalp EEG power
ratio. For the channel correlation, we calculated the
Pearson’s correlation coefficient for each of 8 ear elec-
trodes with 122 scalp electrodes from each subject,
J. Neural Eng. 17 (2020) 066010
Figure 1. Example of the earpiece used in the study.
Figure 2. Schematic diagram of a task.
and the topographic shape of the ear-scalp channel
correlation varied in the range −1 to 1. We used a
permutation test to evaluate the channel correlation
between the in-ear electrophysiological signals and
the motor-related channels (C3 and C4) and see if
the correlation coefficients were statistically differ-
ent from a chance level. In order to maintain tem-
poral structure, data points of each channel within the
same trial was regarded as a whole. We did the per-
mutation test by randomized the trials for 5000 times
and calculated the statistic, i.e. the correlation coeffi-
cient between the in-ear electrophysiological signals
and the motor-related channels, for individual sub-
jects during each permutation. Then we estimated the
p-value for the correlation coefficient calculated from
the real data.
2.2. Classification
2.2.1. Classifier training
For the motor task classification, we applied a com-
pact CNN model (EEGnet) proposed by Lawhern et al
[20]. EEG signals were downsampled to 250 Hz and
bandpass filtered between 4 and 38 Hz, and then an
electrode-wise exponential moving standardization
was executed as introduced in Schirrmeister et al [21].
To compare the classification accuracy of using in-ear
biosignals and scalp-EEG and analyze the influence of
reference location, the dataset was partitioned as fol-
lows: (1) scalp-EEG data (122 channels) all referenced
to scalp REF electrode; (2) in-ear electrophysiological
signals (6 channels, xF, xB, xOD) referenced to scalp
3
X Wu et al
REF electrode, as ear-REF group; (3) in-ear electro-
physiological signals (6 channels, xF, xB, xOD) ref-
erenced to the ipsilateral xOU channel, as ear-IPSI
group; (4) in-ear electrophysiological signals (6 chan-
nels, xF, xB, xOD) referenced to contralateral xOU
channel, as ear-CONTRA group; 5) near ear-EEG (4
channels, T7, T8, T9, T10) referenced to the scalp REF
electrode, as near ear-EEG group. The 3 s epoch of
signals was extracted, starting 1 s before and 2 s after
the presentation of the arrow, see figure 2. Bad epochs
were rejected if any channel has peak-to-peak amp-
litude higher than 800 uV. All EEG data preprocessing
was performed using the Python MNE package [22].
2.2.2. Classification—EEGNet
The EEGNet comprises three building blocks. The
first block is applied with F 2D convolutional filters of
size (1, 10) to learn temporal features, then a depth-
wise convolution of size (C, 1) to learn spatial filters
(C is the number of channels). Followed is a 2D max-
pooling layer of size (1, 3) used to reduce the camp-
ing rate. In the second block, 2∗ F separable convo-
lution filters of size (1, 10) and a 2D max-pooling
layer of (1, 3) are applied. The third block is a clas-
sification layer with a softmax function. In all of the
layers except the final dense layer, the batch normal-
ization, rectified linear unit activation function, and
dropout are applied to prevent overfitting. By manual
fine-tuning, we set F = 25, D (depth multiplier) = 2,
dropout = 0.5 and learning rate = 0.001. See table 1
for more details of EEGNet architecture used in this
study. To fit the model, we used Adam optimizer to
J. Neural Eng. 17 (2020) 066010 X Wu et al

Figure 3. Projected scalp and ear electrode positions for EEG. The blue are scalp electrodes close to ear (T7, T8, T9, T10); the red
are motor-related electrodes (Cz, C1–6); the bold black are ear electrodes (xF, xB, xOU, xOD); the bold black italic are the
reference electrode (REF) and the ground electrode (GRD).

Table 1. EEGNet architecture.

Block Layer #Filters Size Output Options

1 Input (C, T, 1)
Conv2D F (1, 10) (C, T-9, F) mode = valid
Activation (C, T-9, F) ReLU
DepthwiseConv2D D∗ F (C, 1) (1, T-9, D∗ F) (1, T-9, D∗ F)
Activation (1, T-9, D∗ F) ReLU
BatchNorm (1, T-9, D∗ F)
MaxPooling2D (1, 3) (1, (T-9) // 3, D∗ F) mode = valid
Dropout (1, (T-9) // 3, D∗ F) p = 0.5
2 SeparableCon2D 2 ∗F (1, 10) (1, (T-9) //3 - 9, 2 ∗ F) mode = valid
Activation (1, (T-9) //3 - 9, 2 ∗ F) ReLU
BatchNorm (1, (T-9) //3 - 9, 2 ∗ F)
MaxPooling2D (1, 3) (1, T // 9 - 4, 2 ∗ F) mode = valid
Dropout (1, T // 9 - 4, 2 ∗ F) p = 0.5
Flatten (2 ∗ F ∗ (T // 9 - 4))
3 Dense N N
Softmax N
Note: C = number of channels, T = number of time points (T = 750), F = number of filters (F = 25), D = depth multiplier (D = 2),
N = number of classes (N = 2).

minimize the categorical cross-entropy loss function
and an early stop method to save the best model. All
models were trained in a NVIDIA GeForce GTX 1660,
with CUDA 10, Tensor flow 2.0.
2.2.3. Classification accuracy
Movement task classification was evaluated by
three sets of analyses: within-subject classification,
cross-subject classification, and individual sub-
ject retraining after cross-subject classification. In
within-subject classification, the data from each sub-
ject were split into the training sets and testing sets.
We used five-fold cross-validation, with four blocks
of data as the training set, one block as the testing
set. The average classification accuracy was calculated
across all folds and all subjects for the within-subject
model. For the cross-subject classification, we chose
five-subject data as the training set and the leave-out
subject as the testing set. This process was repeated
six times to include all subjects. The average classi-
fication accuracy was calculated across all subjects.

4
J. Neural Eng. 17 (2020) 066010 X Wu et al

Figure 4. Simultaneously recorded in-ear biopotential and scalp EEG signals after down sampling, filtering, and artifacts removal.
The upper signals were one sample epoch from the T9 channel of the scalp EEG, and the lower one was from the LF channel of
in-ear biopotential.

Figure 5. The correlation between ear channels and selected scalp channels during motor tasks (−2 to 6 s). (a) The correlation
coefficients with channel T9, Oz, T10 for all subjects; (b) the correlation coefficients with motor-related channel C3, C4 for all
subjects; (c) the correlation map between ear channels and selected scalp channels (T9, T7, C5, C3, C1, CZ, C2, C4, C6, T8, T10)
example from subject 6. (d) The correlation topography for ear channels, example from subject 6.

The individual subject retraining after cross-subject
classification had additional retraining steps. The
five-subject data were split into a training set (80%)
and testing set (20%), the EEGNet model was trained
using the five-subject training set, and the models’
weights were chosen based on the best testing accur-
acy. Then the leave-out subject data, which also split
into a training set (80%) and testing set (20%), was
retrained on the five-subject model weights. This pro-
cess was also repeated six times to include all subjects.
The average classification accuracy was calculated
across all subjects.
We compared the performance using one-way
ANOVA to evaluate the five data groups as indi-
vidual input in the three sets of analysis. We applied
the Tukey multiple comparison test to compare
the difference between each pair of means, and a
p-value of less than 0.05 was considered statistically
significant.
2.2.4. Feature visualization
In order to understand what are the essential fea-
tures for the classification, we used the deep learning
important features (DeepLIFT) method to calculate
the importance scores for the first input layer of
the EEGNet. DeepLIFT is a gradient-based relevance
attribution method that evaluates how the input fea-
tures contribute to the outcome, compared to the ‘ref-
erence’ input [23]. We here set the reference input
as zeros and calculate the importance scores based
on the specific class (left/right), averaging across the
epochs with good classification results (classification
confidence > 70%) from the testing set. Increasing the
importance score with a positive sign would lead to

5
J. Neural Eng. 17 (2020) 066010 X Wu et al

Figure 6. Channel correlation coefficient between the ear channels and the motor-related channels C3(a) and C4(b). The
shuffling data distribution were from 5000 permutations. The solid lines indicate the channel correlation coefficient calculated
from real data. Comparing the channel correlation coefficient from the real data with the permutation distribution, all channel
correlation had p-value less than 0.001. Example from subject 1.

Figure 7. The power spectral density differences for in-ear biopotential and selected scalp channels. (a) The in-ear biopotential to
adjacent scalp EEG power ratio boxplot for all subjects; (b) The in-ear biopotential to motor-related scalp EEG power ratio
boxplot for all subjects; (c) The median power spectral density for in-ear biopotential (all eight channels) and adjacent scalp EEG
(T7, T8, T9, T10) in solid line, with 25%–75% quantile in dashed line. Example from subject 6.

a higher possibility in that class while increasing the
score with a negative sign would reduce the possibility
of that class.
3. Results
3.1. Data recordings
The filtered in-ear electrophysiological signals and
the adjacent scalp EEG signals are shown in figure 4.
We first investigated the channel correlation between
in-ear electrophysiological signals and the adjacent
scalp EEG channels (T9 is the adjacent channel to
the left ear channels; T10 is the adjacent channel
to the right ear channels; all channel locations are
shown in figure 3). We found that the left/right ear
channels were positively correlated to their adjacent
channels, respectively. The correlation coefficients
with the adjacent channels were closed to 1. In con-
trast, the correlation coefficients with the Cz chan-
nel (located near the center of the scalp) were around
−0.5, and the correlation coefficients with the con-
tralateral channels were around zero (figure 5(a)).
The channel correlation map (figure 5(c)) and the
topography for the ear-correlation coefficients with
all scalp channels (figure 5(d)) showed the in-ear elec-
trophysiological signals were positively correlated to
the adjacent scalp channels. Although the channel
correlation coefficients between in-ear biopotential
and the motor-related channels were small (as shown
in figure 5(b)), the permutation test showed that all
the correlation coefficients were significantly different
from shuffling data (p < 0.001, figure 6), except for the

6
J. Neural Eng. 17 (2020) 066010 X Wu et al

Table 2. Accuracy(%) for within-subject classification.

Subject
EEG signal 1 2 3 4 5 6 Average

Scalp EEG 99.58 89.37 76.79 64.36 94.03 85.41 84.93

ear-IPSI 52.35 50.24 49.41 33.75 49.09 54.58 48.24
near ear-EEG 67.92 80.63 58.87 41.09 63.27 62.08 62.23
ear-REF 64.02 77.00 44.71 43.75 54.09 60.42 57.33
ear-CONTRA 48.95 41.40 48.24 46.25 42.73 49.58 46.19

Figure 8. 5-fold within-subject classification performance averaged over all folds and all subjects for the ear-CONTRA group, the
ear-REF group, the near ear-EEG group, the ear-IPSI group, and the scalp EEG group. The dashed horizontal line indicates the
random classification with 50% accuracy. There was a significant difference between the five groups for the within-subject
classification accuracy (F(4,25) = 13.06, p < 0.001). Tukey’s post-hoc tests were used to compare the differences between each
pair of groups. ∗ indicates p < 0.05; ∗∗ indicates p < 0.01; ∗∗∗ indicates p < 0.001; No stars indicates p > 0.05 in that pair of
comparison.

correlation coefficient between the LOU-C3 channels
in subject 3 (p = 0.087) and subject 4 (p = 0.72).
We then compared the PSD between the in-
ear biopotential and the selected scalp EEG chan-
nels (including the adjacent channels: T9, T10; and
the motor-related channels: C3, C4). We found that
the in-ear electrophsyiolgical signals to the adjacent
channels power ratios were basically near 100% in all
frequency bands for all subjects, except the ear chan-
nel ROU, which is around 150% in all four frequency
bands (figure 7(a)). The in-ear biopotential to the
selected motor channels power ratios were varied
from 10% to 50% in frequency bands for all subjects
(figure 7(b)).
the EEGnet separately. The within-subject five-fold
cross-validation results are shown in figure 8 and
table 2. We found that the scalp EEG group, the near-
ear EEG group and the ear-REF group have average
prediction accuracy higher than the random classific-
ation level, while the other two groups have lower or
around 50% prediction accuracy. There was a statist-
ically significant difference between the five groups.
The scalp EEG group has better prediction accur-
acy than the other four groups. The poor perform-
ance shown in the within-subject classification may
be caused by the limited training samples from indi-
vidual subjects and fewer channels were used in the
in-ear biopotential recordings.

3.2. Classification results

We evaluated the classification performance of the
in-ear electrophysiological signals and scalp EEG, in
three different conditions: within-subject classifica-
tion, cross-subject classification, and individual sub-
ject retraining after cross-subject classification.
3.2.1. Within-subject classification
The within-subject classification was used to evalu-
ate the classification performance of individual sub-
ject’s in-ear biopotential. The ear-REF group, ear-
CONTRA group, ear-IPSI group, near-ear EEG group
and scalp EEG group, were used as the input for
3.2.2. Cross-subject classification
In the cross-subject classification, the classification
result of the leave-out subject was predicted dir-
ectly on the EEGnet weights trained by other sub-
jects. Similar to the within-subject analysis, cross-
subject classification results also showed a statistical
difference between the EEG groups. The prediction
accuracy using the scalp EEG group was significantly
higher than the other four groups (figure 9, table 3).
However, the prediction accuracy of the three in-ear
biopotential groups was close to the random classific-
ation level. The unsatisfactory performance in these

7
J. Neural Eng. 17 (2020) 066010 X Wu et al

Table 3. Accuracy(%) for cross-subject classification.

Subject
EEG signal 1 2 3 4 5 6 Average

Scalp EEG 92.08 80.00 79.49 62.26 91.88 75.00 80.12

ear-IPSI 50.63 47.13 48.24 50.00 47.27 50.42 48.95
near ear-EEG 48.95 60.51 45.88 47.50 55.45 52.50 51.80
ear-REF 50.63 50.32 54.12 46.25 51.82 52.08 50.87
ear-CONTRA 60.26 75.14 48.23 43.75 59.09 64.58 58.51

Figure 9. Cross-subject classification performance averaged over all subjects for the ear-CONTRA group, the ear-REF group, the
near ear-EEG group, the ear-IPSI group, and the scalp EEG group. The dashed horizontal line indicates the random classification
with 50% accuracy. There was a significant difference between EEG groups for the cross-subject classification accuracy
(F(4,25) = 16.9, p < 0.001). Tukey’s post-hoc tests were used to compare the differences between each pair of groups. ∗∗ indicates
p < 0.01; ∗∗∗ indicates p < 0.001; No stars indicates p > 0.05 in that pair of comparison.

Table 4. Accuracy(%) for Individual subject retraining after cross-subject classification.

Subject
EEG signal 1 2 3 4 5 6 Average

Scalp EEG 100 96.88 87.50 81.82 95.74 93.75 92.61

ear-IPSI 56.25 56.25 64.71 56.25 50.00 66.67 58.35
near ear-EEG 72.92 81.25 76.47 68.75 68.18 83.33 75.15
ear-REF 77.08 81.25 70.59 62.50 59.09 70.83 70.22
ear-CONTRA 66.67 81.25 64.71 50.00 70.45 75.00 68.01

three groups may be caused by the between-subject
variation.
3.2.3. Individual subject retraining after cross-subject
classification
In order to overcome the between-subject variation
and the limited channel number, we utilized the
concept of transfer learning. We retrained the cross-
subject EEGnet with the 80% samples of the leave-out
subject. The result is shown in figure 10 and table 4.
Interestingly, with individual subject retraining, we
found that the average classification accuracy for all
groups were now higher than the random classific-
ation level. The performance of the ear-REF group,
which was referencing to the scalp REF electrode,
was better than the ear-IPSI group. Although the
classification accuracy for the ear-CONTRA group is
higher than the ear-IPSI group, the statistical analysis
showed no significant difference. The accuracy of
the scalp EEG group and the near Ear-EEG group
were also significantly better than that of the ear-IPSI
groups.
3.3. Feature visualization
DeepLIFT method was applied to the individual sub-
ject retraining model to investigate what features are
essential for the classification [23]. Based on the pre-
diction result from the individual subject retraining
model, we selected epochs with higher confidence
(probability > 0.7) in either one of the two classes.
And then we calculated the importance scores for
each epoch from the two classes using DeepLIFT.
Compared to the ‘reference’ input, the positive sign
of the scores indicates that an increase of the feature
value, which corresponded to the amplitude of signal
recorded from a channel, will increase the prediction

8
J. Neural Eng. 17 (2020) 066010 X Wu et al

Figure 10. The performance of individual subject retraining after cross-subject classification was averaged over all subjects for the
ear-CONTRA group, the ear-REF group, the near ear-EEG group, the ear-IPSI group, and the scalp EEG group. The dashed
horizontal line indicates the random classification with 50% accuracy. There was a significant difference between EEG groups for
the classification accuracy (F(4,25) = 15.5, p < 0.001). Tukey’s post-hoc tests were used to compare the differences between each
pair of groups. ∗ indicates p < 0.05; ∗∗ indicates p < 0.01; ∗∗∗ indicates p < 0.001; ∗∗∗∗ indicates p < 0.0001. No stars indicates
p > 0.05 in that pair of comparison.

probability of that class, while a negative sign of the
scores will be against that class.
We estimated the importance scores by calculat-
ing the means across the high-confidence trials from
the same class. The left or right movements condi-
tions were calculated independently for the ear-REF
group, ear-CONTRA group, and the scalp EEG group
(figure 11). The ear-IPSI group was not considered
because of poor performance in classification.
Results showed that both the ear-REF group,
ear-CONTRA group and scalp EEG group had
changes in importance scores about 0.25 s after arrow
presentation. We observed that the feature scores for
different groups of ear channels (Left: LF, LB, LOD;
Right: RF, RB, ROD) had opposite signs. The sign
of feature score switches between ear channel groups
when the movement switches side. Similar results
were found across all subjects. In the scalp EEG group,
we also found that, among different groups of chan-
nels, the sign of feature scores was opposite, which as
well correlated to the left/right movements.
4. Discussion
In-ear biopotential monitoring provides a portable,
wearable, and comfortable system, overcomes the
uncomfortable and unsuitable limitation of the scalp
EEG, and enables the integration of the human–
computer interface into daily life. In this paper, we
investigated the quality of the in-ear biopotential
by comparing ear-scalp channels’ correlation and
power spectrum. In-ear biopotential recordings were
highly correlated with the adjacent scalp channels,
and showed statistically significant correlation with
the motor-related channels. And most of the in-ear
biopotential presented similar power distribution of
the adjacent scalp-EEG across different frequency
bands. The results substantiated that a high degree of
similarity was implied between the in-ear biopoten-
tial and the adjacent scalp channels.
We then evaluated the possibility of the in-ear
sensing in the application of motor task and human–
computer interface. We classified the 2-class motor
task using the in-ear biopotential and EEG com-
pact network. The performance of the in-ear biosig-
nals in the within-subject classification and cross-
subject classification was much lower than the scalp
EEG. The small trial numbers and the limited chan-
nels used in the in-ear biopotential may cause poor
performance in the within-subject classification. The
large between-subject variation in the in-ear biopo-
tential may lead to the unsatisfactory performance
in the cross-subject classification. Although the poor
performance of the in-ear biopotential groups com-
pared to the scalp EEG groups may be due to the
lower channel correlation between the ear channels
and the motor-related channels. The channel correl-
ation between the in-ear biopotential and the motor-
related channels were statistically different from the
random level. Moreover, satisfactory performance
was shown when we borrowed the concept of trans-
fer learning and retrained the EEGnet using the
individual subjects’ in-ear biopotential. Therefore,
the in-ear biopotential can be used in the 2-class
motor classification study.
In the feature visualization of the EEGnet classi-
fication, the positive feature scores of one category
indicate the increasing amplitude in that channels
would shift the classification probability to that cat-
egory. Although it was commonly believed that the
hand movement is controlled by the contralateral
motor cortex, event related potential (ERP) study in
[24] showed no obvious correlation of ERP amplitude
with the contralateral hand movement. Moreover, the

9
J. Neural Eng. 17 (2020) 066010
Figure 11. Important feature scores of the scalp EEG group ((a),(b)), the ear-REF group ((c),(d)), the ear-CONTRA group
((e),(f)) for the individual subject retraining model calculated by using DeepLIFT. The average scores for high confidence trials of
left movement (left column), and of the right movement (right column). Example from subject 2.
feature scores revealed that the EEG amplitude differ-
ences within the time windows (0 to 0.5 s after cue)
were closely related to the motor task, which indic-
ates the EEGnet classification was not using random
features.
Nevertheless, one limitation was also found when
referencing ear channels to a near reference electrode
(like ipsilateral referencing in our study). The ear-
IPSI group across all classification analyses was poorly
performed. Some general direction information may
be canceled out when referencing to the nearby elec-
trode. Referencing ear channels to a distant electrode
(like scalp referencing and contralateral referencing in
our study) produced better classification results. We
should consider placing the reference electrode of the
in-ear biopotential recording device with a suitable
distance related to the ear channels. We will extend
the investigation of in-ear biopotential monitoring in
the human–computer interface to other applications
in daily life.
5. Conclusion
We have introduced a novel approach to record-
ing the in-ear biopotential with electrodes placed
both on the ear canal and concha. The results of
this study indicated that the in-ear biopotential has
excellent correlation with the adjacent scalp chan-
nels and evaluated the feasibility of constructing a
human–computer interface for 2-class motor task
using the in-ear biopotential recording device. Over-
all, this study strengthens the idea that the in-ear
10
X Wu et al
biopotential has an advantage of greatest conveni-
ence in implementation and portability, and facil-
itates useful application in the real-world. Consid-
erably more work will be needed to determine the
best location for the reference electrode. With fur-
ther development of the in-ear biopotential recording
device, in-ear biopotential based human–computer
interface system is feasible in real-life applications.
ORCID iDs
Xiaoli Wu  https://orcid.org/0000-0002-5651-1673
Wenhui Zhang  https://orcid.org/0000-0002-7672-
3104
Zhibo Fu  https://orcid.org/0000-0003-0099-3201
Roy T H Cheung  https://orcid.org/0000-0002-
0288-7755
Rosa H M Chan  https://orcid.org/0000-0003-
4808-2490
References
[1] Jackson A F and Bolger D J 2014 The neurophysiological
bases of EEG and EEG measurement: a review for the rest of
us Psychophysiology 51 1061–71
[2] Schomer D L and Da Silva F L 2012 Niedermeyer’s
Electroencephalography: Basic Principles, Clinical
Applications, and Related Fields (Oxford: Oxford University
Press)
[3] Teplan M 2002 Fundamentals of EEG measurement Meas.
Sci. Rev. 2 1–11
[4] Nunez P L and Srinivasan R 2007 Electroencephalogram
Scholarpedia 2 1348
J. Neural Eng. 17 (2020) 066010
[5] Looney D, Kidmose P, Park C, Ungstrup M, Rank M L,
Rosenkranz K and Mandic D P 2012 The in-the-ear
recording concept: user-centered and wearable brain
monitoring IEEE Pulse 3 32–42
[6] Nguyen A, Alqurashi R, Raghebi Z, Banaei-Kashani F,
Halbower A C, Dinh T and Vu T 2016 In-ear biosignal
recording system: a wearable for automatic whole-night
sleep staging Proc. of the 2016 Workshop on Wearable Systems
and Applications pp 19–24
[7] Fiedler L, Obleser J, Lunner T and Graversen C 2016
Ear-EEG allows extraction of neural responses in challenging
listening scenarios—a future technology for hearing aids?
2016 38th Annual Int. Conf. of the IEEE Eng. Med. Biol. Soc.
pp 5697–700
[8] Wolpaw J R, Birbaumer N, McFarland D J, Pfurtscheller G
and Vaughan T M 2002 Brain–computer interfaces for
communication and control Clin. Neurophysiol. 113 767–91
[9] Goverdovsky V, von Rosenberg W, Nakamura T, Looney D,
Sharp D J, Papavassiliou C, Morrell M J and Mandic D P
2017 Hearables: multimodal physiological in-ear sensing Sci.
Rep. 7 1–10
[10] Kidmose P, Looney D, Ungstrup M, Rank M L and
Mandic D P 2013 A study of evoked potentials from ear-eeg
IEEE Trans. Biomed. Eng. 60 2824–30
[11] Mikkelsen K B, Kappel S L, Mandic D P and Kidmose P 2015
Eeg recorded from the ear: characterizing the ear-eeg
method Front. Neurosci. 9 438
[12] Kim Y-J, Kwak N-S and Lee S-W 2018 Classification of
motor imagery for ear-eeg based brain-computer interface
2018 6th Int. Conf. on Brain-Computer Interface (BCI) pp 1–2
[13] Kaongoen N and Jo S 2018 An auditory P300-based
brain-computer interface using ear-EEG 2018 6th Int. Conf.
on Brain-Computer Interface (BCI) pp 1–4
[14] Ahn J W, Ku Y, Kim D Y, Sohn J, Kim J-H and Kim H C 2018
Wearable in-the-ear eeg system for SSVEP-based
brain–computer interface Electron. Lett. 54 413–14
[15] Kim Y-J, Kwak N-S and Lee S-W 2018 Classification of motor
imagery for ear-EEG based brain-computer interface 2018
11
X Wu et al
6th Int. Conf. on Brain-Computer Interface (BCI) IEEE
pp 1–2
[16] Oostra K M, Oomen A, Vanderstraeten G and Vingerhoets G
2015 Influence of motor imagery training on gait
rehabilitation in sub-acute stroke: a randomized controlled
trial J. Rehabil. Med. 47 204–9
[17] Caligiore D, Mustile M, Spalletta G and Baldassarre G 2017
Action observation and motor imagery for rehabilitation in
Parkinson’s disease: a systematic review and an integrative
hypothesis Neurosci. Biobehav. Rev. 72 210–22
[18] Yao D 2001 A method to standardize a reference of
scalp eeg recordings to a point at infinity Physiol. Meas.
22 693
[19] Pion-Tonachini L, Kreutz-Delgado K and Makeig S 2019
Iclabel: An automated electroencephalographic independent
component classifier, dataset and website Neuroimage
198 181–97
[20] Lawhern V J, Solon A J, Waytowich N R, Gordon S M,
Hung C P and Lance B J 2018 Eegnet: a compact
convolutional neural network for eeg-based brain–computer
interfaces J. Neural Eng. 15 056013
[21] Schirrmeister R T, Springenberg J T, Fiederer L D J,
Glasstetter M, Eggensperger K, Tangermann M, Hutter F,
Burgard W and Ball T 2017 Deep learning with
convolutional neural networks for brain mapping and
decoding of movement-related information from the human
eeg arXiv: 1703.05051
[22] Gramfort A et al 2013 MEG and EEG data analysis with
mne-python Front. Neurosci. 7 267
[23] Shrikumar A, Greenside P and Kundaje A 2017 Learning
important features through propagating activation
differences Proc. of the 34th Int. Conf. on Machine Learning
vol 70 pp 3145–53
[24] Huong N T M and Linh H Q et al 2017 Classification of
left/right hand movement eeg signals using event related
potentials and advanced features Int. Conf. on the
Development of Biomedical Engineering in Vietnam pp
209–15