Professional Documents
Culture Documents
4rd Renan Arthur Tourinho 5th Bruno D. Gomes 6th Schubert R. Carvalho
Vale S/A Federal University of Para Vale Institute of Technology
Canaa dos Carajas, Para Belem, Para Belem, Para
renan.arthur.tourinho@hotmail.com brunodgomes@yahoo.com.br schubert.carvalho@itv.org
Abstract—At the mining industry, human safety and pro- critical challenge for EEG decoding is choosing the relevant
ductivity are both desirable in the logistics pipeline. Since the frequency band(s) that can improve the discrimination of
operation of heavy machines requires continued vigilance and mental states. The power spectral density (PSD) is one of
mental activity, fatigue caused by long hours of work and
constant effort generally occurs in this environment. In general, the most used techniques to analyze mental states from EEG
mental fatigue is related to a loss of efficiency, leading to a data because it estimates how the total power is distributed
decrease in productivity and inducing critical errors, which over frequencies [5], [6]. The PSD provides a way to analyze
can provoke equipment breakups or accidents with human changes in the magnitude of the signal through different
victims. At this high cognitive workload environment, there is frequency components [5], [6]. Cajochen et al. [7] used the
a need for the development of robust monitoring techniques
aiming to predict mental fatigue before workers’ movement PSD values of central EEG electrodes positioned at frontal
responses become slower, more variable, and more error-prone. and parietal regions in theta and alpha bands to define a
In this work, we introduce a residual multilayer perceptron set of biomarkers to describe mental workload from EEG
(MLP) network (ResMLPNet) and assess its performance in signals. They found that the PSD magnitudes were positively
the challenging problem of mental fatigue classification from associated with the increasing of mental workload. Chai et
cognitive electrophysiology data, acquired during Virtual Reality
(VR) training sessions mimicking a real operation faced by al. [5] found that using only frontal lobe electrodes were
excavator workers at the mining industry. In a three-step training sufficient to discriminate alert states from fatigue states from
strategy, the ResMLPNet achieved slightly better classification EEG signals.
accuracies compared to its plain MLP architecture. The efforts to detect neural patterns that characterize the
Keywords-Residual multilayer perceptron network, classifica- process of cognitive decline are usually taken in controlled
tion, mental fatigue, EEG time series experiments, in which are specified paradigms for inducing the
desired mental state in cognitive tasks that require wakefulness
I. I NTRODUCTION with sustained attention [1], visual attention [8] and intense use
The operation of heavy machines in the mining industry of work memory [9]. In this context, Min et al. [10] investigated
requires a balance between safety and productivity, which imply the brain regions directly active during the establishment of
continued vigilance that can lead to mental fatigue during the fatigue in drivers in a simulated environment, in a period of
long operation sessions. Mental fatigue is a natural result of up to 2 hours. The first and last five minutes of the recordings
high metal workload demand or prolonged mental effort during were used as the normal and fatigue mental states. By using
the execution of a laborious task. This process induces a decline entropy-based measures of 30 channels, they present evidence
in cognitive performance, which is, among other factors, mostly that with the monitoring of amplitude values from 13 of those
responsible for accidents [1]. This condition usually modifies electrodes is sufficient to distinguish fatigued and non-fatigued
a mental behavior, which can be observed by changes in brain state.
rhythms associated with theta, alpha, and beta frequencies of the The study of Eoh, Chung, and Kim [3] examine the EEG
electroencephalographic (EEG) oscillation. [2], [3]. Changes in changes in fatigued subjects while performing a simulated
the activity of cortical regions, mostly the frontal and parietal driving task. They recorded data from 8 EEG channels
sites, also happen [4]. positioned at frontopolar, temporal, parietal, and occipital
The analysis and classification of mental fatigue from the regions during 50 min. The data were recorded after a night
EEG time series pose serious challenges. Some of the reasons of sleep deprivation. They reported that the beta and alpha
are the low signal-to-noise ratio, highly subject-specific data, frequencies provided statistically significant evidence for mental
and the stochasticity associated with these signals. Another alertness monitoring.
408
calculates the Fourier coefficients [12]. It also circumvents the
effects of the Heisenberg uncertainty principle, which deals
with the troubled relationship of the resolution of the same
signal in time and frequency.
(a) Plain MLP
B. Feature vector
The feature vector combines the average PSD index in each
electrode of the theta, alpha,and beta waves.
Formally, there
is the index vector Fpsd = Cα , Cβ , Cθ ∈ R3ζ , where ζ is
the number of analyzed electrodes. Cα ∈ Rζ , Cβ ∈ Rζ and (b) ResMLPNet
Cθ ∈ Rζ representing the PSD values in each electrode and
band.
This proposal aims to apply the knowledge acquired from
the benchmark database to a particular database, but that the
number of useful electrodes in both databases is different,
ζbenchmark = 30 and ζdatasetII = 19, and to make the data (c) ResMLPNet 1
comparable, the 11 missing electrodes in dataset II are removed
from the benchmark database. Hence, each trial is represented
by a vector such as Fpsd ∈ R57 :
Fpsd = Cα , Cβ , Cθ (d) ResMLPNet 2
409
Because cleaner paths generally improve classification accuracy, The experiments were tested in a system with Windows
differently from [15], [14], the linear mapping is performed 10 Pro, Intel Core i7-7820HK CPU 2.90Gh, 32 GB RAM,
on the residual function instead of xl . In this work, Ws is a and GeForce GTX 1070 graphic card, and the codes were
pooling layer. implemented in Python using the Pytorch platform.
B. Network Architectures A. Dataset I Classification
Figure 2 shows the four architecture models used in this
work. For comparison reasons, Figure 2(a) depicts the plain
network architecture of the ResMLPNet, i.e., without skipped
connections. The term "plain" used to describe the network
without residual layers follows the nomenclature employed by
He et al. [15]. Figures 2(b) and 2(c) shows two ResMLPNet
versions of the plain network. The first with skipped connec-
tions at every 2 fully-connected layers and the second with
skipped connections at every 4 fully-connected, respectively.
In the first and second ResMLPNets, the addition is performed
before the activation function ReLU. The third architecture
shown in figure 2(d) is similar to the network in Figures 2(b),
but the difference is that additions are performed before the
ReLU activation. The network output is a softmax function.
C. Training Protocol
All the three steps described below were followed when Fig. 3. Accuracy and loss of Training in the first stage
training all proposed models.
a) First step: In this phase, dataset I were used as a The training of the models using data from Min et al. [10]
benchmark to classify the two mental states. 80% of the dataset and the process described in the first step took, on average, three
was used for training and the remaining 20% was used for minutes to achieve accuracies close to 100%. The ResMLPNet
validation. The initial learning rate was set to 1 × 10−2 , and took 84 epochs to converge, while its ‘2’, 182. The Figure 3
was reduced by a factor of 10 when the validation loss has presents the four models performances, where the main axis
stopped decreasing by at least 1 × 10−4 for two consecutive in blue pictures the accuracy, while the secondary axis shows
epochs. If the validation loss does not improve after 50 epochs the model loss in red. Solid lines represent values of training
the training process stopped. and the dashed ones, the validation assessments.
b) Second step: According to the experiment design, the
dataset II supposedly comprises the two mental states. To test B. Dataset II Classification
this hypothesis 80% of dataset II was used to adjust the learned
weights from the models of the previous phase. The training
proceeded without regularization and using the same initial
learning rate without decay.
c) Third step: The last phase takes the overtrained model
and regularizes its weights to better generalize the datasets.
At this stage, the reduce-on-plateau strategy is employed, as
well as the validation step and the early stopping. The training
lasts at most 300 epochs or when the validation loss stopped
decreasing. 80% of dataset II was used to train the networks,
10% of dataset II was used for validation, and the remaining
10% of the samples was used as a test set.
V. E XPERIMENTS
In this paper, we investigate the potential of identifying two
kinds of mental states which can be related to the occurrence of
mental fatigue. To do so, we used a three-step learning strategy, Fig. 4. Accuracy and loss of Training in the second stage
being the first one supported by the so called dataset I which
comprising data labeled as mentally normal and fatigued. The The second stage aims to train the model until it overfits,
second and third learning steps used the dataset II, which was starting from the model trained in the first step. It used 80%
divided into three parts, train, validation, and test. The metrics of the training data from dataset II, with any mechanisms of
of overall accuracy, recall, and precision were used to evaluate regularization enabled. It took approximately 5 minutes to train
the results. the model until 300 epochs, as pictured in Figure 4. The plain
410
TABLE I
ACCURACIES OF TRAIN AND VALIDATION OF DIFFERENT ARCHITECTURES
MLP took just a few epochs to reach a plateau around which correctly and incorrectly classified. The last row in the table
it ranged until the last epoch. The other models presented a represents the true positive ratio and the false negative rate,
smoother curve with progressive accuracies rates, achieving also known as recall measures.
accuracies close to 95%. The accuracy of the plain MLP was
VI. D ISCUSSIONS
of 88.6%.
TABLE II TABLE IV
C ONFUSION MATRIX OF THE R ES MLPN ET TEST C ONFUSION MATRIX OF THE R ES MLPN ET 1
The last training step enabled back the method for over- The indices and accuracies presented so far are the result
fitting prevention of the first stage. The Table I presents the of an innumerable amount of tests. Several architectures
assessments of the four models in train Validation and test, in and configurations led to the models presented here. The
which are highlighted the best performances at each phase. ResMLPNet and its two variants, and its plain version achieved
The ResMLPNet have achieved the higher accuracies in similar values of accuracy and loss during train, as well as
train and test, 86.58% and 80.69%, respectively. The 2 of during the validation and test. The tests used the same initial
ResMLPNet have presented similar assessments of train and learning rate of 1e-3, the weight decay rate of 1e-1, and dropout
validation, but its test’s accuracy may indicate the model rate (when applicable) of 10%.
overfitting. The plain MLP performed concisely, including The primary source of confusion and misclassification
giving the lower value of loss in validation dataset. of the plain MLP model came from the data representing
The Table II presents the test assessments of the best model, the non-fatigued state (Table III). The misclassification of
pictured by the confusion matrix of this result. As usual, the 177 over 725 samples represents 46 more samples than the
main diagonal presents the model’s correct assumptions, and misclassification over the fatigued observations. In absolute
the off-diagonal cells correspond to incorrectly observations. numbers, the misclassification by the ResMLPNet, in the same
The test dataset comprises a total of 725 samples from class order, occurred in 138 and 140 samples, i.e., 40 more examples
normal and 715 samples from the class fatigue. The model that were correctly assigned. The ResMLPNet 1 (Table IV)
precision, given in the far right column, shows the percentages misclassified 187 samples from the assumed fatigued data, but
of all the samples predicted to belong to each class that is the ResMLPNet 2 (Table V) achieved the worst performance
411
in classifying these samples, 207 samples. However, this model ChMCTI/CNPq/VALE-ITV (443304/2015-7), CNPq/ITV 2018
misclassified only 7.8% of the 725 samples of the first period. (402764/2018-8) and DT-2018 (315462/2018-3).
The performance of classification of the dataset I overlapped
R EFERENCES
the results presented in [10], 98.3%, in all models implemented.
It can support the following results presented when classifying [1] G. N. Dimitrakopoulos, I. Kakkos, Z. Dai, H. Wang, K. Sgarbas,
N. Thakor, A. Bezerianos, and Y. SUN, “Functional connectivity analysis
the dataset II. The fact of the data of excavator operation has of mental fatigue reveals different network topological alterations between
been successfully differentiated the signals at different times driving and vigilance tasks,” IEEE Transactions on Neural Systems and
also indicates the reliability of the raised assumption. Rehabilitation Engineering, vol. 4320, no. c, pp. 1–10, 2018.
[2] A. Holm, K. Lukander, J. Korpela, M. Sallinen, and K. M. Müller,
The misclassification in dataset II may lie on the fact that “Estimating brain load from the eeg,” The Scientific World Journal, vol. 9,
several factors are biasing this particular experiment. Unlike pp. 639–651, 2009.
from a simulating driving task, the actual simulated task is [3] H. J. Eoh, M. K. Chung, and S.-H. Kim, “Electroencephalographic study
of drowsiness in simulated driving with sleep deprivation,” International
directly related to professional questions, besides the personal Journal of Industrial Ergonomics, vol. 35, no. 4, pp. 307–320, 2005.
ones. The simulation environment is another way to evaluate the [4] M. A. Schier, “Changes in eeg alpha power during simulated driving:
employees in the mining company; thus there is a psychological a demonstration,” International Journal of Psychophysiology, vol. 37,
no. 2, pp. 155–162, 2000.
factor that doesn’t exist in another context. Besides, some of [5] R. Chai, Y. Tran, A. Craig, S. H. Ling, and H. T. Nguyen, “Enhancing
the subjects, mainly the older ones, are reticent to the simulated accuracy of mental fatigue classification using advanced computational
excavation. Some of them reported nausea or another physical intelligence in an electroencephalography system,” in 2014 36th Annual
International Conference of the IEEE Engineering in Medicine and
discomfort during the section. Biology Society, Aug 2014, pp. 1318–1341.
Another noted aspect is the fact that the operators where [6] Z. Yin and J. Zhang, “Identification of temporal variations in mental work-
chose during their work shift. They work in turns of twelve load using locally-linear-embedding-based EEG feature reduction and
support-vector-machine-based clustering and classification techniques,”
hours for three days before a rest period of other three days. Computer Methods and Programs in Biomedicine, vol. 115, no. 3, pp.
The workers were authorized to experiment according to 119–134, 2014.
guidelines of company; hence the data collection occurs with [7] C. Cajochen, D. P. Brunner, K. Krauchi, P. Graw, and A. Wirz-Justice,
“Power density in theta/alpha frequencies of the waking eeg progressively
no standardization over the day in the shift they were. It is increases during sustained wakefulness,” Sleep, vol. 18, no. 10, pp. 890–
possible to suppose that a person in its first day in a row deal 894, 1995.
with the task differently than a person on their last day in a [8] J. Li, J. Lim, Y. Chen, K. Wong, N. Thakor, A. Bezerianos, and Y. Sun,
“Mid-Task Break Improves Global Integration of Functional Connectivity
shift. Also, it is essential to highlight that a person working in Lower Alpha Band,” Frontiers in Human Neuroscience, vol. 10, no.
during the night, i.e., with his circadian cycle shifted, may June, pp. 1–12, 2016.
have its performance altered somehow. It is an excellent point [9] W. S. Helton and P. N. Russell, “Working memory load and the vigilance
decrement,” Experimental Brain Research, vol. 212, no. 3, pp. 429–437,
to be considered in future works. Analyze the dataset in terms 2011.
of work period (day and night) or in terms of day in the shift [10] J. Min, P. Wang, and J. Hu, “Driver fatigue detection through multiple
work are available possibilities to understand the nature of this entropy fusion analysis in an EEG-based system,” PLOS ONE, vol. 12,
no. 12, p. e0188756, dec 2017.
new dataset. [11] R. Oostenveld, P. Fries, E. Maris, and J.-M. Schoffelen, “FieldTrip: Open
Source Software for Advanced Analysis of MEG, EEG, and Invasive
VII. C ONCLUSION Electrophysiological Data,” Computational Intelligence and Neuroscience,
vol. 2011, 2011.
Mental fatigue is related to a decrease in productivity in the [12] P. Mitra and B. Pesaran, “Analysis of dynamic brain imaging data,”
workplace and may induce critical errors. The present work Biophysical Journal, vol. 76, no. 2, pp. 691 – 708, 1999.
has introduced a residual multiple layer perceptron (MLP) [13] A. C. Siravenha and S. R. Carvalho, “Plant classification from leaf
textures,” in Digital Image Computing: Techniques and Applications
network (ResMLPNet) and a three-steps training methodology (DICTA), 2016 International Conference on. IEEE, 2016, pp. 1–8.
to assess the mental fatigue pattern in a private dataset. The [14] H. Kaiming, Z. Xiangyu, R. Shaoqing, and S. Jian, “Identity mappings
monitored activity mimic a real operation of ore excavation in in deep residual networks,” in Computer Vision – ECCV 2016, B. Leibe,
J. Matas, N. Sebe, and M. Welling, Eds. Cham: Springer International
a VR environment. The results have shown that ResMLPNet Publishing, 2016, pp. 630–645.
is efficient for the classification of structured data. The plain [15] ——, “Deep residual learning for image recognition,” CoRR, vol.
version is as accurate as of the residual one, although its abs/1512.03385, 2015.
robustness must be better evaluated. The authors aims to expand
the present conclusions by the dataset enlargement, and also
including different perspectives to the mental fatigue assessment
in a real application.
VIII. ACKNOWLEDGEMENTS
This study was supported by SENAI Innovation Institute
for Mineral Technologies and Vale Institute of Technology.
It was partially financed in part by the Coordination for the
Improvement of Higher Education Personnel (CAPES) at pro-
cess 88887.141251/2017-00, and also by the National Council
for Scientific and Technological Development (CNPq) at calls
412