sensors
Article
System and Method for Driver Drowsiness Detection Using
Behavioral and Sensor-Based Physiological Measures
Jaspreet Singh Bajaj 1 , Naveen Kumar 1 , Rajesh Kumar Kaushal 1 , H. L. Gururaj 2, *, Francesco Flammini 3, *
and Rajesh Natarajan 4
Citation: Bajaj, J.S.; Kumar, N.;
Kaushal, R.K.; Gururaj, H.L.;
Flammini, F.; Natarajan, R. System
and Method for Driver Drowsiness
Detection Using Behavioral and
Sensor-Based Physiological Measures.
Sensors 2023, 23, 1292. https://
doi.org/10.3390/s23031292
Academic Editors: Lu Xiong, Xin Xia,
Jiaqi Ma and Zhaojian Li
Received: 12 December 2022
Revised: 14 January 2023
Accepted: 18 January 2023
Published: 23 January 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Sensors 2023, 23, 1292. https://doi.org/10.3390/s23031292
1 Chitkara University Institute of Engineering and Technology, Chitkara University, Punjab 140401, India
2 Department of Information Technology, Manipal Institute of Technology Bengaluru, Manipal Academy of
Higher Education, Manipal 576104, India
3 IDSIA USI-SUPSI, University of Applied Sciences and Arts of Southern Switzerland, 6928 Manno, Switzerland
4 Information Technology Department, University of Technology and Applied Sciences-Shinas,
Shinas 324, Oman
* Correspondence: gururaj.hl@manipal.edu (H.L.G.); francesco.flammini@supsi.ch (F.F.)
Abstract: The amount of road accidents caused by driver drowsiness is one of the world’s major
challenges. These accidents lead to numerous fatal and non-fatal injuries which impose substantial
financial strain on individuals and governments every year. As a result, it is critical to prevent
catastrophic accidents and reduce the financial burden on society caused by driver drowsiness.
The research community has primarily focused on two approaches to identify driver drowsiness
during the last decade: intrusive and non-intrusive. The intrusive approach includes physiological
measures, and the non-intrusive approach includes vehicle-based and behavioral measures. In an
intrusive approach, sensors are used to detect driver drowsiness by placing them on the driver’s
body, whereas in a non-intrusive approach, a camera is used for drowsiness detection by identifying
yawning patterns, eyelid movement and head inclination. Noticeably, most research has been
conducted in driver drowsiness detection methods using only single measures that failed to produce
good outcomes. Furthermore, these measures were only functional in certain conditions. This
paper proposes a model that combines the two approaches, non-intrusive and intrusive, to detect
driver drowsiness. Behavioral measures as a non-intrusive approach and sensor-based physiological
measures as an intrusive approach are combined to detect driver drowsiness. The proposed hybrid
model uses AI-based Multi-Task Cascaded Convolutional Neural Networks (MTCNN) as a behavioral
measure to recognize the driver’s facial features, and the Galvanic Skin Response (GSR) sensor as a
physiological measure to collect the skin conductance of the driver that helps to increase the overall
accuracy. Furthermore, the model’s efficacy has been computed in a simulated environment. The
outcome shows that the proposed hybrid model is capable of identifying the transition from awake
to a drowsy state in the driver in all conditions with the efficacy of 91%.
Keywords: artificial intelligence; driver drowsiness; hybrid measures; MTCNN
1. Introduction
Road accidents are one of the most severe transportation issues. As per the latest
report by World Health Organization (WHO), 1.3 million people die from road accidents
every year worldwide. Nearly fifty million people have suffered substantial fatal and
non-fatal injuries, resulting in an immense monetary loss for drivers, victims and their
families. The total cost borne by a country due to road accidents is approximately 3% of
its gross domestic product [1]. According to the 2017 National Highway Traffic Safety
Administration (NHTSA) data, 91,000 vehicle accidents were caused by driver drowsiness.
The NHTSA also concedes that reported numbers are fairly low and that this count could
be considerably higher [2]. Therefore, the research community has made significant contri-
butions towards detecting driver drowsiness. Various measures have been investigated,
https://www.mdpi.com/journal/sensors
Sensors 2023, 23, 1292 2 of 19

such as subjective, vehicle-based, physiological and behavioral measures for driver drowsi-
ness detection [3]. Each of these measures has some limitations, such as an inability to
perform in all conditions. Thus, a robust Driver Drowsiness Detection System (DDDS)
is required to detect the driver’s drowsiness at an early stage. Due to advancements in
artificial intelligence-based technology, revolutionary ideas have been proposed to develop
an efficient system that helps to detect driver drowsiness. A hybrid model can overcome
such limitations by combining two or more measures for drowsiness detection in such a
way that one method can reduce the limitations of others in order to improve the system’s
overall accuracy [4].
This paper aims to propose a model for drowsiness detection using an amalgamation
of behavioral and sensor-based physiological measures that will help to detect a drowsy
state at an early stage. A detailed analysis of the hybrid model has been conducted in all
conditions, i.e., low light, a face with eyeglasses or with a beard, to check the efficacy of the
proposed model.
The rest of the paper is organized as follows: A state-of-the-art of the recent develop-
ment in driver drowsiness detection systems is carried out in Section 2. Section 3 explains
the methodology for detecting driver drowsiness. Section 4 explains the architecture of the
proposed hybrid model. Section 5 discusses the implementation of the hybrid model and
its results. The conclusion is presented in Section 6.

2. Background
2.1. Drowsiness
Drowsiness is the intermediate state between alertness and sleepiness. It is the bio-
logical state of a human being where the intensity of sleepiness is directly proportional to
time. In a drowsy state, it is very difficult to keep one’s eyes open and to concentrate, as
the head or the body is unstable. Frequently yawning is also one of the major signs of a
drowsy state. Thus, driving in a drowsy state leads to vehicle accidents. Therefore, it is
necessary to identify a drowsy state and to notify the driver early to avoid a collision. The
drowsy state is very dangerous for the driver and road commuters [2,5]. Due to a drowsy
state, it is very difficult for drivers to concentrate on the road while driving the vehicle,
which restricts them from making quick decisions to control the vehicle components, i.e.,
brakes and steering [6].
Due to the circadian rhythm, most drivers become drowsy on national highways,
notably late at night between 12:00 AM to 7:00 AM or in the afternoon between 2:00 PM to
4:00 PM [7]. Most of the time, the driver is driving alone on the highway and falls in the age
group of 18 to 30. Previous studies have shown that the young generation is at high risk
of a drowsy state, leading to fatal and non-fatal injuries [8]. Road accidents are unwanted
things that happen quite often while travelling. Tracking road accident indicators like
drunk driving, brake failure, bypassing traffic signals/rules and rash driving is much
easier than accidents caused by the drowsy state of the driver. It is even more difficult to
identify the primary cause of the accident due to a lack of a technical glitch in the vehicle
or favorable road and weather conditions [3].
A smart affordable device is required to identify driver drowsiness at an early stage.
A plethora of research has been done in the past to create a system that can identify driver
drowsiness, but no system is ideal for driver drowsiness detection at an early stage. The
present state of the art suggests that such dangers can be avoided by proposing an advanced
AI-based system using an amalgamation of multiple measures.

2.2. Driver Drowsiness Detection Measures

There are two approaches, intrusive and non-intrusive, to identify drowsy driving. In
an intrusive approach, the drowsy condition of a person is identified using physiological
parameters, but in a non-intrusive approach, this is identified by installing devices and
sensors on the vehicle. Based on intrusive and non-intrusive approaches, five different
 2.2. Driver Drowsiness Detection Measures
There are two approaches, intrusive and non-intrusive, to identify drowsy driving.
In an intrusive approach, the drowsy condition of a person is identified using physiolog-
Sensors 2023, 23, 1292 3 of 19
ical parameters, but in a non-intrusive approach, this is identified by installing devices
and sensors on the vehicle. Based on intrusive and non-intrusive approaches, five differ-
ent measures are utilized to identify driver drowsiness at an early stage [8–10]. These
measuresare
measures areutilized
as follows:
to identify driver drowsiness at an early stage [8–10]. These measures
are as follows:
1. Subjective measures (SM)
1.2. Subjective
Vehicle-based measures
measures (SM)(VBM)
3. Vehicle-based
2. Physiological measures
measures (PM)
(VBM)
4. Behavioral measures (BM)
3. Physiological measures (PM)
5. Behavioral
4. Hybrid measures (HM)
measures (BM)
5. Hybrid measures (HM)
Physiological measures are considered intrusive, whereas subjective, vehicle-based
and Physiological measures
behavioral measures areare considered
considered intrusive, whereas
non-intrusive. Hybridsubjective, vehicle-based
measures are a combina-
and
tion of two or more measures. Figure 1 depicts the various measures for drivercombination
behavioral measures are considered non-intrusive. Hybrid measures are a drowsiness
of two or more
detection measures.
and their Figure
respective 1 depicts the various measures for driver drowsiness
techniques.
detection and their respective techniques.

Figure1.
Figure 1. Driver
Driver drowsiness
drowsinessdetection
detectionmeasures
measuresand
andrespective
respectivetechniques.
techniques.

Subjective
Subjective measures are used
measures are usedtotodetect
detectdriver
driverdrowsiness
drowsinessbyby gathering
gathering data
data from
from the
the driver in a simulated environment. The Stanford Sleeping Scale (SSS) and
driver in a simulated environment. The Stanford Sleeping Scale (SSS) and KarolinskaKarolinska
Sleeping
Sleeping Scale
Scale (KSS)
(KSS) approaches
approacheshelp
helptotogather
gatherthe
thedriver’s
driver’s observations
observations while
while operating
operating
the vehicle to assess the driver’s state. In SSS, seven levels of the Likert scale (1-feeling
the vehicle to assess the driver’s state. In SSS, seven levels of the Likert scale (1-feeling
active to 7-extremely sleepy), and in KSS, nine levels of the Likert scale (1-extremely alert
active to 7-extremely sleepy), and in KSS, nine levels of the Likert scale (1-extremely alert
to 9-extremely sleepy) help to evaluate the different levels of drowsiness at a particular
to 9-extremely sleepy) help to evaluate the different levels of drowsiness at a particular
time. The drawback of a subjective measure is that it is impractical and produces biased
time. The drawback of a subjective measure is that it is impractical and produces biased
results, making it impossible to utilize in real driving conditions [3].
results, making it impossible to utilize in real driving conditions [3].
By installing several types of sensors in various places in the vehicle, such as the
By installing several types of sensors in various places in the vehicle, such as the
driver’s seat and steering wheel, it is possible to apply vehicle-based measures. The
driver’s seat and steering wheel, it is possible to apply vehicle-based measures. The two
two most common vehicle-based measures for driver drowsiness detection are Standard
Deviation of Lane Positioning (SDLP) and Steering Wheel Movement (SWM). In SDLP,
a camera is mounted on the front of the vehicle to track the lane position, which helps
to identify the alert or drowsy state. Its biggest drawback is needing to rely on external
variables like road markings, lighting and weather conditions. In SWM, various sensors
positioned on the vehicle’s steering wheel gather information to aid in the detection of
 most common vehicle-based measures for driver drowsiness detection are Standard De-
viation of Lane Positioning (SDLP) and Steering Wheel Movement (SWM). In SDLP, a
camera is mounted on the front of the vehicle to track the lane position, which helps to
Sensors 2023, 23, 1292 identify the alert or drowsy state. Its biggest drawback is needing to rely on external var-
4 of 19
iables like road markings, lighting and weather conditions. In SWM, various sensors po-
sitioned on the vehicle’s steering wheel gather information to aid in the detection of driver
drowsiness.
driver The primary
drowsiness. issue associated
The primary with SWM
issue associated with SWM is thatisitthat
is expensive and has
it is expensive andahashigha
false positive detection rate, making it ineffective in real driving conditions
high false positive detection rate, making it ineffective in real driving conditions [8]. [8].
Usingphysiological
Using physiologicalmeasures
measures toto identify
identify driver
driver drowsiness
drowsiness at an atearly
an early
stagestage pro-
provides
vides promising results. A number of devices are directly attached to the
promising results. A number of devices are directly attached to the driver to capture the rel- driver to capture
the relevant
evant physiological
physiological parameters,
parameters, such as electroencephalograms
such as electroencephalograms (EEG), electrocar-
(EEG), electrocardiograms
diograms
(ECG), (ECG), electromyograms
electromyograms (EMG) and (EMG) and electrooculograms
electrooculograms (EOG). Although
(EOG). Although phys-
physiological
iological measures have a high level of accuracy, they are very intrusive.
measures have a high level of accuracy, they are very intrusive. Using these highly intru- Using these
highly intrusive devices in real driving conditions is challenging [3]. Therefore,
sive devices in real driving conditions is challenging [3]. Therefore, small and lightweight small and
lightweight physiologically based sensors that are less intrusive, such
physiologically based sensors that are less intrusive, such as a Galvanic Skin Response as a Galvanic Skin
Response
(GSR) (GSR)
sensor, cansensor,
be used canto be usedthe
record to record the physiological
physiological parametersparameters
[11]. A GSR [11]. A GSR
sensor is
asensor is a physiologically
physiologically based sensorbased sensor
that that on
is placed is placed
the skin ontothe skin to
capture thecapture
body’sthe body’s
skin con-
skin conductance
ductance [12]. The suggests
[12]. The literature literaturethat suggests thatdriver
detecting detecting driver drowsiness
drowsiness using a GSRusingsensora
isGSR sensorand
possible is possible
can playand can play arole
a supportive supportive
when the role when themeasures
behavioral behavioral measures
are are
ineffective.
ineffective. Figure 2 shows a flow diagram of the detection of the
Figure 2 shows a flow diagram of the detection of the drowsy state of the driver using a drowsy state of the
driver
GSR using a GSR sensor.
sensor.

Figure2.2.Detection
Figure Detectionof
ofdriver
driverdrowsiness
drowsinessusing
usingaaGSR
GSRsensor.
sensor.

TheGSR
The GSRsensor
sensorisisfirst
firstattached
attachedto tothe
thedriver
driverto tocollect
collectbioelectric
bioelectricsignals.
signals.Following
Following
data acquisition,
data acquisition, it transmits
transmits the the signals
signalstotothe
thenext
nextphase
phasefor
forfeature
featureextraction.
extraction.Based
Based on
thethe
on processed
processed data,
data,further
furtherrawrawdata
dataare
areconverted
converted into
into meaningful
meaningful data by by removing
removing
anomalies/missing frequencies
anomalies/missing frequenciesand andduplicate
duplicatevalues.
values.These
Thesedata areare
data transmitted
transmitted again for
again
classification after analysis. The binary classification technique efficiently classifies the
for classification after analysis. The binary classification technique efficiently classifies the
drowsyor
drowsy ornon-drowsy
non-drowsystate. state.
Behavioral measures
Behavioral measures are arebased
basedon onthe
thedriver’s
driver’sfeatures, such
features, as eyes,
such mouth
as eyes, and head
mouth and
inclination. To identify drowsy driving, the researcher primarily focuses on eye blink rate
head inclination. To identify drowsy driving, the researcher primarily focuses on eye blink
and and
rate percentage of eye
percentage of closure (PERCLOS),
eye closure which
(PERCLOS), are further
which examined
are further by machine
examined learn-
by machine
learning
ing (ML) and deep learning (DL) algorithms [13]. Other signs that can aid in
(ML) and deep learning (DL) algorithms [13]. Other signs that can aid in detecting
detecting
drowsiness
drowsiness in inaadriver
driverinclude
includeyawning
yawningand andhead
headmovement.
movement. Due Due tototheir
theirnon-intrusive
non-intrusive
characteristics, these behavioral measurement techniques are
characteristics, these behavioral measurement techniques are frequently frequently usedused
in simulated
in simu-
and real driving conditions. The present state of the art reveals that behavioral
lated and real driving conditions. The present state of the art reveals that behavioral measures
are more accurate
measures are more than vehicle-based
accurate measures [14].
than vehicle-based measures [14].
Due to its non-intrusiveness, the behavioral measure is one of the most widely used
drowsiness detection techniques [15]. A camera is mounted on the dashboard of the vehicle
to capture the driver’s facial features [14]. Figure 3 depicts the procedure to determine
the drowsy state of the driver using behavioral measures. Three phases comprise the
entire process: data acquisition, feature extraction and classification. The first step in data
acquisition is gathering the driver’s image or video. Thereafter, in the second phase, the
face is detected by applying pre-processing techniques. Through the feature extraction
phase, the region of interest (ROI) is identified. The ROI includes capturing eye, mouth and
 Due Due
to itstonon-intrusiveness,
its non-intrusiveness,the behavioral
the behavioral measure is one
measure is of
onetheofmost widely
the most usedused
widely
drowsiness detection techniques [15]. A camera is mounted on the dashboard
drowsiness detection techniques [15]. A camera is mounted on the dashboard of the vehi- of the vehi-
cle tocle
capture the driver’s
to capture facialfacial
the driver’s features [14]. [14].
features Figure 3 depicts
Figure the procedure
3 depicts the procedure to determine
to determine
the drowsy state of the driver using behavioral measures. Three
the drowsy state of the driver using behavioral measures. Three phases comprisephases comprise the en-
the en-
tire process:
tire process: data acquisition, feature extraction and classification. The first stepdata
data acquisition, feature extraction and classification. The first step in in data
Sensors 2023, 23, 1292 acquisition is gathering
acquisition the driver’s
is gathering image
the driver’s or video.
image Thereafter,
or video. in the
Thereafter, insecond phase,
the second the
phase, the
5 of 19
face face
is detected by applying pre-processing techniques. Through the
is detected by applying pre-processing techniques. Through the feature extractionfeature extraction
phase, the region
phase, of interest
the region (ROI)(ROI)
of interest is identified. The ROI
is identified. includes
The ROI capturing
includes eye, eye,
capturing mouth mouth
and head
head
and headpose using machine
pose using
pose using machinemachineor
or deepdeep learning-based
orlearning-based
deep learning-based algorithms. In
algorithms.
algorithms. the classification
In the classification
In the classification phase,
phase,
the the
phase, binary
binary classification
the binary
classification method
classification
method isisused
method used toevaluate
is to
usedevaluate the drowsy
to evaluate
the or
the drowsy
drowsy ornon-drowsy
or non-drowsy
non-drowsy state
statestate
of
of theofdriver.
the the driver.
driver.

Figure 3. Detection
Figure of driver drowsiness usingusing
behavioral measures.
Figure 3.3.Detection
Detection ofdriver
of driver drowsiness
drowsiness behavioral
using behavioral measures.
measures.

Three driver
Three
Three drowsiness
driver
driver drowsiness
drowsiness detection measures
detection
detection measures have
measures been
have
have compared
been
been compared
compared under four
under
under differ-
four
four differ-
different
ent conditions,
conditions, i.e.,
i.e.,poor
ent conditions, i.e.,
poor illumination
poorillumination environments,
illumination environments,
environments, road conditions,
road
road drivers
conditions,
conditions, wearing
drivers
drivers eye- eye-
wearing
wearing eye-
glasses and
glasses drivers
glassesand
anddrivers with
driverswith beards
withbeards or moustaches,
beardsorormoustaches, to
moustaches, find
toto the
find
find most
thethe effective
most
most methods
effective
effective to
methods
methods de-to de-
to detect
tect driver
driver drowsiness.
tect driver drowsiness.
drowsiness. Figure
Figure 44shows
Figure the
thecomparison
4 shows
shows the comparison
comparison and
andindicates
and that
indicates
indicates thatthe
thatphysiologi-
thethe physiologi-
physiological
cal measure
measure shows
cal measure
shows showsbetter
better results
better
results among
results
among allother
among
all other measures
all other
measures ininallall
measures inconditions, butbut
it isitbut
all conditions,
conditions, is it is
highly
highly intrusive
highly
intrusive [4].[4]. [4].
intrusive

100%100%90% 90%
96% 96% 100%100% 96% 96%
95% 95%
75% 75%
80% 80% 65% 65% 80% 80%
Accuracy
Accuracy

Accuracy
Accuracy

60% 60% 60% 60%

40% 40% 40% 40%
20% 20% 20% 20%
0% 0% 0% 0%
1 1 2 2 3 3 1 1 2 2 3 3
Types of Measures
Types of Measures Types of Measures
Types of Measures

(a) (a) (b) (b)

100%100%90% 90%
96% 96%
87% 87% 90% 90%
100%100% 96% 96%
80% 80% 80% 80% 65% 65%
Accuracy

Accuracy
Accuracy

Accuracy

60% 60% 60% 60%

40% 40% 40% 40%
20% 20% 20% 20%
0% 0% 0% 0%
1 1 2 2 3 3 1 1 2 2 3 3
Types of Measures
Types of Measures Types of Measures
Types of Measures

(c) (c) (d) (d)

Figure 4. Accuracy of driver drowsiness detection on the basis of different conditions: (a) low lighting
conditions; (b) road conditions; (c) driver with beard; (d) driver wearing eyeglasses. (1) Vehicle-Based
Measure (2) Physiological Measure (3) Behavioral measure.

Behavioral measures also show promising results for driver drowsiness detection with
higher accuracy in normal conditions, but the accuracy decreases drastically in certain
conditions such as low light and drivers with eyeglasses. In addition, the non-reliable
secondary dataset used by artificial intelligence-based algorithms is also the root cause
Sensors 2023, 23, 1292 6 of 19

of a false positive driver drowsiness detection rate [10]. Due to these limitations, it is not
possible to use this measure alone in real driving conditions.
The limitations of individual measures can be overcome by combining two or more
measures in such a way that one technique can reduce the limitations of the others to
improve the system’s overall accuracy [16]. Table 1 reveals the possibility of various hybrid
approaches used for driver drowsiness detection. Hybrid measures combine two or more
measures that help to develop a highly accurate and reliable driver drowsiness detection
system. The majority of the research on hybrid measures has only been done in a simulated
environment. Due to their high cost and difficulty of implementation in actual driving
situations, all measures cannot be applied at once [4]. Due to their dependence on the state
of roads and lane markings, vehicle-based measures, when combined with other measures,
produce a significant false positive detection rate. Therefore, combining vehicle-based
measures with other measures in real driving conditions is challenging.

Table 1. Possibility of combination of various measures.

Ref. Hybrid Measures Accuracy Advantage Limitation

Behavioral
High false positive detection rate and
[17] + 91% Ease of use
dependent on geographical conditions
Vehicle-based
Behavioral
[18] + 98% High accuracy Highly intrusive
Physiological
Vehicle-based
Extremely intrusive and geographically
[19] + 93% High accuracy
dependent
Physiological
Vehicle-based
+
Expensive and more challenging to
[20] Physiological 81% High accuracy and ease of use
implement in real driving conditions
+
Behavioral

Among all possible combinations, the behavioral and physiological measures produce
promising results. However, the physiological measures are extremely intrusive and thus
challenging for the driver to wear them and drive the vehicle. Numerous studies have
confirmed that biological sensors can be replaced with intrusive physiological components
to identify driver drowsiness. Due to technological advancements in the area of AI and
biological sensors, hybrid measures can be utilized for driver drowsiness detection at an
early stage.

3. Materials and Methods

Several approaches to developing a model for detecting driver drowsiness that can
be used effectively have been investigated. To identify the most effective method and
review the current advancements in the area of DDDS, sixty-eight research publications
from various sources, including IEEE, Google Scholar, ScienceDirect and ResearchGate,
have been selected. Thirty-one papers have been shortlisted out of sixty-eight studies that
discuss face detection techniques, hybrid measures and deep learning algorithms.
A total of 26,344 articles have been published that help the research community to build
an efficient driver drowsiness detection system, of which 12,395 articles are based on hybrid
models, e.g., DDDS. Figure 5 shows the publication trends from 2012 to 2021 in DDDS and
hybrid model-based driver drowsiness detection systems (HMDDDS). These publication
trends reveal that the research community has shown intense interest in building an efficient
DDDS to reduce accidents and protect people’s precious lives [21].
 build an efficient driver drowsiness detection system, of which 12,395 articles are based
A total
on hybrid of 26,344
models, e.g.,articles
DDDS.have
Figurebeen published
5 shows that help the
the publication research
trends community
from 2012 to
to 2021 in
build an
DDDS andefficient
hybriddriver drowsiness
model-based detection
driver system,
drowsiness of which
detection 12,395(HMDDDS).
systems articles are These
based
on hybrid models,
publication e.g., DDDS.
trends reveal Figure
that the 5 shows
research the publication
community has showntrends from
intense 2012 toin2021
interest in
build-
DDDS and hybrid model-based driver drowsiness detection systems (HMDDDS).
ing an efficient DDDS to reduce accidents and protect people’s precious lives [21]. These
Sensors 2023, 23, 1292 publication trends reveal that the research community has shown intense interest in build- 7 of 19
ing an efficient DDDS to reduce accidents and protect people’s precious lives [21].
7000
5768
6000
7000
5000 4294 5768

Publications
6000
4000 3044 3304
5000 4294

Publications
3000 2471 DDDS
2098
4000 3304
3044 1653 3258
2000 2471 1250 1317 HMDDDS
3000 1145 2098 DDDS
1000 1909 1653 1766 3258
2000 281 1199 1178 1250 1317 1384 HMDDDS
0 1145 571 1766
1000 1909 484 365
2011281
2012 2013
1199 2014 2015
1178 2016 2017 2018
1384 2019 2020 2021
0 571
484 365
Publication Year
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
Publication Year
Figure 5. Publication trends based on DDDS and hybrid-based DDDS from 2012 to 2021.
Figure 5. Publication trends based on DDDS and hybrid-based DDDS from 2012 to 2021.
Figure 5. Publication
Figure 6 showstrends based on DDDS
that developing and hybrid-based
countries DDDS
like India are morefrom 2012 to 2021.
interested in developing
a driverFigure
drowsiness detection
6 shows system. The
that developing number
countries of India
like publications
are moreininterested
India is almost dou-
in developing
a Figure
driver
ble that 6 shows
of other that developing countries like India are more interested in developing
drowsiness detection
countries. system. The number of publications in India is almost double
a driver
that ofdrowsiness detection system. The number of publications in India is almost dou-
other countries.
ble that of other countries.
70 60
60
70
50 60
Articles

60
40 30 30
50
30 20 20
Articles

40
20 30 30 10 10 10 10 10
30
10 20 20
20
0 10 10 10 10 10
10
0

Countries
Figure 6. Frequency of publications based on DDDS by country.
Countries
Figure 6. Frequency of publications based on DDDS by country.
3.1. Hardware Requirements
Figure
3.1. 6.For
Frequency
Hardware of publications
Requirements
implementation based on
purposes, theDDDS by country.
following hardware components are used:
• ForRaspberry
implementation
Pi 3 B+purposes, the following hardware components are used:
3.1.•Hardware Requirements
• Raspberry Pi 3v2B+8 MP
Pi Camera
• ForGSR
implementation
Sensor purposes, the following hardware components are used:
• Pi Camera v2 8 MP
Raspberry
•• • GSR SensorPi 3 B+ converter
Analog-to-digital
•• Pi Camera v2 8 MP
Analog-to-digital
Table 2 shows converter
the specification of all hardware components used to develop the hybrid
GSR Sensor
• model. The Raspberry Pi 3 Model B+ features a 64-bit quad-core processor running at 1.4
Table 2 shows the specification of all hardware components used to develop the hy-
Analog-to-digital
• GHz, converter
dual-band wireless LAN and Bluetooth 4.2/BLE. This specification makes it easier to
brid model. The Raspberry Pi 3 Model B+ features a 64-bit quad-core processor running
construct a microcontroller-based
Table 2 shows the specification ofdriver drowsiness
all hardware detection system
components used to[22].
develop the hy-
brid model. The Raspberry Pi 3 Model B+ features a 64-bit quad-core processor running
Table 2. Hardware specifications.

Name of Component Specifications

64-bit quad-core processor running at 1.4 GHz,
Raspberry Pi 3 B+ dual-band 2.4 GHz and 5 GHz wireless LAN,
and Bluetooth 4.2/BLE
Pi Camera v2 8 Mega Pixel
GSR Sensor V2.0, 3.3/5 VDC
Analog-to-digital converter MCP3008
 Name of Component Specifications
64-bit quad-core processor running at 1.4 GHz, dual-band 2.4 GHz and 5 GHz wireless
Raspberry Pi 3 B+
LAN, and Bluetooth 4.2/BLE
Pi Camera v2 8 Mega Pixel
GSR Sensor
Sensors 2023, 23, 1292 V2.0, 3.3/5 VDC 8 of 19
Analog-to-digital converter MCP3008

The Raspberry
The Raspberry Pi PiCamera
Camerav2 v2isisa ahigh-quality
high-quality image
image sensor
sensorfor for
the the
Raspberry
Raspberry Pi mi-
Pi
crocontroller. It can capture high-quality images and videos that
microcontroller. It can capture high-quality images and videos that help to detect the help to detect the driver’s
facial features
driver’s [23]. It connects
facial features to the Raspberry
[23]. It connects to the RaspberryPi viaPi theviaCSI
theinterfaces
CSI interfaceson topon oftopthe
of
board.
the The The
board. Grove GSRGSR
Grove Sensor is used
Sensor in the
is used inproposed
the proposed model to collect
model the electrical
to collect con-
the electrical
ductance of the
conductance skinskin
of the by attaching
by attaching twotwo electrodes
electrodesto any of the
to any twotwo
of the fingers onon
fingers oneone
hand of
hand
thethe
of driver. TheThe
driver. GSRGSR Sensor
Sensorsends a small
sends amount
a small amountof electrical current
of electrical through
current one elec-
through one
trode andand
electrode measures
measures thetheintensity
intensityof of
thethe current
currentreceived
receivedon onthe
theother.
other. Skin
Skin conductance
conductance
(SC) fluctuates
(SC) fluctuates with
withskin
skinmoisture.
moisture.The The body’s
body’s reaction
reactionto physical
to physical exertion andand
exertion stress can
stress
be assessed
can using
be assessed the the
using skinskin
conductance
conductance of the person
of the person[24]. Due
[24]. Dueto the
to thenon-availability
non-availability of
analog
of analoginputs in the
inputs Raspberry
in the Raspberry Pi 3Pi model
3 model B+, aB+,
MCB3008
a MCB3008 Integrated
IntegratedCircuit (IC) is(IC)
Circuit usedis
as an as
used analog-to-digital
an analog-to-digitalconverter. The Python
converter. The Pythoncode is burnt
code on theon
is burnt Raspberry Pi to com-
the Raspberry Pi to
municate withwith
communicate the the
Pi camera
Pi camera and andGSRGSR sensor
sensor totocapture
captureimages
imagesand andskinskin conductance,
conductance,
respectively. The complete hardware implementation is demonstrated by the schematic
respectively. The complete hardware implementation is demonstrated by the schematic
diagram in
diagram inFigures
Figures77and and8,8, showing
showing thethe Raspberry
Raspberry Pi 3Pimodel
3 model B+ connected
B+ connected withwith Rasp-
Raspberry
berry
Pi Pi camera
camera and GSR andsensor
GSR sensor using MCP3008
using MCP3008 IC. IC.

Sensors 2023, 23, x FOR PEER REVIEW 9 of 19

Figure 7.
Figure 7. Schematic
Schematic diagram
diagram of
of the
the hardware
hardware implementation
implementation of
of the
the proposed
proposed model.
model.

Figure8.8.Raspberry
Figure RaspberryPi
Pi33B+
B+ microcontroller
microcontrollerwith
withcamera
cameraand
andsensor.
sensor.

During drowsiness, it is observed that the GSR value tends to decrease. The conduc-
tivity of the skin is used to calculate the GSR value. The equation to measure skin response
is:
C = (1/R) (1)
where C represents the skin conductance, which is inversely proportional to resistance
(R). The skin conductance can be measured in microsiemens (µs), where the normal range
lies between 250 µs to 450 µs for a normal person. The GSR value, often between 128 µs
Sensors 2023, 23, 1292 Figure 8. Raspberry Pi 3 B+ microcontroller with camera and sensor. 9 of 19

During drowsiness, it is observed that the GSR value tends to decrease. The conduc-
tivity ofDuring
the skindrowsiness,
is used to calculate the GSRthat
it is observed value.
theThe
GSRequation to measure
value tends skin response
to decrease. The con-
is: ductivity of the skin is used to calculate the GSR value. The equation to measure skin
response is: C = (1/R) (1)
C = (1/R) (1)
where C represents the skin conductance, which is inversely proportional to resistance
where
(R). C represents
The skin the skin
conductance conductance,
can be measured inwhich is inversely
microsiemens proportional
(µs), where the to resistance
normal range(R).
lies between 250 µs to 450 µs for a normal person. The GSR value, often between 128 µslies
The skin conductance can be measured in microsiemens (µs), where the normal range
andbetween
250 µs, 250 µs to 450
indicates µs for a drowsy
the driver’s normal state,
person. The further
which GSR value,
helpsoften between
to identify the128 µs and
driver’s
250 µs, indicates the driver’s drowsy state, which further helps to identify the driver’s
state, i.e., normal or drowsy, as depicted in Figure 9. It is determined by calculating the
state, i.e., normal or drowsy, as depicted in Figure 9. It is determined by calculating the
slope of the GSR, which provides the average rate of absolute change from a group of data
slope of the GSR, which provides the average rate of absolute change from a group of data
observed over a period of time. It is determined by averaging the initial difference of the
observed over a period of time. It is determined by averaging the initial difference of the
skin conductance signal’s absolute value [25].
skin conductance signal’s absolute value [25].

Figure 9. GSR skin conductance value.

Figure 9. GSR skin conductance value.
To analyze the GSR signal, four parameters are used: mean, standard deviation,
To analyze
kurtosis the GSR signal,
and skewness. The four parameters
baseline are used:
of the signal mean,
is its mean.standard deviation,
The standard kur-
deviation
tosis and skewness.
reveals The to
modifications baseline of thebaseline.
the signal’s signal is Theits mean. Theflatness
signal’s standard deviationthe
concerning reveals
normal
modifications
distributionto is the signal’s
assessed baseline.
using The signal’s
kurtosis. A positive flatness
number concerning the normal
for kurtosis distri- a
often denotes
bution is assessed using kurtosis. A positive number for kurtosis often denotes
signal that is leptokurtic, which is flatter than the normal distribution, whereas a negative a signal
that is leptokurtic,
value whichthat
denotes a signal is flatter than the which
is platykurtic, normalisdistribution,
less flat than whereas a negative
the normal value
distribution [26].
denotes a signal that equation
The implemented is platykurtic, which is
for kurtosis is less flat than the normal distribution [26]. The
the following:
implemented equation for kurtosis is the following: ( 4 )
1 n xj − x


n j∑
Kurtosis( x1 . . . xn ) = (2)
=1
σ

where x is a random variable having n observations. The above kurtosis equation calculates
the sum of deviation from the mean value divided by the standard deviation power of 4.
The term in the equation describes the shape of the GSR signal, whether tall or flat.
Skewness is shown to indicate the GSR signal’s symmetry with respect to its baseline.
A positive skewness number would represent a rightward skew in the signal, whereas a
negative skewness value would represent a leftward skew. The implemented equation for
skewness is as follows:
3
1 n xj − x

Skewness( x1 . . . xn ) = ∑ (3)
n j =1 σ

where x is the random variable having n observations. The above skewness equation is the
sum of the deviation from the mean value divided by the standard deviation power of 3.
The term used in the equation describes the symmetry of the GSR signal, whether it is a
positive skew distribution or negative skew distribution. All these equations are utilized in
Sensors 2023, 23, 1292 10 of 19

the proposed hybrid model to calculate efficacy, as equations will remain same whether it
is a hybrid model or a model with a single measure.
Various tools and techniques have been thoroughly addressed in the preparation of
this work. The detailed analysis of different face detection techniques and feature extraction
using AI-based algorithms are explained below, followed by the hybrid model.

3.2. Face Detection Techniques

Face detection is the crucial step in detecting the face of the driver and extracting the fa-
cial features to evaluate the drowsy state of the driver [27–29]. The three most popular tech-
niques for detecting the face below are available below to implement behavioral measures.
1. OpenCV Haar Cascade
2. Davis King library (Dlib)
3. Multi-task Cascaded Convolutional Neural Network (MTCNN)
Face detection is important for both identifying faces and extracting facial features.
Many face detection methods can capture, recognize and process a face in real time while
extracting various facial features. In the present era, many electronic devices include built-
in facial detection software that may verify the user’s identification. This section explains
all face detection techniques that help to select the appropriate technique for the proposed
hybrid model.
The OpenCV Haar Cascade classifier is an effective method to detect objects. Paul
Viola and Michael Jones first suggested the strategy in 2001 [30]. The face coordinates can
be obtained using a mathematical model to determine the integral image. It is calculated as:
x.y
∑i x 0 , y0


Ii ( x, y) = (4)
i =1

where (x, y), (x’, y’), the brightness of the pixel in the image according to the coordinates,
and Ii (x, y) the value of the ith element of the integral image with coordinates (x’, y’).
The integral image is used to quickly determine the brightness of certain areas of the
image, regardless of the size or location of the image. OpenCV is an ML-based open-source
computer vision library with a trainer and a detector. To detect a human face and eye, a
pre-trained classifier can be used as an XML file [31]. This technique quickly detects the
object efficiently, but it cannot detect faces under occlusion. It also generates many false
predictions during the face detection procedure [27].
Dlib is another open-source library used for implementing various machine-learning
algorithms. It helps to detect human facial features in images and videos. The Dlib library
provides two algorithms that can be used for face detection, which are Histogram of
Oriented Gradients (HOG) + Linear Support Vector Machine (SVM) and CNN based. Dlib
HOG is a simple yet powerful tool to detect the face and is widely used [28]. It works
with the combination of an SVM machine-learning algorithm that effectively detects the
faces of the person. It is a fast and lightweight model that works without special hardware
requirements. However, the highly accurate Dlib CNN face detection method is employed
for face detection. CNN is a deep learning-based algorithm with Dlib that helps detect
the face from different angles. Dlib is more accurate than OpenCV Haar Cascade, but the
computational cost is very high and unable to run in real-time conditions [27].
One of the most popular and reliable facial recognition methods used today is MTCNN.
It is a multi-task cascaded convolutional neural network that uses images to detect the
face and facial features. It is a deep learning algorithm to identify faces and facial fea-
tures accurately [29]. The whole concept of MTCNN can be explained in three stages, as
mentioned below:
P-NET: MTCNN generates several frames that successfully scan the entire image from
the top left corner to the bottom right corner.
R-NET: Most frames without faces are rejected by the next layer of CNN, which uses
the data from P-Net as input.
Sensors 2023, 23, x FORSensors
PEER REVIEW
2023, 23, x FOR PEER REVIEW 11 of 19 11 o
Sensors 2023,
Sensors 2023, 23,
23, x
x FOR
FORSensors
Sensors
PEER REVIEW
PEER REVIEW
2023, 23,
2023, 23, x
x FOR
FOR
Sensors
PEER 2023,
PEER REVIEW
23, x FOR PEER REVIEW
REVIEW 11 of
11 of 19
19 11 o
11 o
Sensors
Sensors 2023,
2023, 23,
23, x
x FOR
FOR PEER
PEER REVIEW
REVIEW
P-NET:
Sensors 2023,MTCNN 23, x FOR generates
P-NET:
PEER REVIEWMTCNN
severalP-NET: generates
frames MTCNN that several
successfully
generates
frames scan that
severalsuccessfully
the entireframesimage that
scansuccessfully
the entire im s
P-NET: MTCNN generates
P-NET: MTCNN
several generates frames that several
successfullyframes scan that successfully
the entire image scan the entire im
Sensors 2023, 23, x FORSensors
PEER REVIEW from the
2023, 23, x FOR top left
PEER REVIEW
P-NET: MTCNN corner
from the
to the
generates
P-NET: top bottom
left
MTCNN corner
from
several right the
P-NET:tocorner.
generates
frames the
top bottom
left
MTCNN
that corner
several right
successfullyto corner.
the
generates
frames bottom
scan
that
several right
successfully
the entire corner.
entire
frames 11 of
image
that
scan19 successfully
the entire11im so
Sensors 2023, 23, 1292 fromP-NET:
the top MTCNN fromgenerates
left corner P-NET:
the
to the MTCNN
topbottom
leftseveral
cornerright generates
frames
tocorner.
the bottom that several
successfully
right frames
corner. scan
that successfully
the imagescan the
of 19entire im
from R-NET:
the top Most
left frames
corner
from R-NET:
the
to without
the
top Most
bottom
left faces frames
corner
fromrightare
R-NET:
the to rejected
without
corner.
the
top Most
bottom
left byfaces
frames
cornerthe
rightare
next
to without
rejected
layer
corner.
the bottomoffaces
byCNN, the are
right next
which
rejected
layer
corner. uses by11CNN,
of the nextwhichlayeru
fromR-NET:
the top Most left corner
from
frames the
towithout
R-NET: the
topbottom
leftfaces
Most cornerright
frames P-NET:
are
P-NET:tocorner.
the bottom
rejected
without MTCNN
MTCNN byfaces right
the generates
are
next layer several
corner.
rejected
generates ofbyCNN,
several frames
the next which
frames layer that
uses
that of successfully
CNN, whichssu
successfully
the data
R-NET: fromMost P-Net the asdata
frames input.
R-NET: fromMost
without P-Net the
faces as
framesdata
areinput.
R-NET: fromMost
rejected
without P-Net
byfaces asare
frames
the input.
next without
rejected
layer offaces
by CNN,the arenext
which
rejected
layer uses of
by CNN,
the next whichlayer u
R-NET:
the data fromMost P-Net frames
the R-NET:
asdata without
input. fromMost P-Net from
faces frames
as arethe
input.
P-NET: top
rejected
without left
MTCNN bycorner
faces the to
are
next the
rejected
generates layer bottomof by CNN,
several right
the nextcorner.
which
frames layer uses
that of CNN,
successfully which u
O-NET:
the data
data from The
P-Net results
the as O-NET:
data of from
input. this Thestep
P-Net
from
results
areas
the data
the
more
O-NET:
input.
top
offromthis
detailed left
The
step
P-Net
corner
results
than
areas
to
ofthe
more
that
input. ofbottom
this
detailed
R-Net.
step are right
than
Themore corner.
that
facial detailed
ofland-
R-Net. than Thethatfacial Rs
of la
the O-NET:
P-NET: from P-Net
The
MTCNN the as
results data
input.
O-NET:
of from
generates
P-NET: this P-Net
The
step
MTCNN
several
from as
results
are R-NET:
input.
more of
generates
frames
the this
detailedMost
step
that frames
than
severalare more
successfullythatwithout
frames detailed
of R-Net.
scan
that faces than
Theare
successfully
the rejected
that
facial
entire of land- by
R-Net.
imagescan theThenext layer
facial
entire imla
markO-NET:
position O-NET: is the The
markoutputresults
position of
of this this
is stage
the step
mark R-NET:
output
after aretop
position more
detecting
of
left
Most
this
corner
the frames
isdetailed
stageaoutput
face
to
than
after
the that
without
from of
bottom
detecting faces
of right
R-Net.are corner.
rejected
The facial by athe
land- next layer
fromO-NET:
mark position
the top The
Theis
left results
results
themark
corner
from O-NET:
output ofthe
O-NET:
of this
this
position
the
to of
top The
step
The
step
this is
bottom
left results
stage
theare
the
results
are
corner
the more
O-NET:
data
more
output
rightafter
R-NET:
data of
of
to this
detailed
from
this
detailed
detecting
of
corner.
the
from The
step
P-Net
step
this
bottom
Most
P-Net results
than
stageare
than
are
a as
face
right
frames
as more
that
of
input.
more
that
afterfrom ofthis
this the
detailed
R-Net.
step
detailed
of
corner.
without
input. R-Net.
detecting
the
stage
given
are
given
faces
aThe
than
than
a
face
The
after
image/video.
more
face
from
that
facial
that
facial
detecting
image/video.
are from of
rejected
the
detailed
ofland-
land-
R-Net.
R-Net.
the by
given
than
giventhe
face
The
The
image/vid
from
that
facial
facial
image/vid
next of th
layerla
R
la
mark position
Identifying is the
prominent output
Identifyingfacial of this stage
features
prominent Identifying after
is detecting
known
facial detecting
prominent
features
asafacial aislandmark
face
known
facialfrom the
features
as given
detection,
facial isfrom image/video.
landmark
known
which asadetection,
facial landmwh
mark position
markIdentifying
positionMost
R-NET: is the
is the mark
markoutput
output
prominent
frames position
positionof
of
Identifying
R-NET: this
this
facial
without is
Most stage
the
mark
isfeatures
stage
the
faces output
after
output
prominent
frames
the position
O-NET:
after
are of
detecting
of
isrejected
known
facial
without this
The
thisis the
bystage
stage
faces output
results
athe
features
as face
faceafter
of
after
facial
are
nextfrom
from
is of
detecting
this this
the
detecting
landmark
known
rejected
layer step
the stage
given
are
ofgiven
byas a
atheface
after
image/video.
more
face detecting
image/video.
detection,
CNN, facial
next from
landmark
which the
detailed
the
which
layer uses given
than
ofgiven face
image/vid
from
that
image/vid
detection,
CNN, which of th
ofwhR u
aids in Identifying
monitoring
Identifying the prominent
aids
prominent driver’s
in monitoring
Identifyingfacial
facial
drowsiness. thedata
aids
features
prominent
O-NET:
features
indriver’sfrom
monitoring
MTCNN
Identifying
isMTCNN
known
facial
isP-Net
The known
drowsiness.
allows
theas
results
prominent
features
as facial
input.
of
asfor this
facial
driver’s
MTCNN
the
isMTCNN
landmark
known
facial
step
landmark
drowsiness.
detection are
allows
features
as
more
detection,
facial for detailed
ofdetection,
isMTCNN
five
the facial
landmark
known
which
than
which
detection
allows
asthan
that
of
forlandm
detection,
facial five
the Rd
fa
wh
aids Identifying
in
the data monitoring
from P-Netprominent
aids
the
the as Identifying
driver’s
in
data
input. facial
monitoring
from P-Net mark
features
prominent
drowsiness. the
mark as position
is
driver’s
input.
O-NET:
position known
facial Theis
is the
features
theas
drowsiness.
allowsoutput
facial
results
output is
for
of of
landmark
known
the
this
of this stage
detection
step
this as
allows
are
stage after
detection,
facial
of
more
after for detecting
landmark
five
thewhich
facial
detailed
detectingdetectionaadetection,
face
face of from
that five
from of th
wh
fa
R
th
aids intwo
landmarks: monitoring
forlandmarks:
thedriver’sthe driver’s
eyes, one
two forfor drowsiness.
landmarks:
the
thenose eyes, and one
two MTCNN
twofor
for for
the
thethe allows
nose
eyes,
mouth.and
one fortwo thethe
for
By detection
using
for the
nose these
mouth.
and ofland-
five
two By facial
for
using theofthese
mouth la
aids in
aids in
landmarks: monitoring
monitoring
O-NET: twoThefor aids
the
aids
the
results in
thedriver’s
in
landmarks:eyes,
O-NET:
ofmonitoring
monitoring
one
this drowsiness.
two
The for
step foraids
drowsiness. the
the
the
mark in
themore
results
are driver’s
nose monitoring
eyes,
ofMTCNN
Identifying
driver’s
MTCNN
position
Identifying and
this one
detailed drowsiness.
twofor
step
is theallows
drowsiness. the
prominent
allows
for
the
than
are driver’s
the for
for
nose
more
output
prominent MTCNN
mouth.
that the
MTCNNfacial
the
and
of drowsiness.
detection
detailed
ofthis
facial two
By
R-Net. allows
features
detectionallows
stageusing
for
than
features The of
of
the
after for
isMTCNN
forfive
these
that the
known
five
the
mouth.
facial
is facial
detection
facial
ofusing
land- By
R-Net.
detecting
known allows
as
detection
as afacial
usingThe
face
facial for
of five
thesetheth
landm
five
facial
from
landm fa
fad
la
marks, landmarks:
drowsy two
detection
marks, for the
drowsy
can eyes,
be one
identified
detection
marks, for the
drowsy
by
can nose
efficiently
be and
detection
identified two
using can
by for thesethe
efficiently
be mouth.
identified
landmarks. using By
by these
Apart
efficiently these
landmarks.using the
A
landmarks:
landmarks:
marks, drowsy
mark position two
two for
for landmarks:
the
landmarks:
the
detection
is the marks,
markoutputeyes,
eyes, one
drowsy
can
position two
one
two
be for
forfor
forlandmarks:
the
aidsthe
the
the
identified
detection
of thisisstagethe
aids nose
in
output eyes,
nose
after
Identifying
in byand
monitoring
eyes, and
can one
two
one two
twofor
for
for
efficiently
detecting be
of this by
monitoring for
the
forthe
the
identified
stage the
nose
eyes,
the
nose
aefficiently
prominent
the faceaftermouth.
driver’s
mouth.
using by
from
driver’s and
one
and these
detecting
the
facial two
for
By
drowsiness.
two
By
efficiently the
using
for
using
for
landmarks.
given a face
features
drowsiness. the
nose
the these
mouth.
these
using and
MTCNN
mouth.
image/video.
isfrom
known
MTCNN land-
two
land-
these
Apart By
By
the given
asfor
using
allows
using the
landmarks.
allows these
mouth
for
these
image/vid
facial the
forlandm A
theonla
d
la
from
marks,
landmarks,
this,drowsy
drowsy
facialdetection
landmarks
from
marks,
detection
this, can
facial
drowsy
can bebebe can
landmarks
from
identified
detection
marks,
be
effectively identified
this, drowsy
by
can facial
can
used be landmarks
efficiently
bebe foreffectively
detection
identifiedbehavior
using can
by canusing
used
detection
these be
efficiently
be
these
for
effectively
identified
landmarks.
landmarks.
behavior
while
using by used
online
detection
these
Apart
Apart
for
efficiently behavior
landmarks. while
using dd
the
A
marks,
from this,drowsy
facial
Identifying
from this, facial detection
marks,
landmarks
from
prominent this,drowsy
can
Identifying canbe
facial
facial
landmarks can[32]. landmarks:
identified
detection
landmarks
effectively
features
prominent
aids
landmarks:in is by
can
known two
can
used
facial
monitoring two for
efficiently
be for the
identified
features
for as thefacial
the eyes,
using
effectively
behavior is by one
landmark
driver’s
eyes,
be effectively used for behavior detection while online [32]. these
known
one for
efficiently
used
detection as
drowsiness.
for the
landmarks.
for
the nose
using
behavior
while
detection,
facial
nose and
MTCNN
and these
Aparttwo
online
landmarkdetection
whichtwo for
allows
for the
landmarks. mouth
while
detection,
the for the
mouth A
on
wh d
[32].
from this, [32].
facial landmarks
landmarks
from this, can
facial belandmarks
landmarks
from
effectivelythis, facial
can
used be landmarks
foreffectively
behavior canused
detection
be for
effectively
behavior
while used
online
detection
for behavior
while on
d
from
aids inthis,
[32]. facial
monitoring from
[32].
aids
the this,
driver’s
in can
facial
monitoring be marks,
drowsiness.effectively
the
landmarks:
marks, drowsy
driver’s
MTCNN
drowsy can
used
two detection
be for
drowsiness.
for effectively
allows
detection behavior
the for
eyes, can
MTCNN
can the
one be
used
be identified
detection
detection
for for
allows
the
identified behavior
while
of
nose for by
five
the
and
by efficiently
online
detection
facial
detection
two
efficiently for using
while
theof
using five
mouth the
on
fa
the
[32]. Table Table3 shows 3 showstheTable
[32].
the 3comparative
comparative shows analysis
[32]. the Table analysis
comparative
of
3 shows of various
various analysis
the face face
comparative
detection detection
of various techniques.
analysis
facetechniques.
detection
of Five
various Five
techniques.
face detecF
[32]. Table
landmarks:
parameters3twoshows for [32].
are the Table
landmarks:
the used comparative
eyes, to 3two
one shows
for
analyze forfromanalysis
the
marks,
from the
the
the this,
nose
eyes,
this, drowsy
best facial
comparative
andofone
facial various
twofor
technique landmarks
for
the
detection analysis
landmarks face
the
nose
for mouth.
can can
detection
and
drivercan be
ofbevarious
two
By
be effectively
techniques.
using
for
identified face
the
effectively
drowsiness these
mouth.
by used
detection
Five
land- Byfor
efficiently
used
detection forusingbehavior
techniques.
[27]. thesethe
using
behavior d
dlaF
parameters
Table are
3areshowsused parameters
thetoTable
analyze
comparativeare
3are the
shows usedparameters
best to
analysis
the technique
Table analyze
comparative
of
3 areshowsthe
used
forbest
various driver
to technique
analyze
analysis
the face drowsiness
comparative
detection
of the
variousforbestdriver
detection
technique
techniques.
analysis
face drowsiness
[27].
detection
of Five for techniques.
various driver
detection
face drow
detec [F
marks,Table
parameters
Table 3
drowsy showsused the
detection
3 revealed marks, Table
parameters
to
that comparative
analyze
drowsy
can 3 shows
be
the MTCNN theused[32].
analysis
best
identified
detection
from
[32]. theto this,
providescomparative
technique
analyze
by
can of various
the
for
efficiently
facial be analysis
best
identified
landmarks
a better face
driverusing
result detection
technique
by of
drowsiness
these
can
than variousfor
efficiently
be
the techniques.
landmarks.face
driver
detection
effectively
OpenCV using detection
Five
drowsiness
[27].
these
Apart
used
Haar techniques.
detection
landmarks.
for behavior
Cascade Ad[F
Table 3
parameters revealedare used that
used Table the
parameters 3MTCNN
to this,revealed
analyze arethe
theprovides
that
usedTable the
parameters
best to 3 MTCNN
revealed
a
technique better
analyze are result
that
provides
the
used
for best the
than
driver
to MTCNN
a the
better
technique
analyze OpenCV
drowsiness result
provides
the forbest Haar
than
driver
detectiona better
the
Cascade
technique OpenCV
drowsiness result
[27]. for driver than
Haar
detectionthe
Casc
drow O
parameters
Table 3 revealed
from this,
and are
facial
Dlib. parameters
that
Table
landmarks
from
MTCNN to
the 3analyze
MTCNN
revealed
cancanare
facial be
also used best
provides
that
landmarks Table
to
the
effectively
[32].
detect technique
Table analyze
MTCNN
a better
the 33can
usedshows
shows
sides the
befor
result
provides
for the
best
driver
ofeffectively comparative
technique
than
behavior
the
faces drowsiness
afrom
the
better
comparative OpenCV
used the for
result
detection for analysis
driver
detection
Haar
than
behavior
while
analysis
images theof
Cascade
[29]. various
drowsiness
[27].
OpenCV
online
of detection
various
The only face
detection
Haar
while
face on[[
detec
Casc
detec
and
Table Dlib.
3 MTCNN
revealed and
that can
Table theDlib.
also
3MTCNN detect
MTCNN
revealed the andcan
provides
that
Tablesides
Dlib.
the also
3 ofbetter
MTCNN MTCNN
detect
faces
revealed
a from
the
result
that can
provides sides
the
the also
than images
of
MTCNN
a thedetect
faces
better [29].
OpenCV from
the
result
provides The
sides
the
Haar
thanonly
images
of
a faces
limita-
better
the
Cascade [29].
OpenCVfrom
result The
the
thanonly
Haarimages
thelim
CascO
Table
and 3 revealed
[32].Dlib. MTCNN that
Table
and
can
[32]. the 3MTCNN
Dlib.
also revealed
detect
MTCNN the parameters
provides
that canisthe
sides
parameters also
Table MTCNN
aofbetter are
itdetect
faces
3takes
areshows used
result
provides
from
the to
sides
the
the analyze
than aof
the
imagesbetter
faces
comparative the
OpenCV result
[29].
from best
The thetechnique
Haar
than
analysis only the
Cascade
images OpenCV
limita-
of for
[29].
various driver
The Haar
faceonly drow
Casc
lim
detec
tion
and
limitation
recorded
Dlib. MTCNN in recorded
MTCNN
tion
and
can Dlib.
also
in
recorded MTCNN
isdetect
thatinitthe
MTCNN MTCNN
tion
takes
andcan
sides
that
recorded
Dlib. more
also of is
MTCNN that
detect
facestime inused
from
more
itthe
MTCNN
totakes
can
to
train
sides
the
analyze
time
more
also the
images
of
to
isdetect
that
system
time
faces
the
train
itfrom
[29].
thetobest
the
takes
than
train
The
sides
the
technique
system
more
other
onlythefaces
images
of
than
system
time
tech-
limita-
for
toother
[29].
from
driver
than
train
The
the
drow
other
the
only
images sys
lim te
and
tion Dlib.
recorded
Table MTCNN
3 shows
techniques. and
in MTCNN can
tion
FromtheDlib.
also
recorded
Table isdetect
MTCNN
that
comparative
this initthe
3 shows
analysis, Table
can
MTCNN sides
takes
analysis
the
parameters
Table
it 33concluded
is also revealed
moreofisdetect
faces
comparative
revealedof that
time
are that
from
variousthe
it
usedtotakes
that
that the
sides
the
train
analysis
face
to
the MTCNN
images
moreof
the
analyze
MTCNNMTCNN faces
system
detection
oftime provides
[29].
from
various
the The
tobest
than
provides
isconcluded
the best the
train
techniques.
face aathe
only
otherbetter
images
technique limita-
system
better tech-
detection
Fiveresult
[29].
result The
thanthan
for techniques.
driver
facethanonly the
other
drow
the O
lim
Ote F
niques.
tion From
recorded this
in analysis,
niques.
MTCNN
tion From
recorded it
is is
thatconcluded
this
init niques.
analysis,
MTCNN
tion
takes that
recorded
moreFromit
is MTCNN
is
that
timethis
concluded
in it analysis,
MTCNN
totakes is
train the that
more
thebest
it
is is
MTCNN
that technique
system
time it to
takes
thanis
train the
for
that
more
otherbest
theface
MTCNN technique
de-
system
time
tech- to is
than
trainthefor best
other
the face
sys t
te
tion recorded
niques.
parameters From
detection are in
this MTCNN
used tion
analysis,
niques. recorded
parameters
to
implementation. is
From
it
analyze that
isare init
concluded
this
theusedand
MTCNN
takes
best
Table
and Dlib.
analysis,
to more
that
technique
Dlib.3 analyzeMTCNN
is
it
revealed
MTCNN that
time
MTCNN
is it
the
forto
concluded
that can
takes
train
best
can is
driver
the also
more
the the
that
MTCNN
also detect
best
technique system
time
MTCNN
drowsiness
detect theto
technique
for
provides
the sides
thantrain
is
driver the
for
detection
sides of
other
athe
of faces
best
facesystem
tech- from
technique
drowsiness
better
faces de-
[27].
result
from the
than images
other
for
detection
than
the face
the
images O te
[
tection
niques. implementation.
From this thisthattection
analysis,
niques. implementation.
From
it is
is concluded
concluded
this tection
niques.
analysis, implementation.
that
niques.
tection
Table From
3 implementation.
revealed analysis,
niques.
tection
Table the3MTCNN From
it this
implementation.
revealed tion
provides
that
and
tion recorded
analysis,
the
Dlib. aFrom
that
MTCNN
recorded it MTCNN
it
better
MTCNNMTCNN
is this
is concluded
in
inresultanalysis,
MTCNN
concluded
provides
can
MTCNN isthan
is the
the
alsothat
ais
thatbest
it
best
the
is is
betterMTCNN
thatconcluded
technique
MTCNN
detect
that it
technique
OpenCV takes
result
the
it takes is
is
Haarthe
for
that
more
the
than
sides for
more best
face
bestMTCNN
face
ofthe time
Cascade
faces technique
de- to
to train
technique
de-
OpenCV
time from is the
Haar
the
train for
for best
the face
syst
sys
face
Casc
images
the
tection
tection
and implementation.
implementation. tection
tection implementation.
implementation. tection
niques. implementation.
From this analysis, it is concluded that MTCNN is the best t
TableDlib.
Table MTCNN
3. Comparative
3. Comparative and
can
Table Dlib.
also
analysis detect
MTCNN
3. analysis
Comparative
of various the
of can
tion sides
niques.
various
Table
face
analysisalso
recorded
3. of detect
From
face
detection faces in
this
ofdetection
Comparative variousfrom
the
MTCNN sides
thedetection
analysis,
techniques images
techniques
analysis
face of
isitoffaces
based that [29].
isvariousitfrom
concluded
based
on on
techniques
various The
takes the only
various
face images
more
that
parameters.
detection
based limita-
time
MTCNN [29].
parameters.
ontechniques
various The
to trainis theonly
the lim
sys
best
paramete
based ot
Table 3. Comparative
tion recorded in MTCNN Table
analysis3.
tion recorded Comparative
of various
is thatinit MTCNN tection
face
analysis
takes more
niques.
tection implementation.
detection of
is that
From various
time
implementation. techniques
thisitto face
takes detection
trainmore
analysis, based
theit is on
system
time techniques
various
concludedtothan parameters.
trainotherbased
the MTCNN
that on
system various
tech- than paramete
is the other
best te t
Table 3.3. Comparative
Comparative Table
analysis3. Comparative
Comparative
of various
Table
niques. From Table
analysis
this analysis,
niques. 3. itof
From isvarious
concludedTable
thistection face
analysis
face
analysis
analysis, 3.detection
Comparative
detection
that of various
itof isvarious
MTCNN
implementation.
techniques
FaceFace
techniques
concluded analysis
face
face
is detection
based
Detection
detection
thebased
Detection
that of various
best on
on
MTCNN techniques
various
techniques
various
Techniques
Face
technique face
Techniques is parameters.
detection
based on
faceon
parameters.
Detection
the
forbased
best techniques
various
various
Techniques
Face
technique
de- paramete
based oT
paramete
Detection
for face
Parameters Table FaceanalysisDetection Techniques
Face Detection Techniques
tection implementation. Parameters
tection implementation. Table 3.
Parameters Comparative
3. OpenCV
OpenCV
Comparative Parameters
Haar
Haar
Face analysis
DetectionOpenCVof
of various
various Haar
Techniques
Face
face
face detection
OpenCV
detection
Detection
techniques
Haar
techniques
Techniques
Face
based
based o
Detection o
Parameters Parameters OpenCV FaceHaar
Cascade analysis of
DetectionOpenCVDLIB Techniques
DLIB Face
Haar Detection
MTCNN MTCNN
DLIB Techniques MTCNN DLIT
Table 3. Comparative Cascade various face MTCNN
Cascade
DLIB detectionCascade
DLIB techniquesMTCNN based o
Parameters
Parameters Parameters
Parameters OpenCV
OpenCV Cascade Parameters
Haar OpenCV
Haar OpenCV Cascade Haar OpenCVFace
Haar Haar
Face Detection
Detection T
T
Table
Work 3. Comparative
inWork
real-time Table
Workanalysis
conditions
in real-time 3.
inComparative
of various
real-time
conditions Workface
analysis
conditions detection of
in real-time various
 techniques face detection
conditions based DLIB
DLIB ontechniques MTCNN
various parameters. based
MTCNN  DLIB

DLIB on
 various MTCNN
paramete
MTCNN DLI

Work inHigh real-time Work
conditions in real-time conditions Parameters Cascade
Cascade Parameters Cascade

Cascade OpenCVCascade
  Haar
OpenCV Haar
Face Detection  T
High accuracy accuracy in different
in different
High accuracy
conditions in High
different accuracy conditions  inFacedifferent 
conditions     DLI

DLI
Work
Work in real-time
inconditions
High accuracy real-time Work
conditions
Work
conditions
in different
High in real-time
in conditions
real-time
accuracy Work
conditions
conditions
in different in real-time
conditions  Detection
conditions
Parameters 
 
Techniques
Face Detection  
Cascade
OpenCV Techniques
 Haar
Cascade 
 
High efficiencyin
High accuracy
accuracy High
different efficiency
High accuracy
accuracy
conditions in High
High
different efficiency
accuracy conditions  different 
  Haar  
Cascade
  
DLI

High efficiency Parameters
in
High efficiency different
High conditions
efficiency Parameters
in Work
different
Work inOpenCV
in real-time
conditions
real-time in Haarconditions
conditions conditions
OpenCV
DLIB

 
 MTCNN  

DLIB  MTCNN  
Detect the
High efficiency sides
efficiency of faces
Detect
High the
efficiencysides of faces
Detect the
efficiency sides 
inof faces 
 
 
  
 
 
 

High
Detect Detect
the sides theof High
sides
faces
Detect the sides ofHigh
of efficiency
faces facesaccuracy
Work
High Cascade
in real-time
accuracy in different
conditions
different 
Cascade
conditions
conditions    
Less
Detecttimethe for
sides training Less
faces
ofDetect
Detect time the for
sidestraining Less
ofHigh time
facesaccuracy
Detect for
the sides 
training
inofdifferent
faces conditions 

 

 
 

 
 
 
Worktime
Detect
Less in
the real-time
Less time
sides
for for
of
training Work
conditions
training
faces
Less timein
the real-time
sides
for of
training conditions
faces
High efficiency
efficiency  
       
Less time
Hightime
Less accuracyfor training
for training Less
in different
Hightime
Less time
accuracy for training
training
conditions
for Less
in Detect
different
High time the
efficiency for
sides
conditions
Detect the sides of faces 
training
 of faces 
 
 

 
 

 

 

4. Architecture of 4. Hybrid
ArchitectureModel of4.Hybrid Architecture Modelof Hybrid Model  
High efficiency
4. Architecture
4. Architecture of 4.High
Hybrid efficiency
of Hybrid
ArchitectureModelModel Less
ofDetect
Hybrid
Less time
time for
theModelsides
for 
training
of faces
training     
  
The
4.REVIEW
Architecture
Detect the proposed
sides of
of 4. hybrid
Hybrid The
Architecture
faces
Detect model
proposed
Model
the sides is
ofof the
hybrid
4.Hybrid
Hybrid The
amalgamation
Architecture
faces proposed
model
Model  ofisHybrid
the
hybrid
of twoamalgamation
model
measures:
Model   is the behavioral
of amalgamation
two  measures:
 and
 of
behavioral
two meas  
Sensors 2023, 23, x FOR PEER4. Architecture The
The proposed hybrid of 4.
Hybrid
proposed ArchitectureModel
hybrid
The model proposed of
model Less
is the hybrid time
is the Model
amalgamationfor training
amalgamation
model is the of two of
amalgamationtwo measures:
measures: behavioral behavioral
of two measures: 12 of
and
and behavioral 19
physiological. Figure physiological.
10The depicts the
Figure architecture
physiological.
10 depicts ofFigure
the
the architecture
hybrid 10twodepicts
model. ofthe the
This
isarchitecture
hybrid
model model.
has ofbeenthe
Thishybridmodel
twomodel has b
Less time
The for training
proposed
physiological.
The
physiological. proposed Figure Less
hybridtime
Figure
hybrid
10The
physiological. 10fordepicts
model
model training
proposed
proposed
depicts the is
Figure 4. Architecture
isarchitecture
the
hybrid
4.the
the
hybridThe
10amalgamation
Architecture proposed
model
architecture
amalgamation
depictsmodel of of
the
the Hybrid
isof
ofis the
hybrid
of
Hybrid
thethetwo
of
architecture
hybrid Model
amalgamation
model
Model
hybrid
amalgamation
model. 
measures: the
ofmodel.
measures: This the behavioral
ofThis
behavioral
of
hybrid
model two measures:
amalgamation
twomodel
model.
has 
measures:
and
has
and
been Thisbeen of
behavioral
behavioral
model  b
meas
has
divided into
physiological. three
Figure divided
phases.
physiological.
10 depictsinto
depicts three
the
Figure divided
phases.
architecture
physiological.
10 into
depicts three
ofFigure
the
the phases.
architecture
hybrid 10 depicts
model. ofthe the
This
divided divided
physiological.
into three into
Figure three
divided10
phases. phases.
physiological. the
into three Figure 4. Architecture
The
architecture
10
phases.The proposed
depicts
proposedof of architecture
the
the Hybrid
hybrid
hybrid
hybrid Model
model
model.
model of isarchitecture
is
the
This hybrid
the model
hybrid
the model model.
hasof
amalgamation
model.
has
amalgamation been
beenthe
This
Thishybridmodel
of twomodel
two
model
of has b
meas
has
meas b
1.
4. Data
Architecture
divided Acquisition
into three
three 1.
of divided
4.
Hybrid
divided
phases. Data
Architecture Acquisition
Model
into three
three 1. Data
ofphysiological.
Hybrid
divided
phases.Theinto Acquisition
Model three
Figure phases.10
divided
1. Data1.1. intoAcquisition
Data phases.
Data Acquisition
Acquisition
1. Data into
Acquisition phases.
physiological. proposed Figure 10 depicts
hybrid model
depicts the
theisarchitecture
the amalgamation
architecture of
of thethe hybrid of twomodel
hybrid meas
model
2. Feature Extraction2. Feature Extraction2.
divided Feature into Extraction
three phases.
1.
1. 2.2. Acquisition
The
Data
2. Data
Feature Feature
proposed
Acquisition
Extraction
Feature 1.
1.Extraction
hybridThe
Data
Data
2.Extraction
Feature model
proposed
Acquisition
Acquisitionis
Extraction1.the
hybrid amalgamation
Data
physiological.
divided into three model
Acquisition is
Figure the
of
phases. two
amalgamation
measures: behavioral
of two
10 depicts the architecture of the hybrid model measures:
and behavioral
3. Classification 3. Classification 3. Classification
3.3. Classification
physiological.
2. Classification
2.
3. Feature
Feature Classification
Figure
Extraction
Extractionphysiological.
2. 10Classification
2.
3. depictsExtraction
Feature
Feature the
Figure
Extractionarchitecture
2.
divided
1. 10 depicts
Feature
Data
1. Data Acquisition into of the
the
Extraction
three
Acquisition architecture
hybrid
phases. model. of theThishybrid modelmodel. has been This model has b
divided
3. into three 3.
3. Classification
Classification divided
phases.
3. into three2.
Classification
Classification 3.
1. phases.
2. Data Classification
Feature
Feature Extraction
Acquisition
Extraction
1. Data Acquisition 1. Data Acquisition 3.
2. Classification
3. Feature Extraction
Classification
2. Feature Extraction 2. Feature Extraction 3. Classification
3. Classification3. Classification

Figure 10. The architecture

Figure 10.
of the
Theproposed
architecture
Figure
hybrid
10.
of The
the
model.
proposed
architecture
hybrid
of the
model.
proposed hybrid model.
Figure 10. The architecture
Figure 10.
of the
Theproposed
architecture
hybrid
of the
model.
proposed hybrid model.
Figure 10.
Figure 10. The
The architecture
architecture
Figure 10.
Figure 10.
of the
of the
Theproposed
Theproposed
architecture
Figure
hybrid
architecture 10.
of The
hybrid
of the
model.
the proposed
architecture
model.
proposed hybrid
of the
hybrid model.
proposed hybrid model.
model.
Figure10.
Figure 10.The
Thearchitecture
architectureofofthe
theFigure
proposed
proposed hybrid
hybrid model.
model.
Figure 10.
10. The
The architecture
architecture of
of the
the proposed
proposed hybrid
hybrid model.
model.
Figure 10. The architecture of the proposed hybrid model.
Data Acquisition: Data collection is an essential first phase for the driver’s drowsiness
Figure 10. The architecture
Figure 10.
of the
Theproposed
architecture
hybrid
of the
model.
proposed hybrid model.
detection. Video data from the pi camera is collected to identify the driver’s face. A camera
is mounted on the vehicle’s dashboard to record the driver’s face. Subsequently, the cap-
tured video is divided into multiple images. The face is detected using a trained dataset
provided by National Tsing Hua University (NTHU) [33]. A physiologically based GSR
sensor is attached to the driver’s fingertips to collect the driver’s skin conductance. The
Sensors 2023, 23, 1292 12 of 19

Data Acquisition: Data collection is an essential first phase for the driver’s drowsiness
detection. Video data from the pi camera is collected to identify the driver’s face. A
camera is mounted on the vehicle’s dashboard to record the driver’s face. Subsequently, the
captured video is divided into multiple images. The face is detected using a trained dataset
provided by National Tsing Hua University (NTHU) [33]. A physiologically based GSR
sensor is attached to the driver’s fingertips to collect the driver’s skin conductance. The
GSR sensor and the pi camera are connected to the Raspberry-Pi controller, which detects
the driver’s face and skin conductance in real time. The driver’s bioelectrical signals are
then passed to the next phase, which is utilized to detect driver drowsiness.
Feature Extraction: In the second phase, after data acquisition, the data are processed
for feature extraction of the face, which is accomplished using the MTCNN algorithm.
MTCNN helps to identify the face in the image and apply landmarks from the data [29].
This information will be used subsequently to classify the driver’s drowsiness. The raw
data extracted by the GSR sensor are converted into digital form and further transmitted
for classification. Two parameters are employed for eye detection and mouth detection
utilizing PERCLOS and Frequency of Mouth (FOM), respectively, to detect the drowsy state
of the driver using behavioral measures. PERCLOS, a drowsy analysis technique, displays
the ratio of closed eyes based on the frequency of open and closed eyes. It is calculated as:

PERCLOS = NC/NOCL × 100 (5)

At any given time, NOCL stands for the total number of closed and open eye frames,
whereas NC stands for closed eye frames. FOM is the ratio of open squares to all squares
over a specified period. FOM calculations resemble PERCLOS calculations.

FOM = NOM/NMCO × 100 (6)

NOM represents the number of open-mouth frames, and NMCO represents the total
number of closed and open-mouth frames in a specified period [34].
Classification: In the third phase, the behavioral and physiological measures are
integrated to classify the driver drowsiness detection system. Various types of ML- and
DL-based classifiers can be integrated with the algorithm to evaluate the drowsy state of
the driver. SVM, CNN and HOG classifiers have been widely used for the evaluation [35].
A classification method like SVM is used to evaluate the driver’s current condition. If the
classifier determines that the user is not in a drowsy condition, the process begins at the
beginning. If the classifier detects a drowsy state, then it generates an alarm to inform
the driver or it goes back to the first phase and restarts the procedure. The steps of the
proposed hybrid model are mentioned in the Algorithm 1.

Algorithm 1: Working of Hybrid Model

Mount a camera on the dashboard of the vehicle and attach a GSR sensor on the fingers of
1:
the driver.
2: Capture the image and collect the reading of the skin conductance of the driver.
3: Detect the face in the captured images and forward the images to step 4 for feature extraction.
Extract facial features like the eyes and mouth from the images using MTCNN and convert
4:
the analog reading of skin conductance into digital form.
5: Forward the above results to step 6 for classification.
6: Classify the current state of the driver and forward the result to next step.
7: Generate an alarm if the driver is drowsy or restart the procedure at step 2.
 3:
ture extraction.
Extract facial features like the eyes and mouth from the images using MTCNN
4:
and convert the analog reading of skin conductance into digital form.
5: Forward the above results to step 6 for classification.
Sensors 2023, 23, 1292 6: Classify the current state of the driver and forward the result to next step. 13 of 19
7: Generate an alarm if the driver is drowsy or restart the procedure at step 2.

5.
5.Results
ResultsandandDiscussion
Discussion
Researchers
Researchers frequently use the
frequently use the NTHU
NTHU secondary
secondarydataset
datasetforfordriver
driverdrowsiness
drowsinessdetec-
de-
tection [33].
tion [33]. The
The NTHU-DDDdataset
NTHU-DDD datasetconsists
consistsofof36
36people
people of
of various
various ethnicities
ethnicities who
whowere
were
seen
seenyawning,
yawning,blinking
blinkingslowly,
slowly, dozing
dozing off
off and
and wearing
wearing glasses
glasses or or sunglasses
sunglassesunderunder both
both
day
day and
and night
night lighting
lighting conditions.
conditions. TheThesample
samplevideos
videosof
ofmale
male andand female
female drivers
drivers from
from
various
various ethnic
ethnic groups
groups comprise
comprise the the NTHU-DDD
NTHU-DDD dataset.
dataset. The
Thesample
samplevideo
video consists
consists of
of
different
different types
types of
of drowsy
drowsy and
and non-drowsy
non-drowsy events.
events. These
Thesevideos
videoswerewerealso
also recorded
recorded in
in
different lighting conditions [34,36]. The overall accuracy of this dataset is higher than the
different lighting conditions [34,36]. The overall accuracy of this dataset is higher than the
other
otherdatasets
datasetsavailable
availablefor fortraining
trainingthethe driver
driver drowsiness
drowsinesssystem
system[37].
[37]. Some
Somesamples
samplesofof
the
thedataset
datasetimages
imagesareare shown
shown in in Figure
Figure 11.
11.

(a) (b)

(c) (d)
Figure
Figure11.
11. NTHU
NTHUdataset
datasetwith
withdifferent
differentstates
stateswith
with glasses
glasses and
and without
without glasses:
glasses: (a,c)
(a,c)NDHU-DDD
NDHU-DDD
sample
samplewithout
withoutglasses;
glasses;(b,d)
(b,d)NDHU-DDD
NDHU-DDDsamplesample with
with glasses
glasses [33].
[33].

During the training process, the datasets in the study are divided into three groups: a
training set, a validation set, and a test set. The dataset contains a variety of NTHU-DDD
used in the multi-task architecture, since the mouth and the eyes are both identified on
images concurrently from the dataset. The NTHU-DDD dataset and MTCNN are utilized
together to detect the drowsy state. An image pyramid is created after scaling the source
image for input. After that, the altered image is sent to the P-net, which creates numerous
candidate face windows of various sizes. This process produces a somewhat unpolished
output. The R-net is then used to filter out even more overlapping windows and discard
them. Finally, the O-net determines whether the candidate window should remain open or
closed. In the end, the main facial feature points are revealed. Four landmark points in the
facial region are located and recorded using the MTCNN algorithm. To locate the driver’s
mouth and eye areas in the video, PERCLOS and FOM are calculated. The mouth and eye
detection regions are displayed in Figure 12. When the user closes his eyes, the area of the
eye turns red from blue. This algorithm also detects the user’s eyes opening and closing
status, even with glasses.
 them. Finally, the O-net determines whether the candidate window should remain open
or closed. In the end, the main facial feature points are revealed. Four landmark points in
the facial region are located and recorded using the MTCNN algorithm. To locate the
driver’s mouth and eye areas in the video, PERCLOS and FOM are calculated. The mouth
and eye detection regions are displayed in Figure 12. When the user closes his eyes, the
Sensors 2023, 23, 1292 14 of 19
area of the eye turns red from blue. This algorithm also detects the user’s eyes opening
and closing status, even with glasses.

(a) (b)

(c) (d)
Figure
Figure 12.
12. Detecting
Detectingfacial
facial landmarks
landmarks using
using MTCNN:
MTCNN: (a,b)
(a,b) face
face with
with open
open and
and closed
closed eyes
eyes in
in normal
normal
conditions; (c,d) face with open and closed eyes with glasses.
conditions; (c,d) face with open and closed eyes with glasses.

For
For driver
driver drowsiness
drowsiness detection,
detection, aacombination
combinationof ofbehavioral
behavioraland andsensor-based
sensor-basedphys-phys-
iological
iological measures
measures waswas deployed
deployed on on eight
eight individuals
individuals between
between25 25 and
and 48 48 years
years ofof age.
age.
This experiment was carried out in a simulated environment using
This experiment was carried out in a simulated environment using simulation parameterssimulation parameters
like
like PERCLOS,
PERCLOS,FOM FOMand andskinskinconductance.
conductance. PERCLOS
PERCLOS andandFOM values
FOM werewere
values usedused
in con-in
junction withwith
conjunction skinskin
conductance
conductance to determine
to determine a person’s current
a person’s state,
current such
state, suchas alertness
as alertness or
drowsiness.
or drowsiness. When
When a driver
a driveris is
drowsy,
drowsy, behavioral
behavioralreactions
reactionssuch
suchasaseyeeyestate,
state,yawning,
yawning,
and
and biological
biological signals
signals continue
continue to to vary.
vary. The
The drowsy
drowsy state
state of
of the
the driver
driver cancan thus
thus bebe assessed
assessed
by
by calculating
calculatingthethePERCLOS,
PERCLOS,FOM FOM and andskin conductance.
skin conductance. When the driver
When is driving
the driver nor-
is driving
mally, the duration of the eyes remaining open is much longer than the time of closure, so
normally, the duration of the eyes remaining open is much longer than the time of closure,
the PERCLOS
so the PERCLOS values are below
values the threshold
are below the threshold(0.24). WhenWhen
(0.24). the driver is drowsy,
the driver the dura-
is drowsy, the
tion of closure
duration is longer
of closure thanthan
is longer the the
time of of
time opening.
opening. Similarly,
Similarly,when
whena aperson
personyawns,
yawns, the the
mouth
mouth remains
remains open
open for
for aa few
few seconds
seconds (6 (6 s).
s). The
The driver
driver is
is considered
considered to to be
be drowsy
drowsy when
when
PERCLOS
PERCLOS >>0.24,0.24,FOM
FOM>>0.16 0.16and
andSC SC<250. When
< 250. Whenthethe
PERCLOS
PERCLOS > 0.24, FOM
> 0.24, FOM > 0.16 andand
> 0.16 SC
> 250, the person is less sleepy, and PERCLOS < 0.24, FOM > 0.16 and SC > 250 show the
SC > 250, the person is less sleepy, and PERCLOS < 0.24, FOM > 0.16 and SC > 250 show
the normal state of the driver. Table 4 represents the mean values of PERCLOS, FOM and
skin conductance for the eight subjects. MTCNN is used for detecting the facial features.
The accuracy of the model is achieved by training the model (split ratio), where 70% from
the provided dataset is used for training and 30% is used for testing purposes, which helps
to achieve 91% accuracy by reducing the false positive rate. The accuracy of the model is
achieved by:
TP + TN
Accuracy =
TP + TN + FP + FN
False Positive (FP): Subject misclassified as “drowsy,” where the subject was
actually normal.
False Negative (FN): Subject misclassified as “normal,” where the subject was
actually drowsy.
True Positive (TP): Subject truly classified as “drowsy,” where the subject was drowsy.
True Negative (TN): Subject truly classified as “normal,” where the subject was normal.
Sensors 2023, 23, 1292 15 of 19

Table 4. Mean value of PERCLOS, FOM and skin conductance for eight subjects.

Subjects
S1 S2 S3 S4 S5 S6 S7 S8
Parameters
PERCLOS 0.37 0.21 0.42 0.26 0.22 0.36 0.18 0.32
FOM 0.23 0.11 0.21 0.14 0.17 0.22 0.09 0.13
Skin Conductance 162.5 275.4 128.9 225.5 247.3 166.2 274.9 178.7

The pi camera is mounted in front of the driver to record facial features, and the GSR
sensor is attached to the driver’s fingers to collect the skin conductance data. This test was
conducted on eight individuals and completed in two weeks. Due to the availability of
just one simulated driving system, few users were available for testing. An experiment
was conducted in the morning on security guards who were assigned the night shift on
previous day to determine the true value of the drowsiness scale.
Table 5 displays the GSR levels for the eight individuals. It displays each person’s skin
conductance while driving the vehicle in a simulated environment. The value shows how
stress or drowsiness affects a person’s present state when driving over time. Throughout
the experiment, the system uses the sensor to track the subject’s GSR values and displays
the skin’s response every second. Each person’s skin response evolves gradually over time.
The skin response levels change marginally for the first several minutes while the drivers
remain active. However, the GSR value shows a change in extremely low dermal activity
with the passage of the 15-minute mark, which may cause drowsiness. Out of the eight
individuals, the GSR value of Subject 3 is observed as the lowest value (74.22 µs) and the
GSR value of Subject 2 as the highest value (348.2 µs).

Table 5. Representation of skin conductance of the eight subjects.

Duration S1 S2 S3 S4 S5 S6 S7 S8
5 181.81 348.2 168.88 231.03 257.42 160.54 286.88 190.23
10 164.87 319.35 150.03 235.47 229.85 156.47 295.23 203.13
15 134.34 305.41 74.22 264.67 215.49 142.79 328.34 172.82
20 154.95 297.13 123.25 221.74 247.89 135.38 302.18 144.51
25 133.82 253.77 119.51 193.13 193.55 147.9 219.76 167.26
30 125.21 298.82 108.08 207.18 199.38 154.27 247.34 174.17
The eight subjects are labeled S1–S8.

The line graph depicts the variation in galvanic skin response for the eight individuals
over time (every 5 minutes) in Figures 13 and 14. The line chart shows the change in the
GSR value of all the individuals. The GSR value of Subjects 1, 3, 5, 6 and 8 falls in the range
of the drowsy state, which is explained in Figure 9. The lower GSR value of the subject
indicates the drowsy state of the driver. Hence, detecting drowsiness using GSR values
reduces the limitations in a situation where the camera is not working effectively.
The parametric measures of the proposed hybrid model are compared with the other
driver drowsiness detection measures proposed by other researchers. Table 6 compares the
state-of-the-art studies with the proposed model [3,4,8,9,38]. The subjective measures are
evaluated according to the KSS and SSS ratings. These ratings cannot provide parametric
measures due to self-introspection that alerts the driver. In vehicle-based measures, 88%
accuracy is provided, but it did not work without road markings or in low light conditions.
The physiological measures provide promising parametric measures, but they are highly
intrusive. It is difficult for the driver to use such intrusive components in real driving
conditions. The behavioral measures outperformed with respect to all other measures,
but it is not feasible to use them alone due to the high false positive detection rate and
inability to work in low lighting conditions. In sensor-based physiological measures, the
drowsy state was identified by evaluating the respiration rate of the driver using the
impulse radio ultra-wideband (IR-UWB) radar system [38]. The accuracy of the system
 Sensors 2023, 23, 1292 16 of 19

claimed by the researcher is 86%, but it lacks consistency in data collection. It depicts that
the proposed hybrid model outperforms in all above-stated conditions. It uses a camera
Sensors 2023, 23, x FOR PEER REVIEW and GSR sensor that simultaneously collect the various behavioral measures, such 16 ofas19
eye
Sensors 2023, 23, x FOR PEER REVIEW and mouth position, and sensor-based physiological measures, such as the driver’s 16 of 19skin
conductance level. The proposed model has a few limitations that researchers intend
to The
overcome in the
line graph futurethe
depicts and is outlined
variation as follows:
in galvanic the model’s
skin response outcome
for the is analyzed
eight individu-
The time
alsusing
over line graph
(everydepicts
the single secondary the video-based
5 minutes) variation
in Figures in 13
galvanic
and 14.skin
dataset; aTheresponse
large dataset
line chartforshows
isthe eight
required individu-
for further
the change in
als over time (every 5 minutes) in Figures 13 and 14. The line chart shows
the GSR value of all the individuals. The GSR value of Subjects 1, 3, 5, 6 and 8 fallsgathering;
investigation. Limited drivers of a certain age were considered during thethe change
data in
in the
the more
GSR individuals
of all can be considered. The GSR sensor was connected 3, 5, to the microcontroller
range ofvalue
the drowsy the individuals.
state, which is The GSR
explained value of Subjects
in Figure 9. The1,lower 6GSR
and 8value
falls in
of the
the
using
range wires that may disturb the driver while driving; the GSR sensor needed to sendthethe
subjectofindicates
the drowsy state, which
the drowsy state isofexplained
the driver.inHence,
Figuredetecting
9. The lower GSR value
drowsiness usingofGSR
readings
subject wirelessly
indicates to the state
thelimitations
drowsy microcontroller, which helped the driver to drive the GSR
vehicle
values reduces the in a of the driver.
situation whereHence, detecting
the camera drowsiness
is not using
working effectively.
easily. In addition, advanced deep-learning techniques can help to build
values reduces the limitations in a situation where the camera is not working effectively. a real-time driver
drowsiness detection system [39].
350 319.35 350
350 305.41 297.13 298.82 350
320 348.2 319.35 320
305.41 297.13 298.82
320
290 348.2 320
290
(µs)

(µs)
264.67 253.77
290 290
(µs)

(µs)
260 231.03 235.47 264.67 253.77 260
221.74
Siemens

Siemens
260
230 231.03 235.47 207.18 260
230
221.74
Siemens

Siemens
193.13
230
200 181.81 207.18 230
200
164.87 193.13
200 181.81 154.95 200
170 164.87 134.34 133.82 170
Micro

Micro
154.95 125.21
170
140 168.88 134.34 133.82 170
140
Micro

Micro
150.03 125.21
140
110 168.88 140
110
150.03 74.22 123.25
110 119.51 110
80 74.22 123.25 108.08 80
119.51
80
50 108.08 80
50
50 5 10 15 20 25 30 50
5 10 15 20 25 30
Minutes
Minutes
Subject 1 Subject 2 Subject 3 Subject 4
Subject 1 Subject 2 Subject 3 Subject 4
Figure 13. Skin conductance graph of the first four subjects.
Figure 13. Skin conductance graph of the first four subjects.
Figure 13. Skin conductance graph of the first four subjects.

350 328.34 350

350 328.34 302.18 350
320 286.88 295.23 320
320 295.23 302.18 320
290 286.88 290
(µs)(µs)

(µs)(µs)

290 257.42 247.89 247.34 290

260 257.42 229.85 260
215.49 247.89 219.76 247.34
Siemens

Siemens

260
230 229.85 260
230
203.13 215.49 219.76 199.38
Siemens

Siemens

230 190.23 193.55 230

200 190.23 203.13 172.82 199.38 200
193.55
167.26 174.17
200
170 172.82 144.51 200
170
174.17
Micro

Micro

170 144.51 167.26 170

140 140
Micro

Micro

160.54 156.47 154.27

140 142.79 147.9 140
110 160.54 156.47 135.38 154.27 110
142.79 147.9
110
80 135.38 110
80
80
50 80
50
50 5 10 15 20 25 30 50
5 10 15 Minutes20 25 30
Minutes
Subject 5 Subject 6 Subject 7 Subject 8
Subject 5 Subject 6 Subject 7 Subject 8
Figure 14. Skin conductance graph of the last four subjects.
Figure 14. Skin conductance graph of the last four subjects.
Figure 14. Skin conductance graph of the last four subjects.
The parametric measures of the proposed hybrid model are compared with the other
The parametricdetection
driver drowsiness measuresmeasures
of the proposed
proposedhybrid model
by other are compared
researchers. with
Table the other
6 compares
driver drowsiness detection measures proposed by other researchers. Table 6 compares
the state-of-the-art studies with the proposed model [3,4,8,9,38]. The subjective measures
the
are state-of-the-art studiestowith
evaluated according the proposed
the KSS model [3,4,8,9,38].
and SSS ratings. Thecannot
These ratings subjective measures
provide para-
are evaluated
metric according
measures to the KSS and SSS
due to self-introspection thatratings. These
alerts the ratings
driver. cannot provide
In vehicle-based para-
measures,
metric measures
88% accuracy due to self-introspection
is provided, that alerts
but it did not work the road
without driver. In vehicle-based
markings or in lowmeasures,
light con-
Sensors 2023, 23, 1292 17 of 19

Table 6. Comparison proposed for hybrid model with different studies.

Drowsiness Detection
Reference Measures Accuracy Limitations
Methods/Sensors
Cannot be used in real driving
[3] Subjective KSS and SSS NA
conditions
[8] Vehicle-Based SWM and SDLP 88% High false positive detection rate
PERCLOS and
[9] Behavioral Close to 100% Does not work in all conditions
Yawning
Highly intrusive and cannot be used
[4] Physiological EEG and EMG 97–99%
in real driving conditions
Sensor-Based
[38] Respiration Rate 86% Low accuracy
Physiological
Behavioral
Need more investigation using large
Proposed Hybrid + PERCLOS, Yawning
91% number of individuals in real
Model Sensor-Based and Skin Conductance
driving conditions
Physiological

6. Conclusions
Drowsiness detection is vital to save precious human life and monetary losses. This
study proposes a hybrid drowsiness detection model using multiple measures to detect
driver drowsiness in all conditions that also reduces the false positive rate. It has been
concluded that none of the four distinct measures, taken separately, can ensure accuracy.
Each measure has limitations in different contexts and is ineffective in detecting drowsiness.
These limitations can be eliminated by combining two or more measures to detect driver
drowsiness and making the system work under all conditions. The literature review
indicates that combining behavioral measures, which are non-intrusive, with sensor-based
physiological measures, which are intrusive, produces better results and overcomes certain
limitations. A hybrid model that helps to detect driver drowsiness in all conditions is
proposed. The driver’s facial features are extracted using a camera as a behavioral measure
and the GSR sensor as physiological measure to investigate the transition from alert to
drowsy state. Improved accuracy and reduced false positive detection rates are the outcome
of the proposed model. This model considers a driver to be drowsy when PERCLOS > 0.24,
FOM > 0.16 and SC < 250. When PERCLOS > 0.24, FOM > 0.16 and SC > 250, the person
is less sleepy, and PERCLOS < 0.24, FOM > 0.16 and SC > 250 shows the normal state
of the driver. The mean value of PERCLOS and FOM are utilized in conjunction with
skin conductance to identify the driver’s current condition. Results are compared with
the threshold values of PERCLOS, FOM and skin conductance of the body, e.g., 0.24, 0.16
and 250, respectively. Additionally, the proposed hybrid model is also cost-effective and
easy to implement. The efficacy of the proposed model may be improved by integrating
other sensors, such as the PPG, pulse rate sensor and IR-UWB radar. In addition, advanced
deep-learning techniques can build a true real-time driver drowsiness detection system.

Author Contributions: Conceptualization, N.K.; Methodology, J.S.B.; Software, J.S.B.; Validation,

N.K.; Formal analysis, R.K.K.; Investigation, R.K.K. and H.L.G.; Resources, H.L.G.; Writing—original
draft, R.N.; Writing—review & editing, R.N.; Visualization, F.F.; Supervision, F.F. All authors have
read and agreed to the published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: The National Tsing Hua University (NTHU) dataset is used with
prior permission.
Conflicts of Interest: The author declares no conflict of interest.
Sensors 2023, 23, 1292 18 of 19

References
1. Road Traffic Injuries. Available online: https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries (accessed on
19 July 2022).
2. Albadawi, M.A.; Takruri, M.; Awad, M. A Review of Recent Developments in Driver Drowsiness Detection Systems. Sensors 2022,
2, 2069. [CrossRef] [PubMed]
3. Sahayadhas, A.; Sundaraj, K.; Murugappan, M. Detecting Driver Drowsiness Based on Sensors: A Review. Sensors 2012, 12,
16937–16953. [CrossRef] [PubMed]
4. Bajaj, J.S.; Kumar, N.; Kaushal, R.K. Feasibility Study on Amalgamation of Multiple Measures to Detect Driver Drowsiness. ECS
Trans. 2022, 107, 1951. [CrossRef]
5. Sakshi; Kukreja, V. A dive in white and grey shades of ML and non-ML literature: A multivocal analysis of mathematical
expressions. Artif. Intell. Rev. 2022. [CrossRef]
6. Machaca Arceda, V.E.; Cahuana Nina, J.P.; Fernandez Fabian, K.M. A survey on drowsiness detection techniques. CEUR Workshop
Proc. 2020, 2747, 152–161.
7. Soares, S.; Monteiro, T.; Lobo, A.; Couto, A.; Cunha, L.; Ferreira, S. Analyzing Driver Drowsiness: From Causes to Effects.
Sustainability 2020, 12, 1971. [CrossRef]
8. Doudou, M.; Bouabdallah, A.; Berge-Cherfaoui, V. Driver Drowsiness Measurement Technologies: Current Research, Market
Solutions, and Challenges. Int. J. Intell. Transp. Syst. Res. 2019, 18, 297–319. [CrossRef]
9. Bajaj, J.S.; Kumar, N.; Kaushal, R.K. Comparative Study to Detect Driver Drowsiness. In Proceedings of the 2021 International
Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, 4–5 March
2021; pp. 678–683.
10. Čolić, A.; Marques, O.; Furht, B. Driver Drowsiness Detection and Measurement Methods. In Driver Drowsiness Detection;
SpringerBriefs in Computer Science; Springer: Cham, Switzerland, 2014; pp. 7–18.
11. Doudou, M.; Bouabdallah, A.; Cherfaoui, V.; Charfaoui, V. A Light on Physiological Sensors for Efficient Driver Drowsiness
Detection System. Sens. Transducers J. 2018, 224, 39–50.
12. Sharma, M.; Kacker, S.; Sharma, M. A Brief Introduction and Review on Galvanic Skin Response. Int. J. Med. Res. Prof. 2016, 2,
13–17. [CrossRef]
13. Perkins, E.; Sitaula, C.; Burke, M.; Marzbanrad, F. Challenges of Driver Drowsiness Prediction: The Remaining Steps to
Implementation. IEEE Trans. Intell. Veh. 2022, PP, 1–20. [CrossRef]
14. Ngxande, M.; Burke, M. Driver drowsiness detection using Behavioral measures and machine learning techniques: A review of
state-of-art techniques. In Proceedings of the 2017 Pattern Recognition Association of South Africa and Robotics and Mechatronics
(PRASA-RobMech), Bloemfontein, South Africa, 30 November–1 December 2017; pp. 156–161.
15. Bajaj, J.S.; Kumar, N.; Kaushal, R.K. AI Based Novel Approach to Detect Driver Drowsiness. ECS Trans. 2022, 107, 4651. [CrossRef]
16. Murugan, S.; Selvaraj, J.; Sahayadhas, A. Analysis of different measures to detect driver states: A review. In Proceedings of the
2019 IEEE International Conference on System, Computation, Automation and Networking (ICSCAN), Pondicherry, India, 29–30
March 2019; pp. 1–6.
17. Cheng, B.; Zhang, W.; Lin, Y.; Feng, R.; Zhang, X. Driver drowsiness detection based on multisource information. Hum. Factors
Ergon. Manuf. 2012, 22, 450–467. [CrossRef]
18. Yang, G.; Lin, Y.; Bhattacharya, P. A driver fatigue recognition model based on information fusion and dynamic Bayesian network.
Inf. Sci. 2010, 180, 1942–1954. [CrossRef]
19. Horne, J.A.; Baulk, S.D. Awareness of sleepiness when driving. Psychophysiology 2004, 41, 161–165. [CrossRef] [PubMed]
20. Gwak, J.; Shino, M.; Hirao, A. Early Detection of Driver Drowsiness Utilizing Machine Learning based on Physiological
Signals, Behavioral Measures, and Driving Performance. In Proceedings of the 2018 21st International Conference on Intelligent
Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 1794–1800.
21. Timeline—Overview for Hybrid Measures . . . Publication Year: 2021, 2020, 2019, 2018, 2017, 2016, 2015, 2014, 2013, 2012 in
Publications—Dimensions. Available online: https://app.dimensions.ai/analytics/publication/overview/timeline?search_
mode=content&or_facet_year=2021&or_facet_year=2020&or_facet_year=2019&or_facet_year=2018&or_facet_year=2017&or_
facet_year=2016&or_facet_year=2015&or_facet_year=2014&or_facet_year=201 (accessed on 6 September 2022).
22. Buy a Raspberry Pi 3 Model B+—Raspberry Pi. Available online: https://www.raspberrypi.com/products/raspberry-pi-3-
model-b-plus/ (accessed on 4 October 2022).
23. Buy a Raspberry Pi Camera Module 2—Raspberry Pi. Available online: https://www.raspberrypi.com/products/camera-
module-v2/ (accessed on 4 October 2022).
24. Grove—GSR Sensor—Seeed Wiki. Available online: https://wiki.seeedstudio.com/Grove-GSR_Sensor/ (accessed on
4 October 2022).
25. Dhruba, A.R.; Alam, K.N.; Khan, M.S.; Bourouis, S.; Khan, M.M. Development of an IoT-Based Sleep Apnea Monitoring System
for Healthcare Applications. Comput. Math. Methods Med. 2021, 2021, 7152576. [CrossRef] [PubMed]
26. Austin, R.; Lobo, F.; Rajaguru, S. GSM and Arduino Based Vital Sign Monitoring System. Open Biomed. Eng. J. 2021, 15, 78–89.
[CrossRef]
27. Singh, N.; Brisilla, R.M. Comparison Analysis of Different Face Detecting Techniques. In Proceedings of the 2021 Innovations in
Power and Advanced Computing Technologies (i-PACT), Kuala Lumpur, Malaysia, 27–29 November 2021; pp. 1–6.
Sensors 2023, 23, 1292 19 of 19

28. Eye Blink Detection with OpenCV, Python, and dlib—PyImageSearch. Available online: https://pyimagesearch.com/2017/04/
24/eye-blink-detection-opencv-python-dlib/ (accessed on 2 September 2022).
29. Shi, W.; Li, J.; Yang, Y. Face Fatigue Detection Method Based on MTCNN and Machine Vision. Adv. Intell. Syst. Comput. 2020,
1017, 233–240.
30. Viola, P.; Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA, 8–14 December 2001; Volume 1,
p. I.
31. Deshmukh, A.D.; Nakrani, M.G.; Bhuyar, D.L.; Shinde, U.B. Face Recognition Using OpenCv Based on IoT for Smart Door. In
Proceedings of the International Conference on Sustainable Computing in Science, Technology and Management (SUSCOM),
Jaipur, India, 26–28 February 2019; pp. 1066–1073.
32. Yongcun, W.; Jianqiu, D. Online Examination Behavior Detection System for Preschool Education Professional Skills Competition
Based on MTCNN. In Proceedings of the 2021 IEEE 2nd International Conference on Big Data, Artificial Intelligence and Internet
of Things Engineering (ICBAIE), Nanchang, China, 26–28 March 2021; pp. 999–1005.
33. Chen, C.S.; Lu, J.; Ma, K.K. Driver Drowsiness Detection via a Hierarchical Temporal Deep Belief Network. Asian Conf. Comput.
Vis. 2016, 10116 LNCS, 117–133.
34. Savaş, B.K.; Becerikli, Y. Real Time Driver Fatigue Detection System Based on Multi-Task ConNN. IEEE Access 2020, 8, 12491–12498.
[CrossRef]
35. Nor Shahrudin, N.S.; Sidek, K.A. Driver drowsiness detection using different classification algorithms. J. Phys. Conf. Ser. 2020,
1502, 12037. [CrossRef]
36. Liu, W.; Qian, J.; Yao, Z.; Jiao, X.; Pan, J. Convolutional two-stream network using multi-facial feature fusion for driver fatigue
detection. Future Internet 2019, 11, 115. [CrossRef]
37. Jabbar, R.; Al-Khalifa, K.; Kharbeche, M.; Alhajyaseen, W.; Jafari, M.; Jiang, S. Real-time Driver Drowsiness Detection for Android
Application Using Deep Neural Networks Techniques. Procedia Comput. Sci. 2018, 130, 400–407. [CrossRef]
38. Siddiqui, H.U.R.; Saleem, A.A.; Brown, R.; Bademci, B.; Lee, E.; Rustam, F.; Dudley, S. Non-Invasive Driver Drowsiness Detection
System. Sensors 2021, 21, 4833. [CrossRef] [PubMed]
39. Phan, A.C.; Nguyen, N.H.Q.; Trieu, T.N.; Phan, T.C. An efficient approach for detecting driver drowsiness based on deep learning.
Appl. Sci. 2021, 11, 8441. [CrossRef]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.