Enhanced Driver Drowsiness Detection using

Ensemble ML and DL Predictions based on Facial

Features
Diksha Manu
Dept of Information Scinece
Cambridge Insitute Of Technology
Bengaluru,Karnataka
diksha.20ise@cambridge.edu.in
Impana S
Dept of Information Science
Cambridge Institute Of Technology
Bengaluru, Karnataka
impanas.20ise@cambridge.edu.in
Pavithra VN
Dept of Computer Science
Cambridge Insitute Of Technology
Bengaluru,Karnataka
pavithravn.20cse@cambridge.edu.in

Precilla Mary B Dr.Jayanthi M.G Prashanth Kannadaguli,

Dept of Computer Science Associate Professor Senior Data Science Trainer
Cambridge Insitute Of Technology Dept of Computer Science Dhaarini Academy of Technical
Bengaluru,Karnataka Education,
precilla.20cse@cambridge.edu.in Cambridge Insitute Of Technology
Bengaluru,Karnataka Bengaluru, India.
Jayanthi.cse@cambridge.edu.in prashscd@gmail.com

Abstract—In response to the escalating contribution of driver

drowsiness to global traffic accidents, we propose a real-time,
non-intrusive system for accurately identifying driver
sleepiness. The system continuously captures images and
employs a prescribed methodology to measure the driver's gaze
position. By analyzing facial features such as eye closure, mouth,
and head postures, our approach provides an accurate picture
of the driver's state without imposing any negative effects. This
method's non-intrusiveness enhances its reliability and
predictive capabilities, enabling timely alerts to prevent
potential accidents and protect both people and property on the
road. Our research contributes to the advancement of driver
safety systems and underscores the significance of non-intrusive
techniques in combating driver drowsiness effectively.
Keywords— Driver monitoring system, face detection, and
driver weariness are all used in driving.
I.INTRODUCTION
Due to lack of sleep, prolonged driving, or other medical
conditions including brain illnesses, the attention level of the
driver diminishes. According to several studies on traffic
accidents, driver weariness is a factor in about 40% of
collisions. Excessive fatigue is brought on when a person
drives for longer than is healthy for them, and this weariness
also leads to drowsiness, which makes the person fall asleep
or lose consciousness [1]. There are one types of exhaustion:
sleep-related fatigue, the latter of which is influenced by
factors such as the quantity and quality of sleep, shifts in the
and other factors [2]. There is a high risk of fatalities, serious
injuries, and vehicle damage due to the drowsy driving. It is
crucial to take considerable action to enhance drivers' working
circumstances as the single most critical safety element, in
order to lessen the harmful effects that a drowsy driver may
cause. By rendering their vital services in a variety of sectors
related to numerous facets of life, computer science and
engineering play a responsible part in the advancement and
improvement of society. Awake, non-rapid eye movement
(NREM), and rapid eye movement (REM) sleep are the three
stages of sleep. The transition from awake to sleep, often
known as drowsy, is the initial stage of the second stage
NREM and is where people fall asleep or enter into micro
sleep subconsciously [3]. Behavioral Using mounted cameras
in the automobile to track facial traits like eye state, head
movement, blink rate, and other indicators of tiredness
yawning [4].
II. LITERATURE REVIEW
Literature Review proposes a method for gauging driver
tiredness utilizing a wearable technology could collect
information and then be further analyzed to ascertain the
driver is genuinely sleepy. The authors created a wearable
biosensor known as Bio-harness_3 to collect physiological
data from drivers. Additionally, the system for detection will
operate in two parts, the initial phase consisting gathering of
information and studying by the model [5]. Additionally, a
driver-alerting. The second phase will include a warning from
the application system. But the study discovered. The study
shown, however, that the placement and configurations of
wearable technology safe driving in routine circumstances,
the complexity may increase have an impact on driving
activity [6]. To decrease accidents caused by driver
inattention, authors in [7] have installed a sleepiness detector
on an automobile The sensor can operate with great precision
and be deployed as an integrated device. The study described
a unique technique for identifying tiredness in driver in real-
time depends on DL, can be set up affordably, has more
accuracy, and can analyze data immediately. compression
from a large baseline model to a small one form is the key
contribution. The system may determine a driver's suitability
for driving by facial landmarks being used as input. Self-
driving cars may eventually contribute significantly to the
transportation sector by enhancing comfort and safety,
according to authors of [8]. The self-driving automobiles are
intended to be constructed in a specific manner that the
vehicles are capable of interacting with human-centric
vehicles and carrying out tasks independently. The
autonomous vehicles can sense traffic signs, prevent crashes,
and change their speed to reduce travel time and boost safety.
In [9] the authors put up a tiredness identifying and alerting
system that couples information gathered from the yawning
and eye movements while simultaneously implementing
RGB-D cameras. Additionally, the system may inform the
driver if it detects sleepy depends on the data it has processed.
However, the outlined. The model is unclear how yawning
state is for identifying tiredness.

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

 III.METHODOLOGY
A. Dataset Description
The method of gathering data indicates that to the systematic
gathering of information. We are collecting the data
depending on specific goals of our project.
 Research objectives: Firstly, we are going to know
about reason for information gathering & what
information should be collected.
 Data sources: The datasets are collected from the
Fig.3: Head dataset
internet using some Websites like: [Kaggle,
Teachable machine]. Facial expression: like mouth opening and closing features
 Data Collection: We are collected the dataset based which will help the driver fatigue detection mechanism as
on our need we are almost collected the data. At depicted in Fig. 4.
present we are having data for we needed to do
project.
The data set is collected from the live video file of persons
who are driving the car. Features of users are captured using
the live camera and live tracking of eye movements is
captured and used as input data for taking a decision. The
shape predictor landmark is used for verifying user data [10].
the video is converted into images/frames then we divided
datasets according to the features captured in the video then.
Eyes-related features: monitoring the movements of the eyes
i.e., eyes open and close duration (as shown in Fig.1and
Fig.2), blink rate, and eye gaze direction which will indicate
the signs of drowsiness and fatigue. Fig.4: Mouth dataset
B. Augmentation
Traditional, white-box approaches or black-box techniques
DNN categorized into two categories. fall existing image
augmentation techniques can be divided. we provide a brief
overview of the groupings of techniques that have had the
most effects on picture synthesis and enhancement [11].
Traditional transformations are Combining affine picture
transformations with color alteration is the current standard
practice for data augmentation. Rotation, reflection, scaling
(zoom in/out), and shearing are the affine transformations
that we define [12] as shown in Fig.5 and Fig. 6.
Fig.1 Eye open

Fig. 2: Eye closed

Head Positions and orientation: tracking the head positions
and orientation using head to detect unusual patterns such as
drooping head or head nodding as depicted in Fig. 3. Shear Zoom in
Fig.5: the same image after various affine adjustments
 FE. Nevertheless, most FE approaches still struggle with the
issue of extracting the right characteristics that can precisely
capture the underlying content of a piece of data or dataset
[15].
1)Eye Aspect Ratio (EAR)
Six points representing each left eye will serve as the
starting point for the measurement, which will then
calculate region of the contour surrounding each eye
[12] as depicted in Fig.9.

Reflection Rotation
Fig.6 Same image after reflection & rotation transformations

Increasing the quantity of samples used for deep neural

training models and balancing the extent of datasets are two
prominent uses of deformations. It is commonly utilized as
affine data manipulations augmentation, although further
research is needed. The popularly used method is being
sharpening, blurring, white-balancing, histogram
equalization, and boosting contrast or brightness [13] as
depicted in Fig.7 and Fig.8.

Contrast+20 Hist equalization Fig.9 The state recognition for eye

Real-time gaze identification done by employing

facial landmarks and the relationship between the
height and width of the point stated above. This
relationship results in the equation for eye aspect
ratio, which is illustrated in equation. (EAR).
||𝑋2 −𝑋6 ||+||𝑋3 −𝑋5 ||
EAR= 2||𝑋1 −𝑋4 ||

where the coordinates of the 2D landmarks are X1,

Fig.7 The identical picture after many colors’ modifications
White balance sharpens
2)Yawning Status
Fig 8 Following white balance, the same picture & sharpen
transformations
These straightforward techniques have been demonstrated to
be quick, reproducible, and reliable, and the implementation
code is reasonably simple and accessible for the most
common frameworks for deep learning, which raises public
interest in them [14]. Traditional data augmentation
techniques, which include affine picture transformations and
color change, are quick and simple to use and have a track
record of being effective for boosting training dataset [11].
C.Feature Extraction
In image retrieval, processing, data mining, and computer
vision, feature extraction (FE) is a crucial step. Extraction of
Useful unstructured data-derived information is referred to as
X2…and X6. When eyelids are open, the EAR
essentially remains constant, and when it is closed,
it approaches zero. The mean is calculated since
both eyes blink simultaneously [16].
The location of the driver's established from facial
landmarks, allowing us to identify regardless of
whether the driver yawning right now. This is
accomplished by keeping track of how often the
driver has yawned in the previous five minutes. The
hue is simple to distinguish due to sharp contrast
between the coloring the mouth. The largest linked
component can be located within following some
post-processing, such as erosion. A geometric
aspect then supports the interaction between the
mouth region and the gaze. The tiredness
identification procedure consists of two parts. First
it was determined that a meant the person was
yawning. Second the image's pixel values earlier
detected mouth information has been compared in
order to confirm its authenticity. A mouth that is
 moving 45 times frames, it is considered to be one
yawning [12] as depicted in Fig.10.
Fig.10 The state recognition for mouth.
D.Gray Scaling
The process of successfully and tastefully introducing color
to grayscale, monochromatic photographs is known as
grayscale to color picture conversion. These days, automated
conversion is a challenging field that blends deep learning,
machine learning, and artistic methods. Even though it is
generally accepted among professionals that grayscale
photographs have a special artistic value, color may be
perceived as a loss of information. This study provides a
summary of techniques and steps for grayscale photo
colorization. [17] as shown in Fig.11, Fig.12 and
Fig.13.Grayscale scaling is a popular picture preprocessing
method for turning color photos into grayscale images, which
is utilized in driver drowsiness detection. In this method, the
RGB (Red, Green, and Blue) color channels are combined
into a single channel that represents the grayscale image's
intensity. Grayscale image conversion simplifies data and
lowers computer complexity by focusing on brightness or
intensity information rather than color information. Eye
movements, facial emotions, and head positions, among other
crucial face traits and patterns for drowsiness detection,
clearly captured in grayscale photos [18].
Fig.11 Gray-scale of eye images
Fig.12: Gray-scale of head images
Fig.13: Gray-scale of mouth images
E.Image Resize
1)Image Resizing
For consistency and compatibility with Convolutional
neural network models (CNNs), it is frequently
necessary to resize the input photos to a fixed size. In
this instance, the photos are adjusted to have 224x224
pixel dimensions [19].
2)Image Aspect Ratio
In order to avoid distortion, it is crucial to maintain the
aspect ratio when scaling the photographs. Resizing
while keeping the aspect ratio guarantees that the
image's object proportions are accurate [11].
3). Libraries for image processing
Several libraries for image processing offer functions
resize images to particular dimensions. Libraries like
OpenCV, PIL (Python Imaging Library), and scikit-
image provide interpolation techniques in addition to
resizing capabilities to obtain the desired size. [12] [13]
[20].
F.CNN Architecture
The pooling layers reduce the computational cost by down
sampling the feature maps, while the convolutional layers use
filters to capture features like edges, corners, and textures.
Classifying the extracted features is done by the completely
linked layers [21] [22]. Training Data: A sizable dataset of
labelled pictures or videos is needed to train a CNN for driver
sleepiness detection. Both tired and awake drivers should be
represented in this dataset in a variety of ways. The CNN
gains the ability to identify the patterns connected to
sleepiness by training on this dataset [21]. Preprocessing:
Techniques for preprocessing may be used to enhance
important features and lower noise before feeding pictures or
video frames into the CNN. Images are frequently resized,
made grayscale, and had their pixel values normalized as part
of the preprocessing process [7] [23]. Transfer Learning:
With little training data, CNN performance can be improved
by using transfer learning. In order to detect driver
drowsiness, this method uses pre-trained tuned using big
visual datasets. Transfer learning can help the CNN detect
tiredness by utilizing the information gained from the original
job [22].
G.Random Forest
For the purpose of making predictions, the Random Forest
method mixes several decision trees. It is a specific kind of
ensemble learning method. It creates a lot of decision trees
during the training phase, and during the output phase it
creates the class that is the average of the classes predicted by Head 20,000
each tree [24] [25]. The Random Forest technique for driver
sleepiness detection requires the extraction of pertinent Classes Drowsy Non-Drowsy
features from the input data. The use of head posture
Test 30,000 30,000
information, mouth-related cues, and eye-related cues (such
as eye closure length) is common in this setting [26]. Training Train 30,000 30,000
and Testing: To train the Random Forest model, a labelled
dataset with examples of cases of both sleepy and not-drowsy
driving is necessary. To ensure generalizability, the dataset F1- Precision Recall
should contain a range of events and subjects. The score
Model Accura
functionality of the training model is next assessed with cy
unobserved data [27]. Feature Importance: Random Forest Convolutional 96% 93% 93% 93%
offers a metric for feature importance that quantifies the Neural Network
relative weights that various characteristics have during the
Random Forest 86% 85% 85% 85%
classification procedure. This knowledge is used for best
indicators of driver intoxication [28]. These measurements
shed light on the model's capacity to distinguish between tired
and awake states. [29]

H. User-friendly Interface
User-friendly platform is implemented using Tkinter
interface. Without any prior programming experience, users
will be able to input video, view the drowsy state or not, and
interact with the model as depicted in Fig.14 and Fig.15.

Fig.16 Plot for Accuracy

Fig 14 App UI Input

Fig.17: Plot for Loss

Fig 15 App UI Output

IV. RESULT Fig.18 AUC Graph

The result obtained from the model is evaluated by using V. DISCUSSION

accuracy as in Fig.16, Fig.17 and Fig.18.Also its performance In this research, the various methods for detecting driver
metrics. It’s displayed in the form of table below: tiredness have been reviewed. Few techniques are been
employed to create the fatigue identification in driver system.
Features Number of datasets
Many automakers have employed the vehicle-based driver
Eyes 20,000 sleepiness detection system to make their systems, although
Mouth 20,000 this method heavily realize the surrounding infrastructure, the
road's structure and surface markings, the vehicle's kind, and
the driver's driving abilities [30]. When compared to few
IEEE papers our accuracy is between 86-96%.

VI. CONCLUSION
Finally, the experiment on the driver sleepiness detection
system from the video was effective in developing an
accurate methodology for foretelling driver’s drowsiness
based on face images/frames. The project uses Canny edges
and other pre-processing techniques as well as grayscale
conversion and image scaling. It employed (CNN)
architecture and random forest model’s fatigue. After being
training and validating using the accuracy measures, the
results demonstrated that the models were adequate to
correctly identifying sleepy. A user-friendly interface was
created with Gradio to increase accessibility and user
interaction. Users could now upload films and receive real-
time forecasts of driver fatigue from trained algorithms with
ease.

REFERENCES

[1] Radhika N. Gandhi, 2 Parth R. Ambhorkar, 3Ayush A. Datir, 4 Pratik

R. Kale, 5 Prof.Shwetambari G. Pundkar,” Driver Drowsiness
Detection System using Embedded System”, 23-May-2022.
[2] I. -R. Adochiei et al., "Drivers’ Drowsiness Detection and Warning
Systems for Critical Infrastructures," 2020 International Conference on
e-Health and Bioengineering (EHB), Iasi, Romania, 2020, pp. 1-4, doi:
10.1109/EHB50910.2020.9280165.
[3] Sleep-Information about Sleep. (n.d.). Retrieved [26] Liao, S., & Chen, T. (2015). by supervised sparse representation.
fromhttps://science.education.nih.gov/supplements/webversions/sleep Journal of Intelligent Transportation Systems, IEEE 16(2), 865-874.
disor ders/guide/info-sleep.html [27] Huang, Y., & Wei, J. (2019). based on machine learning: A review.
[4] M. Ngxande, J. -R. Tapamo and M. Burke, "Driver drowsiness IEEE Access, 7, 187951-187966. [28] Goodfellow, Ian, et al. "Deep
detection using behavioral measures and machine learning techniques: learning." MIT press. 2016.
A review of state-of-art techniques," 2017 Pattern Recognition [28] Chen, K., Lian, J., & Lu, K. (2017). Random forest method is used to
Association of South Africa and Robotics andMechatronics (PRASA- detect driver fatigue. Within the International Conference on
RobMech), Bloemfontein, South Africa, 2017, pp. 156-161, Computer, Communication, and Management (ICCCM) (pp. 101-104).
[5] J. Ahmed, J. Li, S. A. Khan and R. A. Shaikh, "Eye behaviour based [29] Kim, J., Kim, J., & Kim, H. (2018). Real-time on random forest
drowsiness Detection System," 2015 12th International Computer classifier using steering wheel angle. Journal of Sensors, 2018,
Conference on Wavelet Active Media Technology and Information 8920841.
Processing (ICCWAMTIP), Chengdu, 2015, pp. 268-272. [30] .J. S. Bajaj, N. Kumar and R. K. Kaushal, "Comparative Study to
[6] Khan, M. Q., & Lee, S. (2019). A Comprehensive Survey of Driving Detect Driver Drowsiness," 2021 International Conference on Advance
Monitoring and Assistance Systems. Sensors, 19(11), 2574. doi: Computing and Innovative Technologies in Engineering (ICACITE),
10.3390/s19112574 Greater Noida, India, 2021, pp. 678-683, doi:
[7] Reddy, Bhargava & Kim, Ye-Hoon& Yun, Sojung&Seo, Chanwon& 10.1109/ICACITE51222.2021.9404761.
Jang, Junik. (2017). Real-Time Driver Drowsiness Detection for
Embedded System Using Model Compression of Deep Neural
Networks. 438-445. 10.1109/CVPRW.2017.59.
[8] J. Straub et al., "An internetworked self-driving car system-
ofsystems," 2017 12th System of Systems Engineering Conference
(SoSE), Waikoloa, HI, 2017, pp. 1-6.
[9] Yazdi, M. Z., &Soryani, M. (2019). Driver Drowsiness Detection by
Identification of Yawning and Eye Closure. International Journal of
Automotive Engineering, .
[10] K. Satish, A. Lalitesh, K. Bhargavi, M. S. Prem and T. Anjali., "Driver
Drowsiness Detection," 2020 International Conference on
Communication and Signal Processing (ICCSP), Chennai, India, 2020,
[11] "Data Supplements to deep learning for image categorization
problem," 2018 International Interdisciplinary PhD Workshop
(IIPhDW), Winoucie, Poland, pp. 117–122. A. Mikoajczyk and M.
Grochowski.
[12] "An Algorithmic The 2020 IEEE Region 10 Symposium (TENSYMP),
"Approach to Driver Drowsiness Detection for Ensuring Safety in an
Autonomous Car," Dhaka, Bangladesh, 2020, pp. 328–333. M. M.
Islam, I. Kowsar, M. S. Zaman, M. F. Rahman Sakib, and N. Saquib.
[13] Galdran, A., Alvarez-Gila, A., Meyer, M. I., Saratxaga, C. L., Araújo,
T., Garrote, E., ... &Campilho, "Data-Driven Colour Enhancement
Methods for Deep Skin Images Analysis”, ArXivPrepr.
ArXiv170303702, 2017.
[14] The efficiency of data augmentation in picture categorization using
deep learning, J. Wang and L. Perez, technical paper, 2017.
[15] A. O. Salau and S. Jain, "Feature Extraction: A Survey of the Types,
Techniques, Applications," 2019 International Conference on Signal
Processing and Communication (ICSC), NOIDA, India, 2019, pp. 158-
164.
[16] Owais, S. blinktotext/log/68360-eye-blink-detection-algorithms.
[17] "An Overview of Grayscale Image Colorization Methods," 2020
International Symposium ELMAR, Zadar, Croatia, (pp. 109–112) by I.
eger and S. Grgi.
[18] "Real-Time Driver Drowsiness Detection Using Artificial Neural
Networks", 2nd International Conference on Control, Instrumentation,
and Automation (ICCIA), 2014, Ehsan Behravan, EhsanollahKabir,
and Mahdi Rezaei.
[19] Deng, W., Hu, J., Guo, Y., & Zhu, X. (2019). An algorithm to identify
tiredness in drivers in real-time using convolutional neural networks.
Electronics, 8(2), 162.
[20] Bradski, G., &Kaehler, A. (2008). Learning OpenCV: Computer vision
with the OpenCV library. O'Reilly Media.
[21] In 2020, Ren, C., Li, W., Song, Z., and Chen. detection of driver
inattention using a cutting-edge CNN algorithm. 8 IEEE Access

[22] Khan, F., Hussain, A., & Amin, Y. (2021). Driver drowsiness models.
International Journal of Advanced Computer Science and
Applications, 12(2), 73-82.
[23] Wang, Z., Zhao, Y., Liu, W., & Liu, J. (2018). Real-time driver
drowsiness detection via CNN-based facial landmarks and multiscale
entropy.
[24] Breiman, L. (2001). Random forests. Machine learning, 45(1), 5-32..
[25] Liaw, A., & Wiener, M. (2002). Classification and regression by
randomForest. R News, 2(3), 18-22.