See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/344633633

Development of an Enhanced Drowsiness Detection Technique for Car Driver

Article in Xi'an Dianzi Keji Daxue Xuebao/Journal of Xidian University · October 2020
DOI: 10.37896/jxu14.6/369

CITATION READS

1 666

2 authors, including:

Naveen Kumari
Punjabi University, Patiala
26 PUBLICATIONS 24 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Comparison between artificial View project

Facial Emotion Recognition View project

All content following this page was uploaded by Naveen Kumari on 13 October 2020.

The user has requested enhancement of the downloaded file.

Journal of Xidian University https://doi.org/10.37896/jxu14.6/369 ISSN No:1001-2400

Development of an Enhanced Drowsiness Detection

Technique for Car Driver
Manishi#1, Naveen Kumari*2

#*
Department of Computer Science, Punjabi University Regional Centre for Information Technology and Management, Mohali

Abstract— Drowsiness or fatigue is one of the major causes of road accidents. Many fatal accidents can be prevented if the drowsy
drivers are warned in time. A variety of drowsiness detection techniques exist that monitor the state of the driver while driving and a
warning alert is triggered if they do not concentrate on driving. This paper presents development of an enhanced Drowsiness
Detection technique by considering the behavioural parameters i.e. eyes and yawning. CNN , a deep learning technique is used for
classification of the parameters. NTHU Drowsy Driver Detection Dataset has been used to extract the images considering four
scenarios: BareFace, Glasses, ,Night – BareFace and Night-Glasses [11]. MATLAB® has been used for the implementation of the
techniques. Firstly, face is detected using the Viola-Jones algorithm, AdaBoost classifier for the feature selection and cascade of
classifiers for feature extraction. Eye Detection and Mouth Detection techniques have also been mentioned which tell the state of the
driver whether active or sleepy. The proposed technique is able to detect the driver’s drowsiness state with an accuracy of 98.4%.In
the end, the conclusion of the research findings may help researchers for finding further work in the relevant field.

Keywords— Driver drowsiness, fatigue detection, classification, yawning, eye closure, Convolutional Neural Network (CNN)

I. INTRODUCTION

Driver’s Drowsiness is one of the major concerns that can lead to road accidents. Drowsiness or fatigue
can be due to exhaustion, mental stress, and lack of proper sleep, boredom or alcohol consumption. These
factors can also be fatal in certain scenarios. Various studies have shown that many road accidents are
fatigue based. Due to technology and advancement, many techniques have come into existence which can
be fruitful to detect the driver’s state.
Detecting Driver’s Drowsiness is one of the measures to ensure road safety. There have been remarkable
contributions by many researchers for fatigue detection. Many android based applications have been
invented to make the driver alert by looking at the parameters such as face, eyes, head positioning,
yawning.
People who work in night shifts and plan for long distance travelling are more feasible to drowsiness.
Therefore, lack of sleep can make the person’s eyes blink, lead to distraction, and change the head position
which could lead to serious casualties. Hence, it is important to detect a driver's drowsiness by considering
these parameters and make a technique which would make the driver alert and save his life.
This paper is organized into five sections. In section II, the review of the work related to the topic is
presented. Section III describes the proposed methodology. Experimental setup and implementation has
been discussed in Section IV. The computational results obtained from experiments are discussed in
section V. Finally, in Section VI, conclusion and future research are presented.

II. RELATED WORK

To acquire the knowledge related to the work, one needs to review the previous articles. A brief review of
the related work is listed below:

Ashish Kumar and Rusha Patra[1] designed a method that was low cost real time drowsiness detection
with increased accuracy. In the method implemented, a webcam was firstly used to record and the face
was detected from the recorded video from each frame using the image processing techniques. Eye aspect
ratio, nose length ratio and mouth opening ratio was computed from the landmarks on the detected face.
Machine learning algorithms have been implemented in the method along with the SVM classifier. Their

VOLUME 14, ISSUE 6, 2020 3181 http://xadzkjdx.cn/

Journal of Xidian University https://doi.org/10.37896/jxu14.6/369 ISSN No:1001-2400

method outperformed the value of the accuracy that would have been calculated from the Bayesian
technique. They both calculated the sensitivity and the overall accuracy.

Muhammad Tayab Khan, Hafeez Anwar et al. [2] proposed a method for real time video by
determining a person’s eye whether opened or closed. The method included a four step procedure. Firstly,
the determination of the face from the entire region including background. Secondly, extraction of face
from the videos. The methodology only considered the Region of Interest i.e. eyes. It further removed
noise such as eyebrows and focused only on the curvature of the eyes. Based upon the curvature of the
eyes, it was classified as opened or closed and it will trigger the alarm system invented to make the driver
alert.

Jang Woon Baek et al.[3] proposed a novel drowsiness detection technique by acquiring the image of the
driver and then estimated the landmarks on the face region. The algorithm used the AdaBoost Classifier
method. The eye state (open/close) was determined by the Eye Aspect Ratio. The proposed algorithm
provided a real time performance that could be used on any system.

Omar Wathiq and Bhavna D. Ambudkar[4] designed a novel method to improve the efficiency of the
driver’s safety. Firstly, they reviewed the approaches that were used to implement the drowsiness
detection and based on the limitations of the discussed approaches they developed their own new method.
Their method had the objective to primarily improve the safety technique. Their technique is based on
optimized face detector and feature extraction with SVM classifier.

Jun-Juh Yan, Hang-Hong Kuo et al.[5] developed a drowsiness detection using the PERCLOS and the
gray scale image processing. Their methodology consisted of three parts. Firstly, it calculates the position
of the driver in gray scale image and the eye positions are analyzed with and without glasses. Secondly,
the data is extracted and a fatigue model is established. As soon as the model is examined by continuously
monitoring the driver’s state, the alert system triggers the warning to the driver.

Bappaditya Mandal et al. [6] proposed a vision-based fatigue detection method for the heavy vehicles
such as buses and trucks and not for the cars. A measure percentage of eyelid closure (PERCLOS) is
implemented which checks for the eye’s openness and closure. A systematic analysis of the algorithm
proposed was experimented in which the analysis of continuous levels of open eyes is made and a fusion
algorithm is implemented to check the status of both eyes. The proposed robust method is performed for
an oblique angle as well which could be a challenging factor. The experimental results showed that the
method distinguished the sleepy state from the normal state.

Rateb Jabbar et al. [7] proposed a novel approach for real time fatigue detection. The proposed method
was implemented on the android application. The approach used the deep neural network technique i.e.
Multilayer’s perceptron classifier which took the facial images as an input. Their work compressed a
heavy baseline model to a lightweight model i.e. it can be used for compact android devices. Also, a
structure was further invented which primarily focused on the facial landmarks key point detection to
check whether the driver is sleepy or not.

Marchel T. Tombeng et al.[8] designed a technique for drowsiness detection that works for the Android
Operating System. In order to avoid road accidents due to fatigue, the authors used OpenCV library for
digital image processing. OpenCV library uses a Haar Cascade Classifier for detection of the eyes and
face. They further concluded that the camera which captured the images of the driver can be connected to
the android system with a wireless technology.

Maninder Kahlon and Subramaniam Ganesan [9] designed a drowsiness detection technique based on
binary eyes image data. MATLAB had been used for the pre-processing of the images captured from the
camera. Eyes were cropped from the images and then converted into grayscale and then into binary
images. After filtering the binary image, black to white pixel ratio was detected and the black pixel was

VOLUME 14, ISSUE 6, 2020 3182 http://xadzkjdx.cn/

Journal of Xidian University https://doi.org/10.37896/jxu14.6/369 ISSN No:1001-2400

counted for every column. Black to White pixel ratio is higher for the eyes opened than for the eyes
closed. Deciding and verifying the shape, the level of drowsiness was detected and the driver was
informed.

Menchie Miranda et al.[10] proposed a system for the driver’s safety that was based on the scanning of
the eyelid movement. The report was sent to the driver automatically from the web application through the
internet of things. The technology uses the internet of things so that the owner of the vehicle would be
aware of the driver’s actions. They conducted many types of tests, for instance normal status with regular
eyes, normal status with small eyes and so on. Their algorithm was built with dlib, OpenCV, Python
installed in Raspberry Pi 3. Their experiments achieved a high rate of accuracy.

III. PROPOSED METHODOLOGY

The aim of the proposed work is to develop improved driver drowsiness detection for car drivers. The
proposed work will be implemented by using the MATLAB® software. It consists of image processing
and deep learning toolboxes and built-in classifiers. It provides an interactive environment for
implementing and developing algorithms. Steps along with the general framework that will be
implemented are discussed in this section.

Fig. 1 A general framework for the proposed methodology

Step I. Extracting images from the dataset of NTHU-DDD videos.

Step II. Pre-processing of the images for better results.
Step III. The output of the pre-processed phase is provided to the feature extraction phase (behavioral
features i.e. eyes and mouth will be extracted from an image).
Step IV. Training the model is divided into two phases training and testing. The proposed model will be
trained.
Step V. State of the driver (active or sleepy) will be detected using a trained model.
Step VI. Validating the result based on various parameters will be detected.

IV. EXPERIMENTAL SETUP AND IMPLEMENTATION

The proposed methodology will extract the images from the videos, detecting the face region and
subsequent determination of the eye and mouth region. Extracting the features and classification of the
dataset will help in determining whether the driver is active or sleepy.

VOLUME 14, ISSUE 6, 2020 3183 http://xadzkjdx.cn/

Journal of Xidian University https://doi.org/10.37896/jxu14.6/369 ISSN No:1001-2400

4.1 Experimental Setup :

4.1.1) Dataset Collection: The dataset for the research work is NTHU Drowsy Driver Detection (NTHU-
DDD) Video Dataset[11]. The dataset consists of both male and female drivers from five different
scenarios including BareFace, Glasses, Sunglasses, Night-BareFace and Night-Glasses. The videos have
been captured during the day time and night time. The videos are in 640*480 pixels, 30 frames per second
AVI format. From the videos, we have extracted the images and created an image dataset of 250 images
including only four scenarios i.e . Bare Face, Glasses, Night-Bare Face and Night-Glasses. Each image has
been fixed to 512*512 size with 96 dpi resolution.

4.1.2) Designing of Algorithm: Below is the algorithm of an enhanced technique for driver drowsiness
detection:

Algorithm : Drowsiness level (Active or Sleepy)

Input – Image dataset

Output- Drowsiness level (Active or Sleepy)
1. Extracting images from the video
1.1 Selection of the frame
1.2 Apply Viola-Jones (VJ) method for face detection
2. Extracting features- Eyes and Mouth (Region Of Interest) from detected face using VJ
2.1 . For feature selection AdaBoost classifier will be used
2.2. For the feature classification cascade of classifiers will be used
2.3 Sobel Edge Algorithm with a threshold value will be used to detect the edge of the eyes and mouth
2.4 Further, Kalman filter will be used for the eyelid tracking and Lip segmentation for the yawning
parameter
3. Training the model dataset
4. Design CNN (Convolutional Neural Network ) for classification
5. Apply Classification as state of the driver (active or sleepy) will be detected using trained model

4.1.3) Preparation of Script: Script written has been implemented in the MATLAB software. Image
Processing , Computer Vision Toolbox and Deep learning toolboxes have been used in script.

4.2. Implementation:
4.2.1) Image pre-processing – It is the most important step to improve the quality of the image. To remove
the noise (eyebrows) , image filters have been used . Image resizing (512*512) and normalization is also
included.

The images of the dataset from four different scenarios are as below:
Table I: Sample of the images taken in the dataset

BareFace

Glasses

VOLUME 14, ISSUE 6, 2020 3184 http://xadzkjdx.cn/

Journal of Xidian University https://doi.org/10.37896/jxu14.6/369 ISSN No:1001-2400

Night-
BareFace

NightGlas
ses

4.2.2) Image Segmentation: K-Mean clustering : It is one of the most prominent unsupervised machine
learning techniques. It aims to group similar data together. The K-means algorithm identifies the number
of centroids, and then allocates every data point to the nearest cluster. The centroid can be defined as the
centre point of the cluster. It is used for the classification and regression .A cluster is therefore a
collection of the similar data nearest to the centroid.

4.2.3) Face detection and Feature Extraction - Viola-Jones algorithm[9] using Matlab cascade object
detector has been used to detect the objects such as eye, mouth and nose[9]. Computer vision Toolbox
supports several approaches for object detection. It has the following major step:

 It uses Haar like features computed using integral images for eyes and mouth detection
 Feature selection using the AdaBoost algorithm (Adaptive Boost)
 Feature Classification using a cascade of classifiers.
4.2.4) Eye Detection: After detecting the face and applying Viola-Jones algorithm for the feature
extraction, Sobel Edge detection algorithm has been implemented for the eye region as it creates a map of
the face and can be used for feature extraction. Sobel operators are used for implementation and the
gradient value at each pixel can be determined. Kalman filter can be used for determining the eyelid
movement and the state of fatigue can be determined. If the eyes are closed the driver will be sleepy and
otherwise active.

4.2.5) Mouth Detection: Mouth detection plays an important role as yawning is another behavioural
parameter that can be included and tracked along with the eyes to further enhance the drowsiness detection
level. Isobel edge detection algorithm has been implemented to highlight the edge of the mouth which is
the lower part of the face. Lip Segmentation and contouring method has been further implemented to
detect the yawning parameter. If the lips are wide apart, the driver is yawning and is feeling sleepy.

4.2.6) Classification using CNN – Convolutional Neural Network has been used for the training and
validation purpose. The algorithm works on the input fed, it assigns it various aspects and then classifies it
under different categories. The major application areas are object detection, face recognition etc. For
creating a simple deep learning algorithm for classification purposes, the first layer is the input layer
which takes the input as an image. The Convolutional Layer is used to generate the feature map and the
result is fed to the pooling layer which further is used for dimension reduction. Rectified Linear Unit
function is an activation function used for solving the training problems. The flattened matrix finally goes
to the fully connected layers for classification of the images. For training the classifiers, the dataset is
divided into sets of training and testing datasets. To get the best classification rates, 60% of the dataset is
for the training and 40 % of the dataset is for the testing. After classification, the evaluation of the
classifier is carried out. Testing images are passed through the classifier and the results are predicted.

VOLUME 14, ISSUE 6, 2020 3185 http://xadzkjdx.cn/

Journal of Xidian University https://doi.org/10.37896/jxu14.6/369 ISSN No:1001-2400

V. RESULTS:

In order to evaluate the proposed technique, a data set of 250 images has been extracted from the NHTU-
DDD video dataset with four different scenarios. Our methodology combines two behavioural features i.e.
eye detection and yawning parameter and an accuracy of 98.4% has been achieved. The list of the
calculated parameters is shown in Table II below:

Table II. Calculated Parameters

Parameters Value
Accuracy 98.4%
Recall 97.8%
Error Rate 1.6%
Precision 97.8%
F-score 97.8%

VI. CONCLUSION AND FUTURE SCOPE

Driver’s drowsiness is one of the main factors that can prove to be fatal at times. Counter measures need to
be taken to make him/her alert. Drowsiness techniques can be broadly classified in three categories i.e.
behavioural, vehicular and physiological . Drivers play a very important role as the main objective to reach
the destination. Many taxi drivers, truck drivers etc. have to travel to far off places which could lead to
excursion. Excursion in turn can further lead to tiredness, laziness, sleepiness etc. Therefore, lack of sleep
can prove to be fatal while driving. Thus, there is an urge to consider the parameters and look for the new
methods to prevent such fatal accidents. Features such as driver’s eyes curvature, blinking of eyes,
yawning, tilting of head are essential behavioural parameters that can effectively determine the state of the
driver.

A hybrid of these parameters (eye state and yawning detection) has helped in improvisation of the results.
Face detection has been implemented using the Viola-Jones algorithm and the eye , mouth detection
workflow has been listed above. CNN classifier has been used for the classification purpose .The trained
model has achieved the accuracy of 98.4% . Future work can be done to improve the accuracy. However, a
complex environment where the driver is driving with sunglasses during the day time and yawning while
mouth covered with the hand pose a real challenge to determine the state. A large dataset can also be
considered for the same.
REFERENCES

[1] Ashish Kumar and Rusha Patra.2018."Driver Drowsiness Monitoring System using Visual Behaviour and Machine Learning",IEEE, pp:339-344.
[2] Muhammad Tayab Khan , Hafeez Anwar et al. "Smart Real-Time Video Surveillance Platform for Drowsiness Detection Based on Eyelid Closure",
Wireless Communications and Mobile Computing, Vol: 2019, pp: 1-9, Article ID 2036818, https://doi.org/10.1155/2019/2036818.
[3] Jang Woon,Byung-Gil Han et al. 2018,"Real-time Drowsiness Detection Algorithm for Driver State Monitoring Systems", ICUFN 2018, IEEE, pp-73-
75.
[4] Omar Wathiq and Bhavna D. Ambudkar, "Optimized Driver Safety through Driver Fatigue Detection Methods", International Conference on Trends in
Electronics and Informatics ICEI 2017,IEEE, pp:68-73.
[5] Jun-Juh Yan, Hang-Hong Kuo et al., , 2016."Real-time Driver Drowsiness Detection System Based on PERCLOS and Grayscale Image
Processing",2016 International Symposium on Computer, Consumer and Control,IEEE, pp:243-246, DOI 10.1109/IS3C.2016.72.
[6] Bappaditya Mandal,Liyuan Li et. al."Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State", IEEE TRANSACTIONS
ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 18, , NO.3.pp:545-557.
[7] Rateb Jabbar et al. "Real-time Driver Drowsiness Detection for Android Application Using Deep Neural Networks Techniques", The 9th International
Conference on Ambient Systems, Networks, and Technologies, Procedia Computer Science 130 Vol. 2018, pp- 400–407.
[8] Marchel T. Tombeng, Hence Kandow et al.,"Android-Based Application To Detect Drowsiness When Driving Vehicle",2019 1st International
Conference on Cybernetics and Intelligent System (ICORIS) 22nd-23rd August 2019 at Institut Teknologi dan Bisnis (ITB) STIKOM Bali, Indonesia,
pp:102-104.
[9] Maninder Kahlon and Subramaniam Ganesan, “Driver Drowsiness Detection System Based on Binary Eyes Image Data”, IEEE, pp: 0209-0215.
[10] Menchie Miranda, Alonica Villanueva et al.," PORTABLE PREVENTION AND MONITORING OF DRIVER’S DROWSINESS FOCUSES TO
EYELID MOVEMENT USING INTERNET OF THINGS",IEEE, 2018.
[11] Ching-Hua Weng,Ying-Hsiu Lai, Shang-Hong Lai, “Driver Drowsiness Detection via a Hierarchical Temporal Deep Belief Network”, In Asian
Conference on Computer Workshop on Driver Drowsiness Detection from Video,Taipei,Taiwan,Nov. 2016.

VOLUME 14, ISSUE 6, 2020 3186 http://xadzkjdx.cn/

View publication stats