2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)

Real-Time Eye State Detection System for Driver

Drowsiness Using Convolutional Neural Network
1st Wissarut Kongcharoen
Department of Information
TechnologyKing Mongkut's University
of Technology North Bangkok
Bangkok, Thailand
firstwantup@gmail.com
2nd Siranee Nuchitprasitchai
Department of Information
TechnologyKing Mongkut's University
of Technology North Bangkok
Bangkok, Thailand
siranee.n@it.kmutnb.ac.th
3rd Yuenyong Nilsiam
Department of Information
TechnologyKing Mongkut's University
of Technology North Bangkok
Bangkok, Thailand
yuenyong.n@eng.kmutnb.ac.th

4 4th Joshua M. Pearce

Department of Materials Science &
Engineering and Department of
Electrical & Computer Engineering
Michigan Technological University
Houghton, MI, USA
pearce@mtu.edu

Abstract— One of the top reasons for road accidents that
result in injuries and deaths is the dozing off of drivers. In this
study eye tracking using a novel open source Internet of Things
(IoT)-based system has been developed. This study evaluated
three driver’s eye recognition algorithms to be integrated into
the open source solution to wake drivers as they begin to dose
off: 1) Convolutional Neural Network (CNN) with Haar
Cascade, 2) 68 facial landmark points and 3) gaze detection in
three different face positions for both day and night driving
conditions as well as with and without glasses. Each combination
of those factor is tested 100 times. The best algorithm is chosen
based on the numbers of correct detections and then this
algorithm is tested again based on light (day and night), angle of
face (left, right, and center), angle of camera (left and right), and
glasses (on and off) to detect both blinking and closed eyes. The
results show that the most accurate algorithm to detect a
driver’s eyes is CNN with Haar Cascade algorithm with 94%
accurate. The system can detect the status of the eyes of drivers
during driving and if drivers close their eyes longer than two
seconds, it sounds an alert to wake the driver and avoid the
accident. The proposed open source system costs about US$100
and could be widely deployed to help reduce accidents on the
road throughout the world.
Keywords— Drowsiness Detection Algorithm, Eye Blinking,
Convolutional Neural Network, Haar Cascade, Automobile
Safety, Traffic Fatalities.
I. INTRODUCTION
Thailand’s roads are among the most dangerous in the
world and the government vowed at a United Nations
(UN) forum in 2015 to halve the number of road traffic
deaths by 2020 [1,2]. The World Health Organization
(WHO) reports that Thailand is far from achieving its
2020 goal of bringing road deaths down to less than 20
for every 100,000 people [3]. Traffic related fatalities
has been a concern for a number of years and significant
analysis to find interventions has been undertaken [4-8].
Unfortunately, today road accidents still account for
most deaths in Thailand (particularly among the poor),
with more than 20,000 preventable fatalities each year
[9]. Last year (2019) there were 20,169 deaths [10]. An
accident study conducting by the Department of
Transportation of Bangkok and Provincial Department of
Transportation offices around the country during the month of
February 2019 shows that the accidents are often caused by
the behaviors of the drivers (47 times) [11]. One of the top
reasons that causes accidents, which result in injuries and
deaths, is the dozing off of the drivers (10 out of 47 times)
[11]. The symptoms of drowsiness during driving are: 1)
yawning, 2) distraction, 3) tired and worry, 4) cannot
remember what have passed in a few kilometers, 5) cannot
open their eyes, 6) dizzy or giddy, 7) swerving between lanes
and 8) skip traffic signs [12].
In order to overcome these challenges this study evaluates
techniques and systems to prevent drowsiness during driving.
Systems have been developed to monitor drowsiness or
dozing off while on the road including ways to monitor the
driver for drowsiness including electroencephalogram (EEG),
electrocardiography (ECG), and eye status tracking [13]. In
this study eye tracking was selected as the best option because
no equipment needs to be attached to the driver. In this study
a novel open source Internet of Things (IoT)-based system has
been developed and is disclosed and evaluated here. This
system will detect the status of the eyes of the drivers whether
they are open or closed and the status of the eyes is used to
indicate if the driver is drowsy or distracted then alarm the
driver with a sound. First the face detection and face
recognition literature is briefly reviewed in the context of the
IoT. Next several of the most popular algorithms are tested
with different controlled factors which include light (day and
night), and angle of face (left, right, and center) to detect both
blinking and closed eyes. Each combination of those factor is
tested 100 times. The best algorithm is chosen based on the
numbers of correct detections and then this algorithm is tested
again based on light (day and night), angle of face (left, right,
and center), angle of camera (left and right), and glasses (on
and off) to detect both blinking and closed eyes. Next, open
source software is developed to prevent drowsiness driving
accidents, which can use three algorithms. This study
evaluated three driver’s eye recognition algorithms: 1)
Convolutional Neural Network (CNN) with Haar Cascade, 2)
68 facial landmark points and 3) gaze detection in three
different face positions for both day and night driving
conditions as well as with and without glasses. The results of

978-1-7281-6486-1/20/$31.00 ©2020 IEEE 551

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 09,2020 at 14:48:31 UTC from IEEE Xplore. Restrictions apply.
 2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)
this study are discussed in the context of the proposed system
being able to help reduce accidents on the road in Thailand
and other regions.
II. LITERATURE REVIEW
Face detection and face recognition [13] have been used
for many applications, such as, for security purpose. Face
detection is a process of finding a face of human in images or
video footage and face recognition is a continued process
after the detection to compare the detected faces with the
faces in the database to identify the person. To extract the
features of the face is a crucial part of face recognition and
face analysis. There are many techniques have been studied:
Convolutional Neural Network (CNN) [14], Facial Landmark
Detection [15], Viola-Jones Method [16, 17] and others.
Regardless of the method, it is important that any device
follows Internet of Things and Thailand 4.0 policy [18]. the
National Broadcasting and Telecommunications Commission
(NBTC) has been working with many parties to bring all
different standards of communication between devices in
order to make them able to communicate with each other
[18]. There have also been many attempts to use
these techniques to detect drowsiness. First, Vivekanandan
et al. [19] proposed the machine learning model for
drowsiness detection. Even though the results of detection
were accurate, the computation was done on a personal
computer running Windows operating system, which is not
practical for use in the application of this study. Next,
Danisman et al. [20] tried to detect drowsy driver using
eye blinking patterns. The accuracy of the system is 94%,
which is very high. However, with high accuracy, the system
used a video footage to analyze the patterns that come with
high resources consuming and sensitive to the external
light of the system. Irtua et al. [21] built the system to
detect fatigue of driver using facial landmarks. The
system also has the problem of high resources consuming
due to the use of image processing technique.
Next, Tiwari et al. [22] implemented IoT based driver
drowsiness detection and health monitoring system. The
system analyzes the status of the eyes of the drivers and also
monitors heart rate, temperature, gas of alcohol in the car,
and location of the car. All information is sent to the cloud.
This is particularly interesting as it could also be used to
reduce drunk driving. Finally, Rawal [23], Joshi et al. [24],
Rajneesh et al. [25], Sooksatra et al. [26], Said et al. [27], and
Jayswal et al. [28] have developed systems to detect
drowsiness of drivers and alert system to prevent accident
from happening. These prior studies are based on Haar
Cascade algorithm, but this research proposes and tests both
the Convolutional Neural Network (CNN) with Haar
Cascade, which have the potential to both reduce the
processing time and improve the accuracy.
III. METHODOLOGY
Three approaches are taken to monitor drowsiness to
prevent driving accidents by using eye recognition algorithms:
1) Convolutional Neural Network (CNN) with Haar Cascade,
2) 68 facial landmark points and 3) gaze detection.
A. Convolutional Neural Network (CNN)
Convolutional Neural Network (CNN) as shown in Figure
1 is a kind of neural network to help analyze images. In this
research, it was used to classify images of drives into the status
of the eyes of drivers: open or close.
552
Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 09,2020 at 14:48:31 UTC from IEEE Xplore. Restrictions apply.
Fig. 1. Convolutional Neural Network
A convolutional layer example, the input neuron is a
5x5x3 array of pixel values. A filter or a neuron or a kernel
and this filter covers a 3x3x3 array of weights or parameters.
The depth of this filter has to be the same as the depth of input.
The first filter position starts from the top left corner. As the
filter is sliding or convolving to the right by 1 pixel/stride (or
more than 1 pixel/stride) at a time around the input image,
element wise multiplication is applied by multiplying the
values in the filter with the original pixel values of the image.
These multiplications are all summed up in a single number
called the receptive field in first hidden layer or feature map
with 3x3x1 array. The filter can be used more than 1 filter in
one convolution layer. If there are 16 filters, the feature map
of size is 3x3x16. The feature map of size is smaller than the
input of size. To pad zeros to the boundary of the feature map
is increasing the size. Rectified Linear Unit (ReLU) is Linear
function, which is rectified from S shape. For ReLU, if the
input is positive, then the slope is always equal one as shown
in Equation 1 and 2. A pooling, the max pooling layer is the
most common used after the convolutional layer to reduce the
feature map of size only in width and height, not in depth.
0, 𝑥 ≤ 0
𝑓(𝑥) = max(0, 𝑥) = (1)
𝑥, 𝑥 > 0
0, 𝑥 ≤ 0
𝑓′(𝑥) = (2)
1, 𝑥 > 0
The model was developed using 104,559 pictures of an eye
from Media Research Lab [29]. 52,727 open eye and 48,903
close eye from Media Research Lab. 10,355 open eye and
8,572 close eye were taken by the authors. The ratio of training
data to test data is 80 to 20.
B. 68 Facial landmarks points
68 Facial landmarks points are used to estimate the
location of 68 (x,y)-coordinates that represent eyes,
eyebrows, nose, mouth, and jawline on the human face. To
detect an eye blinking in an image, a face needs to be
localized using Histogram of Oriented Gradients (HOG) and
Linear Support Vector Machine (SVM) object detectors [30].
Histogram of Oriented Gradients (HOG) [31] is a feature
descriptor with histograms and oriented gradients
information of an image. The magnitude and direction of
gradient are used to create a histogram. The histogram of
gradients is divided into nine bins between 0 and 180 degrees.
The gradient’s direction shows which bin is selected and the
gradient magnitude is the amount of number added into the
bin. Linear Support Vector Machine (SVM) [32] is one
technique to analyze data for classification in supervised
learning model. Linear SVM is categorizing data into two
classes.
C. Eye gaze detection
Eye gaze detection (EGD) is to locate the eye positions on
human face and to analyze the eye blinking. The right and left
eye coordinates correspond to the 68 Facial landmarks points
 2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)

with indexes: 37 to 42 and 43 to 48 respectively. These

featured points are used to detect the edges of eye lids and then
calculate the distance between the top and bottom edges of
each eyes (left and right). These distances are used to
determine if a driver’s eyes are blinking or closed.
D. Experiments
Using the approaches discussed in section A-C an open
source algorithm was coded in Python (3.5.6) The full code is
available at: https://github.com/wissarutkong/cnn_eyedete
ction.git. The hardware used to conduct the experiments was
shown in Table 1. The experiments are run under two
Fig. 3. Examples of results of 68 facial landmark points algorithm.
environments in day time and night time with three
algorithms: 1) Convolutional Neural Network (CNN) with
Haar Cascade, 2) 68 facial landmark points and 3) gaze
detection. Each algorithm is tested with three different face
positions: 1) the face is in the central straight (C), 2) the face
turns to the left side for 45 degree (L), and 3) the face turns to
the right side for 45 degree (R). Turning 45 degree is used to
imitate when drivers look at side mirrors. Finally, the
experiments were repeated with and without glasses. Each
experiment is repeated 100 times.

TABLE I. BILL OF MATERIALS. ALL MATERIALS WERE SOURCED

FROM ARDUITRONICS.

Item Number Cost(Baht) Fig. 4. Examples of results of eye gaze detection algorithm.
Raspberry Pi 4 Model B 1 1,980
Raspberry Pi NoIR Camera Module Infra- 1 1,150
red Sensitive Camera V2
Infrared Night Vision Unit (LED 1 Watt) 1 89.50
Active Buzzer module (3.3 – 5 V.) 1 35
Jumper Female to Female (40 x 10zm) 1 45
Total 3,299.5

Fig. 5. Real-Time Eye State Detection (RTESD) System Installation.

IV. RESULTS TABLE II. THE EYE BLINKING DETECTOIN ALGORITHMS COMPARING

Some examples of the outputs of the system are shown

below, Convolutional Neural Network (CNN) with Haar
Cascade (See Fig. 2), 68 facial landmark points (see Fig. 3),
and eye gaze detection (see Fig. 4). The 68 facial landmark
points and eye gaze detection are able to detect status of the
eyes (open or close). However, the efficiency of the CNN with
Haar Cascade algorithm is significantly better than the other
two algorithms. The number of correct detections of blinking
and closed eyes out of 100 times. The eye blinking detection TABLE III. ALGORITHM COMPARING FOR SYSTEM LOCATION
results are show in Table 2. Table 3 shows that the best
position to detect the status of the eyes is the face in the central
straight for almost all cases, with glasses, without glasses,
camera installed on the left, and camera installed on the right
(see Fig.5).

Based on the results in this section, a system architecture

is proposed (see Fig. 6) using a Raspberry Pi 4 and 4G/LTE
Shield, which sends data to a Linux server running Python 3.
The buzzer module for the raspberry pi is used to alert the
driver and wake the driver. The cost of the system is about
3,299.50 baht (around $105 USD). This makes it accessible to
both after-market installations by third party vendors as well
as automobile manufacturers for whom buying the
components in bulk would reduce the costs well under $100
USD. Python is popular programming language because it is
Fig. 2. Examples of results of CNN with Haar Cascade algorithm.

553

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 09,2020 at 14:48:31 UTC from IEEE Xplore. Restrictions apply.
 2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)

easy to learn and is an applicable programming language for [10] Permpoonwiwat, C. K., & Kotrajaras, P. (2012). Pooled time-series
everyone and a wide collections of industries. A large number analysis on traffic fatalities in Thailand. World Review of Business
Research, 2(6), 170-182.
of people around the world has contributed their time and
[11] Statistic of public transportation and truck,
effort to develop new libraries in most fields. It is an open https://www.dlt.go.th/site/ltsb/m-news/3937/ view.php?_did=20224,
source programming language, so everyone can use and lass accessed 2019/09/05.
improve the existing libraries as needed. Python will continue [12] How to deal with drowsy during driving,
to grow and the quality of the libraries will be improved as it https://www.thaihealth.or.th/Content/40214-วิธีปฏิ บตั ิหาก "
is used by additional developers. Overall the results of this ง่วง" ขณะขับรถ.html, lass accessed 2019/09/14.
study were extremely promising. More advanced cameras [13] Fazli, S., & Esfehani, P. (2012, March). Tracking eye state for fatigue
may help solve some of the challenges uncovered if they have detection. In International Conference on Advances in Computer and
Electrical Engineering (ICACEE 2012) (pp. 17-20).
better resolutions and are able to handle low light situations.
For example, Arducam 16MP Pi Camera 4K, IMX298 costs [14] Convolutional Neural Netowrk (CNN), https://blog.datawow.io/มาลองดู
วิธีคิดของ-cnn-กัน-e3f5d73eebaa, last accessed 2019/12/15.
about 1,883.69 baht ($59.99 USD) [33] and could be used in
[15] Facial Landmarks with dlib, OpenCV, and Python,
future work. https://www.pyimagesearch.com/ 2017/04/03/facial-landmarks-dlib-
opencv-python/, last accessed 2019/09/14.
[16] Face Detection, https://pongpich-v.blogspot.com/2018/11/blog-
post.html, last accessed 2019/09/14.
[17] Viewer Tally from Visual and Depth Image, https://isl2-
dev.cp.eng.chula.ac.th/sites
/default/files/Senior%20Project%20Proposal.docx, last accessed
2019/09/14.
[18] Internet of Things and Thailand 4.0 Policy,
Fig. 6. Proposed Real-Time Eye State Detection (RTESD) Architecture. http://www.nbtc.go.th/getattachment/Services /quarter2560/ปี -
2561/32279/เอกสารแนบ.pdf.aspx, last accessed 2019/09/05.
V. RESULTS [19] Ranjith, V. et al.: Machine Learning Model for Drowsiness Detection.
2019 International Journal of Research in Engineering and
This paper has shown that the proposed system, the Real- management 1(1), 334-338 (2019).
Time Eye State Detection System Using Convolutional [20] Danisman, T., Bilasco, I. M., Djeraba, C., & Ihaddadene, N.: Drowsy
Neural Network with Driver Drowsiness Alert, was driver detection system using eye blink patterns. In: 2010 International
implemented successfully. The system can detect the status of Conference on Machine and Web Intelligence, pp. 230-233, IEEE
the eyes of drivers during driving. When the drivers close their (2010).
eyes longer than two seconds, the alert system would notify [21] IRTIJA, N., SAMI, M., & AHAD, M. A. R.: Fatigue Detection Using
Facial Landmarks. In: International Symposium on Affective Science
the drivers by the sound. There are some false alarms and and Engineering ISASE2018, pp. 1-6, Japan Society of Kansei
misses due to the environment around the drivers, such as, Engineering (2018).
glasses and external light. For future work, these issues should [22] Tiwari et al.: IOT Based Driver Drowsiness Detection and Health
be taken into account and resolved. Monitoring System. International Journal of Research and Analytical
Review 6(2), 163i-167i (2019).
REFERENCES [23] Rawal, R. S., & Nagtilak, S. S.; Drowsiness Detection Using
[1] Beech, H. Why are Thailand’s roads among the deadliest in the world? RASPBERRY-Pi Model Based On Image Processing (2016).
(2019, September 3). The Independent. [24] Joshi, A., Gujrati, S., & Bhati, A. (2013). Eye state and head position
https://www.independent.co.uk/news/long_reads/thailand-roads- technique for driver drowsiness detection. International Journal of
deadly-traffic-accidents-class-inequality-a9071696.html Electronics and Computer Science Engineering, 2(3), 874-879.
[2] Initiative to improve Road safety in Thailand. (2015). [25] Goraya, A., & Singh, G. Real Time Drivers Drowsiness Detection and
http://www.uncrd.or.jp/content/documents/3489Thailand%20Initiativ alert System by Measuring EAR. International Journal of Computer
e%20-.pdf Applications, 975, 8887.
[3] Road accidents still account for most deaths in Thailand, report shows. [26] Sooksatra, S., Kondo, T., & Bunnun, P. (2015, June). A robust method
(2019). Https://Www.Nationthailand.Com. Retrieved January 29, for drowsiness detection using distance and gradient vectors. In 2015
2020, from https://www.nationthailand.com/news/30373606 12th International Conference on Electrical Engineering/Electronics,
[4] Tanaboriboon, Y., & Satiennam, T. (2005). Traffic accidents in Computer, Telecommunications and Information Technology (ECTI-
Thailand. IATSS research, 29(1), 88-100. CON) (pp. 1-5). IEEE.
[5] Nakahara, S., Chadbunchachai, W., Ichikawa, M., Tipsuntornsak, N., [27] Said, S., AlKork, S., Beyrouthy, T., Hassan, M., Abdellatif, O. E., &
& Wakai, S. (2005). Temporal distribution of motorcyclist injuries and Abdraboo, M. F. Real Time Eye Tracking and Detection-A Driving
risk of fatalities in relation to age, helmet use, and riding while Assistance System.
intoxicated in Khon Kaen, Thailand. Accident Analysis & Prevention, [28] Jayswal, A. S., & Modi, R. V. (2017). Face and eye detection
37(5), 833-842. techniques for driver drowsiness detection. Int. Research. J.
[6] Kumphong, J., Satiennam, T., & Satiennam, W. (2016). A correlation Engineering and Technology (IRJET), 4(04).
of traffic accident fatalities, speed enforcement and the gross national [29] MRL Eye Dataset, http://mrl.cs.vsb.cz/eyedataset, last accessed
income of Thailand and its cross-border countries. International 2019/09/14.
Journal of Technology, 7(7), 1141-1146. [30] Facial landmarks with dlib, OpenCV, and Python,
[7] Permpoonwiwat, C. K., & Kotrajaras, P. (2012). Pooled time-series https://www.pyimagesearch.com/ 2017/04/03/facial-landmarks-dlib-
analysis on traffic fatalities in Thailand. World Review of Business opencv-python/, last accessed 2019/12/15.
Research, 2(6), 170-182. [31] Histogram of Oriented Gradients,
[8] Suriyawongpaisal, P., & Kanchanasut, S. (2003). Road traffic injuries https://www.learnopencv.com/histogram-of-oriented-gradients/, last
in Thailand: trends, selected underlying determinants and status of accessed 2019/12/15.
intervention. Injury control and safety promotion, 10(1-2), 95-104. [32] Cortes, C., & Vapnik, V. (1995). N, Support Vector Networks.
[9] Thaiger, T., & Nation, T. (2019, July 25). Road incidents still the Machine Learning, 20(3), 273-295.
biggest killer in Thailand. The Thaiger. https://thethaiger.com/hot-
news/road-deaths/road-incidents-still-the-biggest-killer-in-thailand

554

Authorized licensed use limited to: Auckland University of Technology. Downloaded on August 09,2020 at 14:48:31 UTC from IEEE Xplore. Restrictions apply.