DROWSY DRIVER DETECTION SYTEM

INTERIM PROJECT REPORT

Submitted by
HABEEL LATHEEF - MES18CS034
MOHAMMED AMEEN PATHAYIL - MES18CS050
ASIF RASHEED - MES18CS018
MOHAMMED SAABITH C.P. - MES18CS055
to

the APJ Abdul Kalam Technological University

in partial fulfilment of the requirements for the award of the Degree

of

Bachelor of Technology

in

Computer Science and Engineering

Department of Computer Science and Engineering

[B. Tech Programme accredited by NBA]
MES College of Engineering Kuttipuram
Thrikkanapuram P.O., Malappuram Dt, Kerala, India 679582

2021-2022
 DECLARATION

We, hereby declare that the project report “Drowsy Driver Detection System”,

submitted for partial fulfilment of the requirements for the award of degree of Bachelor

of Technology of the APJ Abdul Kalam Technological University, Kerala is bonafide

work done under the supervision of MS. Farzana T., Assistant Professor, Computer

Science and Engineering. This submission represents my ideas in my own words and

where ideas or words of others have been included, we have adequately and accurately

cited and referenced the original sources. We also declare that we have adhered to ethics

of academic honesty and integrity and have not misrepresented or fabricated any data

or idea or fact or source in my submission. We understand that any violation of the

above will be a cause for disciplinary action by the institute and/or the University and

can also evoke penal action from the sources which have thus not been properly cited

or from whom proper permission has not been obtained. This report has not been

previously formed the basis for the award of any degree, diploma or similar title of any

other University.

Place: Kuttipuram Habeel Latheef

Mohammed Ameen Pathayil
Date: 05-03-2022 Asif Rasheed
Mohammed Saabith C. P.
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

MES COLLEGE OF ENGINEERING, KUTTIPURAM

CERTIFICATE

This is to certify that the report entitled “Drowsy Driver Detection System”
submitted by Habeel Latheef, Asif Rasheed, Mohammed Ameen Pathayil,
Mohammed Saabith C. P., to the APJ Abdul Kalam University in partial requirements
for the award of the Degree of Bachelor of Technology in Computer Science and
Engineering is a bonafide record of the project work carried out under my guidance and
supervision. This report in any form has not been submitted to any other University or
Institute for any purpose.

Project Guide
Ms. Farzana T. Dr. Anil K. Jacob
Assistant Professor Professor and Head
Dept. of Computer Science and Engg Dept. of Computer Science and Engg
MES College of Engineering MES College of Engineering
 ACKNOWLEDGEMENT

First of all we sincerely thank the Almighty GOD who is most beneficent

and merciful for giving our knowledge and courage to complete the project successfully.

We derive immense pleasure in expressing sincere thanks to our Principal K. A.

Navas for the kind co-operation in all aspects of our project. We express our gratitude to Dr.

Anil K. Jacob, HOD, Department of Computer Science and Engineering for his kind co-

operation in all aspects of our project. We express our sincere thanks to Ms. Farzana T.,

Assistant Professor, our internal guide and our project co-ordinators, Mr. Aneesh M. Haneef,

Ms. Annes Philip for their encouragement and motivation during the project.

We also express our gratitude to all the teaching and non-teaching staff of the college

especially to our department for their encouragement and help during our project. Finally, we

appreciate the patience and solid support of our parents and enthusiastic friends for their

encouragement and moral support for this effort.

HABEEL LATHEEF
MOHAMMED AMEEN PATHAYIL
ASIF RASHEED
MOHAMMED SAABITH C. P.
 ABSTRACT

Drowsiness and Fatigue of drivers are amongst the significant causes of road accidents.

Every year, they increase the amounts of deaths and fatalities injuries globally. Drivers who do

not take regular breaks when driving long distances run a high risk of becoming drowsy, a state

which they often fail to recognize early enough according to the experts. Studies show that

around one quarter of all serious motorway accidents are attributable to sleepy drivers in need

of a rest, meaning that drowsiness causes more road accidents than drink-driving. Researchers

have attempted to determine driver drowsiness using the following measures: (1) vehicle-based

measures; (2) behavioural measures and (3) physiological measures. The face, an important

part of the body, conveys a lot of information. When a driver is in a state of fatigue, the facial

expressions, e.g., the frequency of blinking, are different from those in the normal state. The

proposed system detects the drivers’ fatigue status, such as blinking, and duration of eye

closure, using video images, without equipping their bodies with devices.
 CONTENTS

CHAPTER-1 INTRODUCTION .......................................................................... 1

CHAPTER 2 LITERATUE REVIEW .................................................................. 3

2.1 Camera Based Drowsiness Reference for Driver State Classification

Under Real Driving Conditions ............................................................. 3

2.2 Driver Drowsiness Recognition Based on Computer Vision Technology ... 3

2.3 Visual Analysis of Eye State and Head Pose For Driver Alertness
Monitoring ............................................................. 4

CHAPTER 3 THE PROPOSED SYSTEM .......................................................... 5

3.1 Image Processing ............................................................. 5

3.2 Haar Cascade Classifier ............................................................. 7

3.3 Eye Aspect Ratio (EAR) ............................................................. 9

CHAPTER 4 SYSTEM REQUIREMENTS....................................................... 11

5.1 Hardware Requirements ........................................................... 11

5.2 Software Requirements ........................................................... 12

CHAPTER 5 CONCLUSION............................................................................. 13

CHAPTER 6 REFERENCES ............................................................................. 14

 LIST OF FIGURES

Figure 1. Methods involving face using Haar Cascade classifier and Eye Aspect Ratio .......... 5

Figure 2. Overall face and eye including EAR detection flowchart .......................................... 6

Figure 3. Edge feature, Line feature and Four-Triangle feature ................................................ 7

Figure 4. Input image and Integral image .................................................................................. 8

Figure 5. Description of computing a sum in the summed-area table data................................ 9

Figure 6. Open and Close eye with landmarks ........................................................................ 10

Figure 7. Raspberry Pi 3 Model B ........................................................................................... 11

Figure 8. Raspberry Pi 8 Mega Pixel camera sensor ............................................................... 12

 CHAPTER-1
INTRODUCTION

Currently, transport systems are an essential part of human activities. We all can
be victim of drowsiness while driving, simply after a night’s sleep too short, altered
physical condition or during long journeys. The sensation of sleep reduces the driver’s
level of vigilance producing dangerous situations and increases the probability of an
occurrence of accidents. Driver drowsiness fatigue are among the important causes of
road accidents. Every year, they increase the number of deaths and fatalities injures
globally. In this context, it is important to use new technologies to design and build
systems that are able to monitor drivers and to measure their level of attention during
the entire process of driving.

Drowsiness detection is studied by monitoring vehicle-based measurements,

behavioural and physiological.
(1) Vehicle-based measures – A number of metrics, including deviations from
lane position, movement of steering wheel, pressure on the acceleration pedal, etc., are
constantly monitored and any change in these that crosses a specified threshold
indicates a significantly increased probability that the driver is drowsy.
(2) Behavioural measures – The behaviour of the driver, including yawning, eye
closure, eye blinking, head pose, etc., is monitored through a camera and the driver is
alerted if any of these drowsiness symptoms are detected.
(3) Physiological measures – The correlation between physiological signals
(electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EoG) and
electroencephalogram (EEG)) and driver drowsiness has been studied by many
researchers.

Subjective measurements based on questionnaires and electrophysiological

measures of sleep can also be made but it is generally difficult to obtain drowsiness
feedback from a driver in a real driving situation. There are reasons for and against
using each measurement, and currently it is not clear which measurement is most
reliable and cost effective. The advantage of computer vision techniques is that they are
non-invasive, and thus are more amenable to use by the general public. For predicting

1
falling-asleep, head nods, blinks, eye closure, and yawning are used using a facial
recognition algorithm. This project is focused on detecting drowsiness by monitoring
facial features and head movements of the driver.

2
 CHAPTER 2
LITERATUE REVIEW

2.1 Camera Based Drowsiness Reference for Driver State

Classification Under Real Driving Conditions [2]
They proposed that measures of the driver’s eyes are capable to detect
drowsiness under simulator or experiment conditions. The performance of the latest eye
tracking based in-vehicle fatigue prediction measures are evaluated. These measures
are assessed statistically and by a classification method based on a large dataset of 90
hours of real read drivers.

The results that eye-tracking drowsiness detection works well for some drivers
as long as the blinks detection works properly. Even with some proposed
improvements, however, there are still problems with bad light conditions for persons
wearing glasses.

2.2 Driver Drowsiness Recognition Based on Computer Vision

Technology [3]
They presented a non-intrusive method for drowsiness recognition with eye-
tracking and image processing. A robust eye detection algorithm is introduced to
address the problems caused by changes in illumination and driver posture.

Six measures are calculated with percentage of eyelid closure, maximum

closure duration, blink frequency, average opening level of the eyes, opening velocity
of the eyes, and closing velocity of the eyes. These measures are combined using
Fisher’s linear discriminated functions using a stepwise method to reduce the
correlations and extract and independent index.

3
2.3 Visual Analysis of Eye State and Head Pose For Driver
Alertness Monitoring [8]
They presented visual analysis of eye state and head pose (HP) for continuous
monitoring of alertness of a vehicle driver. Most existing approaches to visual detection
of non-alert driving patterns rely either on eye closure or head nodding angles to
determine the driver drowsiness or distraction level.

The proposed scheme uses visual features such as eye index (EI), pupil activity
(PA), and HP to extract critical information on non-alertness of a vehicle driver. A
support vector machine (SVM) classifies a sequence of video segments into alert or
non-alert driving events. Experimental results show that the proposed scheme offers
high classification accuracy with acceptably low errors and false alarms for people of
various ethnicity and gender in real road driving conditions

4
 CHAPTER 3
THE PROPOSED SYSTEM

The proposed system is mainly aiming at a novel approach towards designing and
deploying an integrated system that includes a complete set of functionalities to aid in
detecting eye and facial movements

3.1 Image Processing

Face and eye detection is a very important and challenging matter in the field of image
processing. It is also a crucial step of face recognition. Open-Source Computer Vision
Library (Open CV) is used to implement the Haar Cascade Classifier. In this project,
the driver drowsiness detection requires a video sensor to detect the faces of drivers.
Then the drowsiness level of drivers can be determined by checking the eye blinking
rate. Figure 4 illustrates the methods involving the implementation of detection of face
and eye including eye blinks using Haar Cascade Classifier and Eye Aspect Ratio,
respectively. In Haar Cascade Classifier, there are four main steps that need to be
executed. The steps are Haar Feature, Integral image, AdaBoost and Cascade Classifier.
As for Eye Aspect Ratio, it is used to detect the eye blinks (eyes open and close) using
the ratio formula based on the eyes width and height.

Figure 1. Methods involving face and eye as well as eye blinks detection using Haar Cascade
classifier and Eye Aspect Ratio, respectively
On the other hand, Figure 1 shows the overall flowchart of face and eye
including EAR detection when the system is operated. The acquisition of real-time

5
video from the camera is done by using command raspistill. As explained previously,
the Haar Cascade Classifier is used to detect face and eyes images whereas the EAR
formula is being implemented to detect the blinking of eyes. Finally, the buzzer is
implemented in this prototype as a simple mechanism to alert the driver when the EAR
value output is approximate to 0 for two seconds long.

Figure 2. Overall face and eye including EAR detection flowchart

6
3.2 Haar Cascade Classifier
Haar Cascade Classifier is one of the few object detection methods with the ability to
detect faces. It offers high-speed computation depending on the number of pixels inside
the rectangle feature and not depending on each pixel value of the image. This method
has four steps for detecting an object namely as Haar-like feature, integral image,
AdaBoost learning and Cascade Classifier. For the detection of the face, Haar features
are the main part of the Haar Cascade Classifier. The Haar features are used to detect
the presence of feature in given image. Each feature results in a single value, which is
calculated by the sum of pixels under black rectangle. The Haar-like feature is a
rectangular feature providing specific indication to an image for rapid face detection.
Figure 6 shows the examples of common variety of Haar-like features.

Figure 3. (a) Edge feature, (b) Line feature and (c) Four-Triangle feature

In obtaining object detection value, Haar-like feature value was calculated using
integral image. It starts scanning the image for the detection of the face from the top
left corner and ends the face detection process at the right bottom of image in order to
detect the face from an image as shown in Figure 4(a). The integral image could
calculate values accurately and relatively quick by creating new presentation of image
by using value of region previously scanned by specific Haar-like feature as depicted
in Figure 4(b). The value at any point (x, y) is the summed area table of the sum off all
the pixels above and to the left of (x, y), inclusive as shown in Equation (1).

where i(x, y) is the value of pixel at (x, y) whereas I(x, y) is the sum of integral of pixel
values. The value of integral image, I(x, y) is obtained by sum value previous index,

7
starting from the left top until right bottom. Moreover, the summed-area table can be
computed efficiently in a single pass over the image, as the value in the summed-area
table at (x, y) in Equation (2).

Figure 4. (a) Input image and (b) Integral image [26]

Once the summed-area table had been computed, evaluating the sum of intensities over
any rectangular area requires exactly four array references regardless of the area size.
Equation (3) shows the sum of i(x, y) over the rectangle spanned by A, B, C and D.

The notation in the Figure 5 shows the example of the description of computing a sum in the
summed-area table data structure having A = (x0, y0), B = (x1, y0), C = (x0, y1) and D = (x1, y1).

8
 Figure 5. Description of computing a sum in the summed-area table data

Value which has been calculated by using integral image would then be compared with
threshold value of specific features provided by AdaBoost. This should be completed
to find potential features because not all features were relevant to use for specific object
detection. AdaBoost combines potential features called weak classifier to become
strong classifier. The Cascade classifier can be divided into two which are strong
classifier and weak classifier. Weak classifier means less accurate or also irrelevant
prediction and strong classifier means more accurate or relevant prediction. Strong
classifier made by AdaBoost can detect object level by level on a cascade.

3.3 Eye Aspect Ratio (EAR)

After detecting the face of the driver, the calculation of drowsiness level of the driver
is based on eye blink rate. The Eye Aspect Ratio (EAR) formula, which was proposed
is able to detect the eye blink using the scalar value. For instance, if driver blinks eyes
more frequently, it means that the drivers are in the state of drowsiness. Thus, it is
necessary to detect the eyes shape accurately in order to calculate the eye blink
frequency. From the landmarks detected in the image with face, the EAR is used as an
estimate of the eye openness state. For every video frame, the eye landmarks are
detected between height and width of the eye that had been computed. The eye aspect
ratio can be defined by the Equation (4).

Equation (4) shows the eye aspect ratio formula where p1 until p6 are the 2D landmark
locations. The p2, p3, p5 and p6 are used to measure the height whereas p1 and p4 are
used to measure width of the eyes in meter (m) as shown in Figure 6(a). The eye aspect

9
ratio is a constant value when the eye is opened, but rapidly falls approximately to 0
when the eye is closed as shown in the Figure 6(b).

Figure 6. (a) Open and (b) Close eye with landmarks

Equation (5) shows the EAR output range during eyes open and eyes close. During eyes
closed the EAR result will be approximately to 0 whereas during eyes open the EAR
can be any integer number which is x greater than 0.

10
 CHAPTER 4
SYSTEM REQUIREMENTS

5.1 Hardware Requirements

The main apparatuses to develop the prototype-based image processing are the
Raspberry Pi 3 Model B and Raspberry Pi 8 Mega Pixel camera sensor. Figure 7 shows
the Raspberry Pi 3 Model B, which was created by the Raspberry Pi Foundation. The
Raspberry Pi 3 Model B is the third generation of Raspberry Pi. Whilst maintaining the
popular board format, the Raspberry Pi 3 Model B brings more powerful processer that
is 10 times faster than the first-generation Raspberry Pi. Moreover, it is equipped with
wireless LAN and Bluetooth connectivity making it the ideal solution for powerful
connected designs.

Figure 7. Raspberry Pi 3 Model B

Meanwhile, Figure 8 shows the Raspberry Pi camera sensor that is equipped with a high
quality of 8-megapixel Sony IMX219 image sensor. It is capable to capture 3280 x
2464-pixel static images and supports 1080p30, 720p60 and 640x480p60/90 video. Its
small size and light (weighs just over 3 g), make it suitable for portable applications. It
can be connected to Raspberry Pi through a short ribbon cable.

11
 Figure 8. Raspberry Pi 8 Mega Pixel camera sensor

Raspbian Operating System is a free operating system based on Debian optimized for
the Raspberry Pi hardware. It contains a set of basic programs and utilities that run the
Raspberry Pi.

5.2 Software Requirements

Python is a programming language that can be used to write computer program in
Raspberry Pi while IDLE is an application that creates and runs the Python programs.
Python IDLE software with extensions of OpenCV computer vision is used to program
the Haar Cascade and EAR algorithms. In order to initiate the program execution, the
following libraries such as NumPy, Open CV, and pycamera are needed to be installed.
The Open CV is released under a BSD license and free for both academic and
commercial use. It has C++, C, Python and Java interfaces and supports Windows,
Linux, Mac OS, IOS and Android. It is designed for computational efficiency and with
a strong focus on real time application. It is an open-source computer vision project that
aims to provide a development platform for computer vision algorithms with collection
of libraries and applications. It provides I/O libraries for easy video data acquisition
and manipulation from multiple camera inputs.

12
 CHAPTER 5
CONCLUSION

The various measures used to detect drowsiness include subjective, vehicle-based,

physiological and behavioural measures; these were also discussed in detail and the

advantages and disadvantages of each measure were described. Although the accuracy rate

of using physiological measures to detect drowsiness is high, these are highly intrusive.

However, this intrusive nature can be resolved by using contactless electrode placement.

Hence, it would be worth fusing physiological measures, such as ECG, with behavioural

and vehicle-based measures in the development of an efficient drowsiness detection

system. In addition, it is important to consider the driving environment to obtain optimal

results.

This system is able to determine the driver state under real day and night conditions.

The results are satisfactory with an opportunity for improvement in face detection using

other techniques concerning the calculation of symmetry.

Machine Learning algorithms play a major role in identifying driver drowsiness by

using captured images or videos. Light weight algorithms can be used at the same time

reducing the execution time and cost while increasing the accuracy.

13
 CHAPTER 6
REFERENCES

[1] Hong Chung, K.: “Electroencephalographic study of drowsiness in simulated

driving with sleep deprivation,” International Journal of Industrial
Ergonomics 35(4), 307–320 (2015)

[2] F. Friedrichs and B. Yang, "Camera-based drowsiness reference for driver state
classification under real driving conditions," 2010 IEEE Intelligent Vehicles
Symposium, 2010, pp. 101-106, doi: 10.1109/IVS.2010.5548039.

[3] W. Zhang, B. Cheng and Y. Lin, "Driver drowsiness recognition based on computer
vision technology," in Tsinghua Science and Technology, vol. 17, no. 3, pp. 354-
362, June 2012, doi: 10.1109/TST.2012.6216768

[4] Zhang, Z., shu Zhang, J.: “Driver fatigue detection based intelligent vehicle
control,” in: ICPR 2006. Proceedings of the 18th International Conference on
Pattern Recognition, Washington, DC, USA, pp. 1262–1265. IEEE Computer
Society Press, Los Alamitos (2010)

[5] M. Miranda, A. Villanueva, M. J. Buo, R. Merabite, S. P. Perez, and J. M.

Rodriguez, “Portable prevention and monitoring of driver's drowsiness focuses to
eyelid movement using internet of things,” in 2018 IEEE 10th International
Conference on Humanoid, Nanotechnology, Information Technology,
Communication and Control, Environment and Management (HNICEM), pp. 1–5,
IEEE, 2018.

[6] S. S. Kulkarni, A. D. Harale, and A. V. Thakur, “Image processing for driver's

safety and vehicle control using raspberry Pi and webcam,” in 2017 IEEE
International Conference on Power, Control, Signals and Instrumentation
Engineering (ICPCSI), pp. 1288–1291, Chennai, 2017.

14
[7] S. W. Jang and B. Ahn, “Implementation of detection system for drowsy driving
prevention using image recognition and IoT,” Sustainability, vol. 12, no. 7, p. 3037,
2020.

[8] R. Oyini Mbouna, S. G. Kong and M. -G. Chun, "Visual Analysis of Eye State and
Head Pose for Driver Alertness Monitoring," in IEEE Transactions on Intelligent
Transportation Systems, vol. 14, no. 3, pp. 1462-1469, Sept. 2013, doi:
10.1109/TITS.2013.2262098.

[9] D. Mollicone, K. Kan, C. Mott et al., “Predicting performance and safety based on
driver fatigue,” Accident Analysis Prevention, vol. 126, pp. 142–145, 2019.

15