Implementation of Travel Buddy

Abstract :-
In this paper we have shared the solution for the drowsiness detection (travel
buddy) along with which we have given insights about the implementation. We
have used machine learning to provide the solution to one of the recurring
causes for the road accidents that is drowsiness.
Keywords: - Machine learning,Dlib, Feature Extraction, EAR, MAR etc.

Introduction: -
Transportation had always been core components for the business and overall
aspect of logistics. There are numerous causes for the road accidents such as
over speeding, not following traffic rules etc. Drowsiness being one of them
causes around 20% of accidents. Drowsiness is a process in which a person's
level of awareness diminishes due to lack of sleep and fatigue. Because of this
the vehicle driver may lose control and it can either get off the road or hit
someone.
Fortunately, it is possible to detect driver drowsiness in its early stages and
alarm the driver to avoid any potential accident. Drowsy drivers exhibit various
signs, which include repeated yawning, frequent eye closure, and repeatedly
departing street lanes. There is a need to develop a system in vehicles to
monitor the state of the driver and alert him whenever needed.

Literature Survey: -
When a driver is drowsy or sleepy, the body is in a position where it is
transitioning from being awake to being sleepy. At this point, they may lose
focus and be unable to respond to situations such as avoiding head-on
collisions or using the brakes in a timely manner. There are obvious signs that
suggest a driver is drowsy, such as
1) Frequently yawing
2) Inability to keep eyes open
3) Swaying the head forward.
The system implemented by the eye blink patterns has shown detection of an
autonomous sleepy driver monitoring and accident prevention system based
on variations in eye blink length. The detection system is composed of a video
camera and software system. Our suggested technique identifies visual
changes in eye positions by utilising the claimed horizontal symmetry property
of the eyes. Our novel approach identifies eye blinks in real-time using a
conventional camera at 110 frames per second with a 320x240 resolution. The
suggested approach identifies eye blinks with high accuracy and a 1% false
positive rate in the JZU eye-blink database.[1] Another drowsiness detection
system implemented makes use of Cb and Cr components of YCbCr colour
space; system locates the face with vertical projection function and the eye
with horizontal projection function. This system deals with automatic driver
drowsiness detection based on visual input and Artificial Intelligence. Using the
Hough Transform The study has presented a method for locating, tracking, and
analysing the driver's face and eyes in order to calculate PERCLOS, a
scientifically established measure of tiredness associated with sluggish eye
closure.[2]

Proposed System: -
Till date various implementations are done on drowsiness detection which is
important for drivers safety as millions of people travel through roads. Many
industries had developed technology for drowsiness detection like Cadillac
super cruise which is an advanced system for drowsiness detection in which a
camera is mounted on the steering wheel that monitors whether the driver is
active or dozed off due to drowsiness, then the system sends an alert to the
driver. Cadillac super cruise is the world's first true hands free driving system
for freeways. Audi, Mercedes and Volvo currently offer a drowsiness detection
system that monitors a vehicle's movements like steering wheel angle, lane
deviation, time driven and road condition. Bosh, a German supplier of
technology, is also developing a camera based system that will monitor body
posture, heart rate and body temperature. We also have developed a
drowsiness detection system based on eyes and mouth movement under
different luminescence. Our system has 3 modules, the first model comprises
face detection, second calculates EAR(eyes aspect ratio) and MAR(Mouth
Aspect Ratio) ,third is UI to host the model for the user. When the calculated
eye's aspect ratio is less than the threshold value which is set during system
implementation then the system gives alert to the driver. Similar aspect of
implementation is followed by the mouth aspect ratio for which if the value
calculated is greater than the threshold value it sends an drowsiness alert.

Fig no.1:- Proposed System diagram

Above block diagram shows the basic yet necessary steps for the model
implementations which have been proposed. We have designed a GUI for
hosting the model through which real time face is captured using opencv. After
that, the captured face is mapped with landmarks which are present in dat file.
These captured frames per second(FPS) are compared with the assigned
threshold value for ERA, MAR and accordingly alert is given to the driver. The in
depth explanation of the implementations is given below.
With the proposed solution we have tried to solve the issues occurred during
the other implementations that are as follows:
- Accurate detection of eyes for the people wearing specs
- Face detection under different illumination conditions
-Fusion of the two models that are EAR and MAR detection

Implementation: -
The main idea of this project is to plan a system to distinguish driver’s
sluggishness by persistently checking the facial expressions of the driver by
facial landmark detection concept.
For the implementation of the travel buddy we have used the Python
programming language which is known for its versatile applications in the
different fields such as web development, machine learning etc.
Along with it we have scrammed the basics of computer vision to get to know
about the process of face detection and feature extraction.The whole
implementation has been divided in three main parts.
 1. Face detection
2. Mapping of the features using DLIB
3. Algorithm for the Drowsiness Alert
Various methods have been initiated to implicate the face detection for
different models.Opencv is a famous library which is used for computer vision
and image processing. Here with the help of which we have captured the real
time data for the monitoring and processed it with the features of Dlib file. The
workflow of the project is explained as follows: The image is captured using the
webcam.The captured image is processed in the openCV environment. In order
to detect drowsiness the first step is to detect the face taken by the camera.
Then facial expressions like the eye closure, yawning, speaking, position is
detected by Facial landmark detection concept. The captured image is sent to
the processor for image processing. It converts the received image to digital
signal using openCV. The eye closure is detected by estimating Eye Aspect Ratio
(EAR).When EAR goes below the threshold value,drowsiness is confirmed and
the driver is alerted. The yawn detection is carried out by estimating Mouth
Aspect Ratio (MAR).When MAR value goes above the threshold value, yawning
is confirmed. Also, the speaking alert is given with the help of MAR value.
EAR and MAR values can not be randomly selected as there can be various
shapes to the eyes; the values that we have selected are the optimal ones as
they can respond to the maximum number of cases. Also, the people having
the specs in front of their eye should be able to detect the eyes without any
defect hence the values selected are justified and are verified to use.
After the face detection has been done the next step is to map the features of
Dlib landmarks .dat file with the captured data. The Dlib has its own features
which makes it more usable and accurate. If we compare the requirements that
are needed for the successful implementation the parameters on which we
compare the module are accuracy, response time and robustness of the model.
The requirements for the drowsiness detection are as follows:-
1. Face detection within a certain distance
2. Face detection under different illumination condition
3. Eyes detection for the people with specs
These conditions are satisfied with the use of Dilb model which is accurate to
the cases under which the model has to detect the faces of the driver.
Dlib has the following features:
1. Fastest method on CPU
2. Works very well for frontal and slightly non-frontal faces
3. Lightweight model as compared to other models for face detection.
4. Works under small occlusion, and in different illumination
conditions.

After mapping the facial features with the real time face we locate the
ROI(Region Of Interest) here they are eyes and mouth. The mapped points are
used to calculate the euclidean distance between them by which the threshold
values are compared.Following diagram shows the 68 landmarks which are
used to map the features.

Fig no.2:- Face shape with 68 landmarks

HOG + Linear SVM (Using Dlib) Dlib is a general-purpose cross-platform

software library written in the programming language C++.Its design is heavily
influenced by ideas from design by contract and component-based software
engineering. Thus, it is first and foremost, a set of independent software
components. It is open source software released under a Boost Software
License. Since development began in 2002, Dlib has grown to include a wide
variety of tools. As of 2016, it contains software components for dealing with
networking, threads, graphical user interfaces, data structures, linear algebra,
machine learning, image processing, data mining, XML and text parsing,
numerical optimization, Bayesian networks, and many other tasks. In recent
years, much of the development has been focused on creating a broad set of
statistical machine learning tools. The pre-trained facial landmark detector
inside the dlib library is used to estimate the location of 68 (x, y) coordinates
that map to facial structures on the face. Euclidean Algorithm It is only a
distance measure between a couple of tests p and q in a n-dimensional
element space: Concerning Euclidean calculation, an assessment is made in
one assessment by fixing two on a line, and picking one to be the root.

Fig no.3:- Euclidean distance graph

The length of the line part between these centers portrays the unit of bundle
and the heading from the inspiration to the subsequent point is depicted as the
positive bearing. This line section may be made an explanation of along the line
to manufacture longer divides whose lengths identify with eventual outcomes
of the unit isolated. The subsequent point is then astoundingly picked as the
point on hold that is at a unit of one certain unit from the soonest beginning
stage put together. The division between any two on the certified line is the
firm assessment of the mathematical difference of their direction. It is
completely expected to see the name of a point with its Cartesian help.

Eyes closure detection:

Fig no.4:- Flow chart for EAR

Eyes closure detection also known as perclos frequency by which we can
understand if a person is feeling drowsy or not.On average a person has a
percol movement around 15-20 times per minute.If the driver’s eyes are closed
for continuous frames captured the alert is given. The eyes have a total 12
features points to be mapped given six to each eye as per shown below.

Fig no.5:- Eyes feature points

The EAR is calculated for each frame taken as input and compared with the
threshold value.When eyes are opened,EAR value will be high.When the eyes
are closed,value of EAR will be lesser.When the EAR value goes below a certain
threshold,eye closure will be detected.Upon detecting the eye closure driver is
alerted by triggering the alarm. The formula for the EAR Calculation is as
follows:

Fig no.6:- EAR formula

The numerator of this equation computes the distance between the vertical
eye landmarks while the denominator computes the distance between
horizontal eye landmarks, weighting the denominator appropriately since there
is only one set of horizontal points but two sets of vertical points.The graph
that we obtain after the comparison of the EAR :

Fig no.7:- Theoretical graph of EAR

The drop in the graph indicates that the person hasn't opened their eyes for a
certain time.At the time of implementation continuous drop in the value is
monitored for the accurate result.The threshold value for the implementation is
0.35.

Speaking and yawning detection:

 Fig no.8:- Flow chart of MAR

While driving, talking continuously can be distracting as well which may lead to
an accident, at the same time a person yawns when he/she feels drowsy hence
calculation of MAR is equally important while detecting the fatigue of the
driver.MAR is the ratio of vertical distance of mouth to the horizontal distance
of the mouth. The three vertical distances and one horizontal distance are
considered in MAR estimation.When driver yawns,the vertical distances will
start increasing and the horizontal distance will decrease slightly. MAR value for
yawning person is assumed to be greater than 15 for faithful detection while
the other values of MAR less than threshold are omitted. For the calculation of
the MAR we consider the landmarks associated.
 Fig no.9:- Mouth landmarks points

The formula for the MAR calculation as follows:

Fig no.10:- MAR formula

When the mouth is opened the three vertical distances will be greater and the
horizontal distance will be lesser. So as a result, MAR value will increase upon
yawning.Also, when person will talk the MAR value fluctuates accordingly
hence by applying the upper threshold 10 and lower threshold 5 the value for
the MAR the speaking is detected if the counter exceeds the limit assigned for
the speaking value the alert is send to the driver to have less distraction.The
MAR graph is also plotted for the visual comparison
 Fig no.11:- Theoretical graph of MAR

Results: -
Face detection output:-
When the option for monitoring is selected from the GUI provided the face
recognition immediately starts where multiple frames are captured per minute.

Fig no.12:- Output of normal face detection

Drowsiness detection:-
Upon closing eyes our yawning drowsiness alert is given to the driver hence the
window also shows the alert message on the display.

Fig no.13:- Output of drowsiness alert by calculating EAR

 Fig no.14:- Output for drowsiness alert by calculating MAR

In day to day life there can not be the same illumination condition all day long,
luminance changes with time hence the model should be able to detect the
face under various conditions.We have considered the three basic conditions
which can affect the face recognition of the model the conditions are as
follows:-
1. Normal front light
2. Side light; slightly dark
3. Backlight
The analysis of these conditions is given in the following table.

SR.No Condition Input Output Accuracy

1 Normal Front
Light

2 Side Light

3 Backlight
For the eye closure and yawning detection the analysis as follows:

Sr.No. Condition No. of times no. of times Accuracy

blinked detected

1 Normal Front
Light

2 Side Light

3 Backlight

Sr.No. Condition No. of times no. of times Accuracy

yawned detected

1 Normal Front
Light

2 Side Light

3 Backlight

Real time graph for EAR: -

Fig no.15:- Real time graph of EAR

Real time graph for MAR: -
 Fig no.16:- Real time graph for MAR

Conclusion: - This paper has proposed a real time solution for one of the major
causes of the road accidents. We have proposed the system for drowsiness
detection with the help of machine learning techniques. We are calculating the
eyes aspect ratio and mouth aspect ratio for finding out drowsiness of the
driver.

Reference:-
[1] Taner Danisman, Ian Marius Bilasco, Chabane Djeraba, Nacim Ihaddadene
“Drowsy Driver Detection System Using Eye Blink Patterns”
[2] Belal ALSHAQAQI; Abdullah Salem BAQUHAIZEL; Mohamed El Amine OUIS;
Meriem BOUMEHED; Abdelaziz OUAMRI; Mokhtar KECHE “DRIVER
DROWSINESS DETECTION SYSTEM”
[3] M. Omidyeganeh, A. Javadtalab, S. Shirmohammadi “Intelligent Driver
Drowsiness Detection through Fusion of Yawning and Eye Closure”
[4] Saroj K.L. Lal, Ashley Craig “A critical review of the psychophysiology of
driver fatigue”
[5] Jun-Juh Yan, Hang-Hong Kuo, Ying-Fan Lin, Teh-Lu Liao “Real-time Driver
Drowsiness Detection System Based on PERCLOS and Grayscale Image
Processing”2016 International Symposium on Computer, Consumer and
Control
[6] https://morth.nic.in/road-accident-in-india

[7] Kusama Kumari B.M “Review on Drowsy Driving: Becoming Dangerous

Problem”

[8] Vandna Saini, Rekha Saini, “Driver Drowsiness Detection System and
Techniques: A Review”

[9] Mkhuseli Ngxande, Jules-Raymond Tapamo, Michael Burke, “Driver

drowsiness detection using Behavioral measures and machine learning
techniques: A review of state-of-art techniques”

[10] Maliha Khan, Sudeshna Chakraborty, Rani Astya, Shaveta Khepra, “Face
Detection and Recognition Using OpenCV”

[11] Mamata S. Kalas “Real Time Face Detection and Tracking Using OpenCV”

[12] Nataliya Boyko, Oleg Basystiuk, Nataliya Shakhovska, “Performance

Evaluation and Comparison of Software for Face Recognition, based on Dlib and
OpenCV Library”

[13] Tiesheng Wang, Pengfei Shi, “Yawning Detection for Determining Driver
Drowsiness”

[14] Nikolay Neshov, Agata Manolova, “Drowsiness Monitoring in Real-time

based on Supervised Descent Method”