2023 3rd International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST)

7-8 th January 2023, Dhaka, Bangladesh

Thermogram-based Regions With Convolutional

Neural Network (RCNN) and Facial Biometrics
for Safe Driving
Sukanta Sarkar
Tammat Bin khayer Nafish Haque Kisan
Dept. of Electrical and Electronic
Dept. of Electrical and Electronic Dept. of Electrical and Electronic
Engineering,
Engineering, Engineering,
American International University-
2023 3rd International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST) | 979-8-3503-4643-5/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICREST57604.2023.10070063

American International University- American International University-

Bangladesh
Bangladesh Bangladesh
Dhaka, Bangladesh
Dhaka, Bangladesh Dhaka, Bangladesh
sukantosarkar88@gmail.com,
tammat385@gmail.com, NafishHaque97@gmail.com,
ORCID:0000-0001-6078-8758
ORCID:0000-0001-6917-1631 ORCID:0000-0001-5483-8931

Mohammad Nasir Uddin

Dept. of Electrical and Electronic
Engineering,
American International University-
Bangladesh
Dhaka, Bangladesh
drnasir@aiub.edu,
ORCID:0000-0001-5331-3906
Abstract— A significant number of wrongful death cases and vehicle security. Shortcomings of glare-free high
involve motor vehicle accidents. In most car accidents, the beams.[6] are they can’t provide good visuals in fog or rain,
driver is somehow at fault. This can be due to a lack of visibility, on the other hand, the automotive night vision system [5] uses
slow decision-making, or bad weather, among other things. The image intensification [7] which requires very less amount of
proposed system aims to create a safe driving assist technology light for that reason in heavy rain the vehicle’s headlight is not
consisting of thermal camera-based object detection and enough to see in the distance. So, to solve the above
intelligent vehicle anti-theft measures to assist in safe driving shortcomings thermography has been proven well-suited. In
and provide vehicle security on top of the existing system. This terms of security, car owners can create up to 36 pre-approved
proposed system not only can easily detect objects in low
authorized users. They don’t need any internet-based approval
visibility under unsuitable weather conditions, with an average
accuracy of 97% but also provides vehicle safety by using facial-
to start the vehicle. In case of unauthorized users, vehicle
biometrics-based vehicle authentication where the accuracy is owners can remotely give access to the car to anyone by facial
95%. This also has a 36 facial data saving capacity in the biometric confirmation through the app connected to a server.
database at a time. The authorized user doesn’t always require On the other hand, if car owners want to prevent someone
internet support to access the vehicle, whereas the unregistered from driving their vehicle, they can do that through this app
user needs app-based permission from the user to access the car. by only deleting the facial data from the server. This proposed
system also provides live location and real-time battery
Keywords—Regions with convolutional neural network voltage to the owner through the app.
(RCNN), Thermogram, Facial biometric, Immobilizer, YOLOv3,
YOLOv5, Safe driving, Assistive technology. II. RELATED WORK
Technologies of various kinds had been offered to a
I. INTRODUCTION range of driver assistance systems and vehicle authentication
With the rapid use of vehicles, the rate of car theft and systems. For the sake of the driver's safety and the security of
vehicle accident is increasing day by day. According to a the vehicle, some systems are extremely crucial and cannot be
report by WHO human error is to blame for at least 80% of all avoided. However, some features are merely extra benefits
motor vehicle collisions [1]. Due to poor visibility, road that solve only minor problems.
accidents increase by 19.7% [2]. Fog and heavy rain, as well
as wind-blown snow, dust, and smoke, all diminish visibility A. Research related to visual assisting technology
distance. Reduced visibility increases speed variation, which One of the earliest and most important systems that have
raises the accident risk. On the other hand, according to the been developed and used now in vehicles is adaptive cruise
FBI, 7.4 billion dollars was lost in vehicle theft and 810400 control (ACC). The ACC helps to adjust the vehicle’s speed
vehicles were stolen in 2020 only inside the USA [3]. In this to the driver’s concern [8] which helps in poor visibility
context, a driver assistance system has been developed to deal situations but is not enough to reduce the risk of accidents. To
with the visibility issue in harsh weather by detecting objects provide more safety, LEDs, are used in anti-glare high beam
using thermography and a modified object detection algorithm to exclude two or more vehicles from the light pattern. As a
called customized “You Only Look Once” YOLOv3, and an result, highlights won't interfere with approaching
innovative vehicle authentication system has been developed automobiles [6]. Though these glare-free high beams have
that can be operated by the vehicle owner through the android been providing great results at night in the nominal situation
app and facial biometrics [4]. This proposed solution but have significant shortcomings during unsuitable weather
overcomes the shortcomings of the previously implemented conditions. To overcome that shortcoming researchers have
driver-assisting technologies such as glare-free high beams developed a new technology which is called automotive night
and automotive night vision systems [5] in terms of visibility vision [5] where drivers can notice objects, including

979-8-3503-4643-5/23/$31.00 ©2023 IEEE Page 207

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DA PARAIBA. Downloaded on August 14,2023 at 18:36:07 UTC from IEEE Xplore. Restrictions apply.
 Fig. 1. Block diagram and App flow chart of the system
pedestrians, at night or in low visibility conditions during
severe weather with the help of a variety of technologies like
an intensification of the image [7], combination of sensors,
sonar sensor to identify gangers and other impediments.
Though this technology performs well but has some issues as
it uses image intensification [7] which requires minimal light
sources.
B. Research related to vehicle safety technology
A car alarm is a basic and entry-level security system that
is available in any vehicle. The headlights and turn signals of
the car could flash because of such alarms [9]. To provide
more security a vehicle immobilizer has been introduced and
an electronic security system has been developed that keeps
the engine from starting until the right key is in the ignition
[10]. Though these systems are still not enough to provide the
overall security of a vehicle. For that reason, a Biometric-
based vehicle access system has been developed where
biometric-based methods like figure print scanning, password,
or face detection have been used to unlock the car and the
owner can easily drive the car with the help of any physical
keys with the help of this system [11]. The GSM-Based
vehicle access system is the most modern and has been around
for some time to increase vehicle security. This also
overcomes the shortcomings of biometric-based
authentication through GSM [12]
C. Research related to automatic vehicle location
Car navigation systems employ digital mapping
programs like GPS and TMC [13] to provide the most up-to-
date navigational and traffic information [14].
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DA PARAIBA. Downloaded on August 14,2023 at 18:36:07 UTC from IEEE Xplore. Restrictions apply.
III. DESIGN AND SYSTEM CONCEPT
Here in Fig. 1. the block diagram has been provided. The
block diagram has shown the general arrangements of the
components which have been used in this system in a simple
schematic format. The solid purple line indicates object
detection, and the blue dotted line indicates the face detection-
based vehicle authentication system. The purple dotted box
has shown the internal processing of the raspberry pi where
the customized YOLOV5 model has been running to do the
face recognition task. The microcontroller that has been
utilized here is a Raspberry pi 4b. This specific
microcontroller incorporates two sets of camera modules and
other pieces of hardware into its overall design. The dash-
dotted line has shown the app’s flow chart, which was made
inside the mobile outline. Outside of the mobile outline the
dash-dotted line also refers to the tasks which have been
performed by the app.
A. Object Detection part
The microcontroller has included not just a display but
also a FLIR Lepton 2.5 thermal camera module for doing
thermal imaging-based object detection as part of this design.
The thermal camera module has been working after turning on
the power supply. In this part, the thermal camera module has
been adjusted according to desired output with the help of
python-based code. Then customized YOLOv3 Open CV
model has been used for object detection. This setup can
provide real-time object detection at 0.9 frames per
millisecond and can detect an object within 1 second.
B. Vehicle Authentication part
A raspberry pi camera module has been used for the face
recognition-based authentication system. This raspberry pi
camera module relates to the raspberry pi. A voltage sensor
Page 208
 and a GPS module also have been connected to the raspberry
pi. These three parts have been the input for the car
authentication system. For output, there has been used a relay
module that relates to both the raspberry pi and the self-starter
motor controls the starting system of the car. A mobile app
that has been used to control the whole authentication system.
For face detection, in raspberry pi, the modified Open CV
YOLOv5 model and coco’s modified model have been used.
This setup can detect faces within 1 second. After launching
the app, it shows the face of the driver who has been trying to
start the car. If the driver’s face is known to the owner, then
the owner may press the allow button and the unknown face
has been saved in the server after that the driver is given access
to the car. In the case of an unknown driver’s face, the car
owner may select the decline button which deletes the face
data of that unknown driver, and the driver may become
unauthorized.
IV. METHODOLOGY AND EXPERIMENTAL SETUP
This system has two parts: object detection and face
recognition and app-based vehicle authentication. In both
places, customized versions of Open CV YOLOv3 and
YOLOv5 models have been used for object detection.
Thermal Imaging is the art of converting infrared radiation
into a radiometric form. In a thermal imaging camera, a focal
plane array sensor has been used which has a pixel dimension
of 80×60 pixels. Here each pixel works as a sensory element.
infrared (IR) radiation known as a thermogram, has been
captured by a thermal imager. IR is not visible to humans, and
it ranges from 400nm to 1400nm [15]. As thermal camera
module has been producing images only from thermal energy
emission that’s why they have been used for object detection
in poor weather condition such as fog and heavy rain, as well
as wind-blown snow, dust, and smoke. In all these conditions
thermal camera has provided a significant output and can
overcome the shortcomings of conventional systems. For the
detection task, a modified YOLOv3 model has been used. Out
of four types of neural networks Faster R-CNN has been
chosen. The time required for real-time object detection is
decreased with the Faster R-CNN method. In this system,
customized YOLO models with the combination of thermal
imaging have been emphasized for the detection of humans
and vehicles in different weather conditions. All developed
models have been successfully implemented to detect objects.
According to Alexander et al. [16], the resolution for object
detection corresponds to its processing time had been a
flexible option. A standard YOLOv3 has been trained on the
coco’s data file (trained model for various objects on road)
which has achieved 97% accuracy in segmenting objects.
FLIR LEPTON 2.5 thermal imaging camera module [17]
helped to evaluate the performance.
The vehicle authentication system is sub-categorized into
three portions which are: API Service, Facial biometrics using
modified YOLOv5, and owner identification. The modified
YOLOv5 has been operated on a convolutional neural
network (CNN). Three elements make up the YOLO v5
architecture [18]. For creating a data set to train for face
recognition an image has been collected from the raspberry pi
camera module and after that, the image has been labeled
using LabelImg which is called annotation [19], and the result
of annotation has been converted into XML format. Then
these XML formats have been combined for the processing of
facial image data. For the training process, a custom YOLOv5
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DA PARAIBA. Downloaded on August 14,2023 at 18:36:07 UTC from IEEE Xplore. Restrictions apply.
has been used for face detection which has later been used as
the custom detection model. After that videos and images has
been tested. The face data that the camera has caught provides
the input. Then the built-in model has been loaded throughout
the detecting procedure and then utilized bounding boxes to
conduct categorization and prediction and the confidence
level. The outcomes of the detection have taken the form of a
class in the report database, after which the information on the
report page has been displayed in the database. The page of
the report has been developed to provide data in the app in a
form of an image. If the Face data matches, then the user didn’t
require any involvement from the app and database. Thus, the
circuit was completed, and it passed signals to the relay
modules, which further passed electricity to the starter motor.
For unmatched face data, one notification through the app has
been provided to the user where he has accepted the face data
to be recognized data if that face has been known to the owner,
if not user simply deleted the data which results in a non-
starting starter motor condition.
V. RESULTS AND ANALYSIS
Here the image analysis has been done on FLIR
Research IR Max software, and it has been only done for the
thermal image as the raspberry pi camera module provides
proper images where face detection is pretty much easier than
object detection. In this software here one thermal feed has
been inserted which has detected objects in various conditions
like glare, dark, and fog. then this thermal feed snapshot has
been inserted into Research IR software, and then it has been
analyzed. The image analysis had been done in four situations,
nominal situation, dark situation, foggy situation, and glare
situation. In Fig. 2. the nominal condition had been explained
with the help of a histogram and image profile. The
conventional image shows the user view, and the thermal
image shows the object detection. The histogram here shows
the pixel activation rate according to the infrared radiation
which means the greater the radiation greater the pixel
activation rate. On the other hand, the image profile explains
at a time, how many pixels had been activated for a certain
period of infrared radiation. In Fig. 3. for glare conditions, the
performance of the thermal imaging camera is moderate
where it had been partially activating around 1406 pixels due
to the heat produced in the glare condition which is about 121
degrees Fahrenheit. As the headlamp of a car produces bright
light they work as a heat source that interferes with specific
objects in front. The image profile is also dull in this situation
as in this condition the object detection accuracy is moderate.
Then Fig. 4. and Fig. 5. explain the image quality in dark and
fog. In both cases, this system provided excellent outcomes.
The histogram and image profile charts also properly matched
with the nominal condition histogram and image profile
charts. So as a result, it can be said that this system may
provide promising output in dark and foggy situations and can
remove the shortcomings of existing solutions. Though
existing systems provide more good results in glare conditions
rather than this proposed system. The AP measure has been
used to evaluate model performance. The detection boxes and
the corresponding ground truth bounding box's “intersection
over-union” (IoU) value have been fixed at 50%. After the
detection findings have been compared to the
ground truth. If is to be evaluated as a true positive. In the
coco data set the maximum AP value has been given 0.95 and
Page 209
 Truck 1. 00

Car 1.00

(a) Traditional perception (b) The IR photo (c) Histogram (d) Image profile
Fig. 2. Normal condition

(a) Traditional perception (b) The IR photo (c) Histogram (d) Image profile
Fig. 3. Glare condition

Person 1.00

Car 1.00

(a) Traditional perception (b) The IR photo (c) Histogram (d) Image profile
Fig. 4. Dark condition

Person 0.58

(a) Traditional perception (b) The IR photo (c) Histogram (d) Image profile
Fig. 5. Foggy condition
The minimum value is 0.50 and the interval has been given 
to 0.05. So, for the coco data set this set of values for true (P)
ĀPositiveā ı 0.5; false (P) “Positive” <0.5, and false (N)
“Negative” for no detection. Only one detection has been
deemed a genuine positive even though the same object has
been picked up more than once. For each class, the average
precision (AP) has been determined, but only one class Person
has been considered in this system. The detector's confidence
matrix's threshold has been modified, and precision-recall
curves for the desired classes have been made. With the help
of the confusion matrix [20] the calculation, recall, and
precision curve has been developed.

ܶ‫)ܲ( ݁ݑݎ‬
Recall = (1)
ܶ‫)ܲ( ݁ݑݎ‬+‫)ܰ( ݁ݏ݈ܽܨ‬
ܶ‫)ܲ( ݁ݑݎ‬
Precision = (2)
ܶ‫)ܲ( ݁ݑݎ‬+‫)ܲ( ݁ݏ݈ܽܨ‬
‫݌݈ܽݎ݁ݒ݋ ݂݋ ܽ݁ݎܣ‬
Iou = (3)
‫݊݋݅݊ݑ ݂݋ ܽ݁ݎܣ‬
mAP = 1 ∑ܰ ‫ܲܣ‬ (4)
ܰ ݅=2 ݅
Fig. 6. Fig. 6. For a stock Yolo model, the slope of precision to recall
here, N= number of classes and
‫ = ݅ܲܣ‬The Average precision of class “i” 

Page 210
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DA PARAIBA. Downloaded on August 14,2023 at 18:36:07 UTC from IEEE Xplore. Restrictions apply.
 [1] Y.-H. Ren, “Analysis of Road Traffic Accident Caused by Human
Error,” ICTIS 2013, Jun. 2013, doi: 10.1061/9780784413036.073
[2] “Association of reduced visibility with crash outcomes,” IATSS
Research, vol. 42, no. 3, pp. 143–151, Oct. 2018, doi:
10.1016/j.iatssr.2017.10.003.
[3] M. P. Levy and C. Tartaro, “Auto Theft: A site-survey and analysis of
environmental crime factors in Atlantic City, NJ,” Security Journal,
vol. 23, no. 2, pp. 75–94, Sep. 2008, doi: 10.1057/palgrave.sj.8350088.
[4] S. Mehra, “A Review on Enhancement of Face Recognition
Technology in Biometrics,” International Journal of Mechanical
Engineering and Information Technology, Aug. 2016, doi:
10.18535/ijmeit/v4i8.05.
[5] “Review of Pedestrian Detection by Video Processing in Automotive
Night Vision System,” International Journal of Science and Research
(IJSR), vol. 5, no. 4, pp. 2310–2312, Apr. 2016, doi:
10.21275/v5i4.nov163154.
[6] J. Kobbert, K. Kosmas, and T. Khanh, “Glare-free high beam
optimization based on country road traffic simulation,” Lighting
Research & Technology, vol. 51, no. 6, pp. 922–936, Nov. 2018, doi:
Fig. 7. For accustom trained Yolo model, the slope of precision to 10.1177/1477153518810997.
recall [7] P. Teichmann, “Night vision 2 — The first night-vision system to
Fig. 6. represents the AP scores of the stock YOLOv3 recognize pedestrians and cyclists,” ATZ worldwide, vol. 109, no. 10,
which is untrained, and Fig. 7. corresponds to the AP scores pp. 10–12, Oct. 2007, doi: 10.1007/bf03224960.
obtained with the trained YOLOv3 model further trained on [8] MarkVollrath, S. Schleicher, and C. Gelau, “The influence of Cruise
the custom dataset. The 29% AP score achieved in the custom Control and Adaptive Cruise Control on driving behaviour – A driving
trained YOLOv3 is higher than the 7% AP score achieved in simulator study,” Accident Analysis & Prevention, vol. 43, no. 3, pp.
1134–1139, May 2011, doi: 10.1016/j.aap.2010.12.023
stock YOLOv3. The stock YOLOv3 achieves 97% accuracy
[9] “Car alarm at the FBI,” Network Security, vol. 2016, no. 4, p. 2, Apr.
with 6% recall, while the custom trained YOLOv3 model 2016, doi: 10.1016/s1353-4858(16)30032-0.
achieves similar accuracy with a significantly greater recall of [10] J. C. van Ours and B. Vollaard, “The Engine Immobilizer: A Non-
about 20%. In TABLE I. a comparative analysis between Starter for Car Thieves,” SSRN Electronic Journal, 2013, doi:
stock and the custom-trained YOLO models has been given. 10.2139/ssrn.2214895.
[11] “MegaMatcher Automated Biometric Identification System for
national-scale projects,” www.neurotechnology.com.
TABLE I. OBJECT DETECTION RESULTS OF STOCK AND https://www.neurotechnology.com/megamatcher-
CUSTOM-TRAINED ALGORITHM
abis.html?gclid=Cj0KCQjwjvaYBhDlARIsAO8PkE3J48FHFTG9M
AkXthMbxMyfBbvA7al1WZhFHrcY-
YOLOv3 YOLOv5 GlcKEAOhxnK1Y4aAm91EALw_wcB (accessed Sep. 11, 2022).
Algorithms Stock Custom Custom Stock [12] N. Kiruthiga, L. latha, and S. Thangasamy, “Real Time Biometrics
trained trained Based Vehicle Security System with GPS and GSM Technology,”
Procedia Computer Science, vol. 47, pp. 471–479, 2015, doi:
Mean 0.803 0.82 0.82 0.747 10.1016/j.procs.2015.03.231
average 2 1 [13] I. Journal, “Advanced Car Security System Using GSM,”
precisio www.academia.edu, Accessed: Sep. 14, 2022. [Online]. Available:
n https://www.academia.edu/10175076/Advanced_Car_Security_Syste
(mAP) m_Using_GSM
Recall 0.229 0.06 0.20 0.110 [14] B. Almadani, S. Khan, T. R. Sheltami, E. M. Shakshuki, M. Musaddiq,
0 9 and B. Saeed, “Automatic Vehicle Location and Monitoring System
based on Data Distribution Service,” Procedia Computer Science, vol.
Accuracy 0.97 0.97 0.95 0.89 37, pp. 127–134, 2014, doi: 10.1016/j.procs.2014.08.021.
Speed (FPS) 41 30 30 50 [15] M. Kristo and M. Ivasic-Kos, “An Overview of Thermal Face
Recognition Methods,” in 2018 41st International Convention on
Information and Communication Technology, Electronics and
VI. CONCLUSION Microelectronics (MIPRO), 2018.
A significant number of road accidents are caused by [16] A. Alexander and M. M. Dharmana, “Object detection algorithm for
vision issues or the inability to recognize obstacles, typically segregating similar colored objects and database formation,” in 2017
because of low visibility under unsuitable weather conditions. International Conference on Circuit ,Power and Computing
Technologies (ICCPCT), Apr. 2017, pp. 1–5.
To assist the driver to understand better about his surrounding,
[17] M. G. Acorsi, L. M. Gimenez, and M. Martello, “Assessing the
we have created a project. Its goal is to show the driver a Performance of a Low-Cost Thermal Camera in Proximal and Aerial
thermal image and night vision of the road ahead and classify Conditions,” Remote Sensing, vol. 12, no. 21, p. 3591, Nov. 2020, doi:
the objects accordingly. Life forms are easier to detect in 10.3390/rs12213591.
hazardous weather using thermal vision. In addition to that, a [18] Z. Li, “Road Aerial Object Detection Based on Improved YOLOv5,”
smart anti-theft system is incorporated to stop theft from Journal of Physics: Conference Series, vol. 2171, no. 1, p. 012039, Jan.
happening. This works by sending pictures of the driver to the 2022, doi: 10.1088/1742-6596/2171/1/012039.
owner’s smartphone where the owner can allow or reject the [19] S. CAI and Q. YU, “Optimization and application of connected
component labeling algorithm based on run-length encoding,” Journal
person from starting the car. If our idea is deployed in a of Computer Applications, vol. 28, no. 12, pp. 3150–3153, Feb. 2009,
significant number of automobiles, it might prevent numerous doi: 10.3724/sp.j.1087.2008.03150.
accidents and save countless lives. [20] “(PDF) Confusion Matrix-based Feature Selection.,” ResearchGate.
https://www.researchgate.net/publication/220833270_Confusion_Mat
REFERENCES rix-based_Feature_Selection

Page 211
Authorized licensed use limited to: UNIVERSIDADE FEDERAL DA PARAIBA. Downloaded on August 14,2023 at 18:36:07 UTC from IEEE Xplore. Restrictions apply.