111Equation Chapter 1 Section 1VIETNAM NATIONAL UNIVERSITY –

HOCHIMINH CITY
INTERNATIONAL UNIVERSITY
SCHOOL OF ELECTRICAL ENGINEERING

Developing AI Based Automatic Door to

Detect Mask & Measure Temperature

BY
GROUP 1 – KLÚTA

ĐÀO NGỌC QUỐC KHÁNH

TRƯƠNG HOÀNG BẢO LONG
PHẠM VĨNH PHÚ
NGUYỄN DƯƠNG SONG TOÀN
NGUYỄN LÊ QUỐC ANH
-
EEEEIU18003@student.hcmiu.edu.vn

A SENIOR PROJECT SUBMITTED TO THE SCHOOL OF ELECTRICAL

ENGINEERING IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE
DEGREE OF BACHELOR OF ELECTRICAL ENGINEERING
HOCHIMINH CITY, VIETNAM
2021

ii
 Developing AI Based Automatic Door to
Detect Mask & Measure Temperature
BY

GROUP 1 – KLÚTA

Under the guidance and approval of the committee, and approved by its members, this

thesis has been accepted in partial fulfillment of the requirements for the degree.

Approved:

________________________________
Chairperson

_______________________________
Committee member

________________________________
Committee member

________________________________
Committee member

________________________________
Committee member

i
 HONESTY DECLARATION

My name is ______________ (full name). I would like to declare that, apart from

the acknowledged references, this thesis either does not use language, ideas, or other

original material from anyone; or has not been previously submitted to any other

educational and research programs or institutions. I fully understand that any writings in

this thesis contradicted to the above statement will automatically lead to the rejection

from the EE program at the International University – Vietnam National University

Hochiminh City.

Date:

Student’s Signature

(Full name)

ii
 TURNITIN DECLARATION

Name of Student:

Date:

Advisor Signature Student Signature

iii
 ACKNOWLEGMENT

It is with deep gratitude and appreciation that I acknowledge the professional

guidance of Dr. Nguyen Van B. His constant encouragement and support helped me to

achieve my goal.

………………………

iv
 TABLE OF CONTENTS

HONESTY DECLARATION.............................................................................................ii
TURNITIN DECLARATION............................................................................................iii
ACKNOWLEGMENT.......................................................................................................iv
TABLE OF CONTENTS....................................................................................................v
LIST OF TABLES............................................................................................................vii
LIST OF FIGURES..........................................................................................................viii
ABSTRACT........................................................................................................................x
CHAPTER I INTRODUCTION.........................................................................................1
CHAPTER II DESIGN SPECIFICATIONS AND STANDARDS...................................4
2.1. Hardware specification............................................................................................4
2.2. Software specification..............................................................................................4
CHAPTER III PROJECT MANAGEMENT......................................................................5
3.1. Budget and Cost Management Plan.........................................................................5
3.2. Project Schedule.......................................................................................................5
3.3. Resource Planning....................................................................................................6
CHAPTER IV LITERATURE REVIEW............................................................................7
4.1. COVID-19: Face Mask Detector with OpenCV, Keraas/TensorFlow, and Deep
Learning..........................................................................................................................7
4.1.1. Creating dataset................................................................................................8
4.1.2. Training Face Mask Classifier with Keras and TensorFlow............................8
4.1.3. Applying Face Mask Classifier to image with OpenCV..................................9
4.1.4. Applying Face Mask Classifier to real time video with OpenCV..................10
4.2. TouchFree: Automated Temperature Checkup and Mask Detection.....................11
4.3. Automatic Door Opener using Arduino and PIR Sensor.......................................14
CHAPTER V METHODOLOGY.....................................................................................16
5.1. Overview................................................................................................................16
5.2. Objectives...............................................................................................................17
5.2.1. Hardware Setup..............................................................................................17
5.2.2. Constructing Mask Detection Program..........................................................21
5.2.3. Constructing Temperature Checkup Program................................................23
5.2.4. Constructing Auto-Operating Door................................................................24
5.2.5. Integration and Test........................................................................................24

v
CHAPTER VI EXPECTED RESULTS............................................................................27
CHAPTER VII CONCLUSION AND FUTURE WORK................................................28
REFERENCES..................................................................................................................29
APPENDICES...................................................................................................................30

vi
 LIST OF TABLES
Table 2.1: Component list....................................................................................................4
Table 2.2: Software list........................................................................................................4
Table 3.1: Component Price................................................................................................5

vii
 LIST OF FIGURES
Figure 1.1: Thermal checkup using thermometer................................................................1
Figure 1.2: System Illustration............................................................................................2
Figure 3.1: Gantt Chart........................................................................................................6
Figure 4.1: Phases and individual steps for building a COVID-19 face mask detector......7
Figure 4.2: Original/artificial face mask applied image......................................................8
Figure 4.3: Coronavirus face mask......................................................................................8
Figure 4.4: COVID-19 face mask detector training accuracy.............................................9
Figure 4.5: Not wearing mask result..................................................................................10
Figure 4.6: Mask Detection by Deep Neural Network......................................................11
Figure 4.7: Find the best Spot on the Forehead using DLIB Module................................12
Figure 4.8: Temperature Reading using Contactless IR Temperature Sensor...................12
Figure 4.9: High temperature warning test........................................................................13
Figure 4.10: Email Based Alert.........................................................................................14
Figure 4.11: Automatic door system-based Arduino.........................................................15
Figure 4.12: Circuit Diagram for the system.....................................................................15
Figure 5.1: System Overview............................................................................................16
Figure 5.2: Raspberry setup...............................................................................................17
Figure 5.3: Hardware schematic for PID system...............................................................20
Figure 5.4: Localizing face in image.................................................................................21
Figure 5.5: Detecting ROI.................................................................................................21
Figure 5.6: Display result..................................................................................................22
Figure 5.7: PID system result............................................................................................23
Figure 5.8: Measuring temperature process.......................................................................23
Figure 5.9: Door operating condition................................................................................24
Figure 5.10: Test the mask detection program..................................................................25
Figure 5.11: Test the temperature checkup program.........................................................25
Figure 5.12: Expectation results from testing....................................................................26

viii
ABBREVIATIONS AND NOTATIONS

PAN: Personal Area Network

MAC: Medium Access Control

ix
 ABSTRACT

Under the threat from unpredictable evolving of COVID-19 variants, the need to improve

the pandemic prevention method is crucial. Currently, social distancing policy has shown

its effectiveness in minimize the spreading chance at public places. Nevertheless,

majority of the temperature checking methods used in Vietnam are done manually by

security staffs, which is against the mentioned policy and may indirectly gain the chance

of infection. Thus, our project aims to develop a method implementing AI to create an

automatic entrance kiosk – a door that automate the temperature checking process as well

as mask detection at crowd places without the interference from human. In this project,

Facial Landmark is used in automatic temperature checkup process and Deep Learning

Neural Network is used in mask detection process. The door will, then, operate based on

data collected from temperature checkup and mask detection process. About application,

our project can be practically installed at public places such as school, hospital, airport,

etc.

Keywords: COVID-19, prevention method, temperature checking method, mask

detection, automatic entrence kiosk, social distancing

x
xi
 CHAPTER I

INTRODUCTION

Until now, there is no exact treatment for COVID-19 virus. Thus, outbreak of the pandemic can
result in city or even national lockdown, which causes huge damage to the economics of the
world. So far, to prevent the spread of the pandemic people need to follow social distancing
policy from the governments. Nevertheless, majority of public places in our country use infrared
thermometer to check the temperature of entering people (illustrated in Figure 1.1). This method
requires human to operate, which may indirectly cause spreading of the pandemic. Thus, there
will be needed certain automatic system to improve the efficiency of this process by replacing
the human workforce.

Figure 1.1: Thermal checkup using thermometer

Symptoms of COVID-19 including fever. We believe that implementing AI into our system, we
can easily measure temperature without interference from human. Furthermore, there are a lot of
research on developing mask-wearing detection program using machine learning out there. Thus,

1
we would like to integrate the two mentioned AI programs into our system to construct an
automatic door that can be able to measure temperature and detect temperature of passing
people.

Figure 1.2: System Illustration

This project proposes an automatic door – based AI, which illustrated by Figure 1.2. This
system aims to set up in social places such as mall, hospital, supermarket, school, etc. People
come and the system will detect mask and temperature. First, when people come to the door, the
system will detect people wearing the mask or not by applying trained AI. The system will have
a screen which will warning people if they do not wear mask. This is described in the number (1)
in Figure 1.2. Next, if people are wearing mask, they have to pull the mask down to show their
full face. The system will use Facial Landmark to detect the forehead on the face, then using a
temperature sensor to measure the temperature (described as (2) in Figure 1.2) and show the data
on the screen (described as (3) in Figure 1.2). The model of the door is a bar connecting to servo
motor. The automatic door will open if both condition in (1) & (2) are met. In general, the goal
of the project is to develop an AI based door to detect mask and measure temperature. To archive
the goal, the following objectives should be completed.

1. Objective 1: Hardware set up

1.1. Camera set up & Touchscreen set up
1.2. Sensor set up
1.3. Hardware setup for PID system & automatic door

2. Objective 2: Construct the mask detection program

2.1. Create dataset

2
 2.2. Training AI with created data

2.3. Apply the mask detection

3. Objective 3: Construct temperature checkup program

3.1. Develop Detecting Forehead Program

3.2. Apply DLIB Module for Facial Landmarking
3.3. Develop PID program

4. Objective 4: Build the automatic door

4.1. Construct the door

4.2. Program the door

5. Objective 5: Test the system

5.1. Test the mask detection
5.2. Test the temperature checkup
5.3. Test the door program

6. Objective 6: Finish the report.

3
 CHAPTER II

DESIGN SPECIFICATIONS AND STANDARDS

In this chapter, we will discuss about the specifications of our project. There are two sections, we
will mention about hardware specifications in section 2.1 and software specifications in section
2.2.

2.1. Hardware specification

Table 2.1 below shows hardware components used in this project.
The complete model product is expected to have length, width, and height is 20x7x25 cm.

Table 2.1: Component list

Components and supplies

MCU Raspberry Pi 3 Model B
Camera Raspberry Pi camera module
Non-Contact Infrared Temperature sensor
Sensor
module (MLX90614)
Display Raspberry Pi touch display
Servo SG90 Micro-servo motor
External memory card Any 16 Gb+ card
Adapter Raspberry Pi power adapter

2.2. Software specification

Table 2.2 below shows software components used in this project.

Table 2.2: Software list

Apps and online services

Raspberry OS Raspberry Pi Raspbian
TensorFlow
Libraries OpenCV
DLIB – Facial Landmarking

4
 CHAPTER III

PROJECT MANAGEMENT

In this chapter, we will discuss about the price of the components as well as the task assignment
in this project in three sections.

3.1. Budget and Cost Management Plan

Table 3.1 below shows the price list for the component used in this project.

Table 3.3: Component Price

Components Prices
Combo Raspberry Pi 4 Model B 2.065.000
Raspberry Pi Camera Module 650.000
Non-contact Infrared Temperature Sensor Module (MLX90614) 180.000
Pan Tilt Kit 282.000
SG90 Micro-servo motor 35.000
Raspberry Pi Touch Display 1.400.000
Premium Female/ Female Extension jumper wires, 40 x 6” (150) 33.000
Total 4.645.000

3.2. Project Schedule

- Based on the goal and objectives mention in Chapter 1, the Gantt chart of our project is
shown in Figure 3.1

5
 Figure 3.3: Gantt Chart

3.3. Resource Planning

- According to Figure 3.1, we assign the task to each member as following. Phú will prepare the
hardware components. Toàn will do the preparation for software. Khánh and Toàn will develop
the mask detection program. Long, Phú and Anh will develop temperature checkup program.
Toàn, Khánh and Anh will construct the automatic door. Finally, Anh will complete the report.

6
 CHAPTER IV

LITERATURE REVIEW

In this chapter, we discuss about three reference articles (in section 4.1, 4.2, and 4.3) that have
solutions we would like to base on building our project.

4.1. COVID-19: Face Mask Detector with OpenCV, Keraas/TensorFlow, and Deep
Learning
In studying for mask detection program, an article [CITATION Adr20 \l 1033 ] posted by Ardian
Rosebrock clearly demonstrates the process of developing mask detection program. Being
generalized by Figure 4.1, the article discusses two phases in programming mask detector
program. In the first phase or training phase, dataset is reviewed to train custom face mask
detector. Then, the training process begin using Keras and TensorFlow. In the second phase or

Figure 4.4: Phases and individual steps for building a COVID-19 face mask detector

7
deployment phase, the author, first, create a script to detect face in image and resize the image
such that the face is clearly appear in the frame. Then, the program will apply face mask
classifier to the resized frame to analyze the face and show result

4.1.1. Creating dataset

In this article, 1376 images of people not wearing mask used to create dataset. The dataset
included two classes with_mask (690 images) and without_mask (680 images). To create
with_mask set, Facial Landmark is used to extract the face and apply a mask image (as shown in
Figure 4.3). This process yields the result as shown in Figure 4.2.

8
 Figure 4.5: Coronavirus face mask

Figure 4.6: Original/artificial face mask applied image

This method help reduce time spending on finding images of people wearing mask. Though, it
should be notice that we cannot use the images in without_mask set to create with_mask set, as it
can make the result get heavily bias[ CITATION Adr20 \l 1033 ].

4.1.2. Training Face Mask Classifier with Keras and TensorFlow

MobileNet V2 architecture is fine-tuning in the article [ CITATION Mar19 \l 1033 ] to establish

base line model for training face mask classifier. The article fine-tuning by a three-step process.
First, MobileNet is loaded with pre-trained ImageNet weights, without head of network, Then, A
new FC head is constructed, and appended to the base in place of the old one. Finally, the base
layers are frozen. The weight of these base layers will not be updated during the process of
backpropagation, whereas the head layer weights will be tuned.

9
 Figure 4.7: COVID-19 face mask detector training accuracy

Figure 4.4 shows the test set result in the article with accuracy ~99%. Additionally, there are
little signs of overfitting, with the validation loss lower than the training loss. Thus, the classifier
is expected to generalize well to images outside of the training and testing set.

4.1.3. Applying Face Mask Classifier to image with OpenCV

This step can be generalized as following. First, the input image is loaded from disk. Then,
the program will detect faces in the image and eventually, apply the face mask detector to
classify the face as either with_mask or without_mask. Figure 4.5 below shows result to no
mask wearing image after applying the classifier.

10
 Figure 4.8: Not wearing mask result

4.1.4. Applying Face Mask Classifier to real time video with OpenCV

With similar approach to the program mentioned in section 4.1.2, the author suggests looping
over frame in the video. In every frame of the video, the program will extract Region of Interest
(ROI) of face and apply face mask classifier to that ROI to get result.

After studying the article, we recognize that the face mask classifier in real time streaming video
mentioned in section 4.1.4 is the solution to our mask detection program. Thus, we will construct
our mask detection program based on the mentioned section. We also follow the reference in
section 4.1.1 and 4.1.2 to create data set and train the face mask classifier.

11
4.2. TouchFree: Automated Temperature Checkup and Mask Detection

Moving on to the last article [CITATION Sak20 \l 1033 ], the kiosk of Saksham Bhutani
operates by first trying to detect mask on the passing person as shown in Figure 4.6. Then, after
recognizing mask on, the system will ask the person to remove mask, so that the temperature
checking program can work properly.
The protocol he used to connect and control the sensor through raspberry pi is the I2C protocol.
The

Figure 4.9: Mask Detection by Deep Neural Network

Once, the System has detected the Mask it asks the user to remove the mask so that it
can perform Facial Landmarking. The System is using DLIB Module for Facial
Landmarking to find the best Spot on the Forehead of the person to take Temperature
from. (Figure 4.7)

12
 Figure 4.10: Find the best Spot on the Forehead using DLIB Module

Then by using PID Control System with Servo Motors, the system tries to align the Selected
Spot on the Forehead with the Sensor. Once aligned the system takes Temperature Reading using
Contactless IR Temperature Sensor (Figure 4.8)

Figure 4.11: Temperature Reading using Contactless IR Temperature Sensor

13
When the temperature too high, it will alarm and not allow the person with high temperature to
enter (Figure 4.9)

Figure 4.12: High temperature warning test

The temperature checking program used Facial Landmark & PID control system to find the
forehead of people. When operating, the kiosk will use constantly finding the preset facial
landmark spots that represent the forehead by PID control system and try to align the infrared
sensor to those spots. Thus, the temperature can be accurately measured. Each time, when there
are people passing through the kiosk, the system will send an alert mail to the administrator as
shown in Figure 4.10.

14
 Figure 4.13: Email Based Alert

After analyzing article[ CITATION Sak20 \l 1033 ], we recognize the temperature checkup
program used in the article has similar purposed to ours. Thus, we will construct our mask
detection program based on the solution in section 4.2 to develop the PID system to search for
the forehead and measure the temperature.

4.3. Automatic Door Opener using Arduino and PIR Sensor

In the article [CITATION Kat20 \l 1033 ], they use Arduino to control the door, which is
modeled by a CD tray as shown in Figure 4.11.

15
 Figure 4.14: Automatic door system-based Arduino

In this article, PIR Motion Detector Sensor is used to detect motion of people. If the sensor
detected a person, the door will open for a while and automatically close after that. They also use
the L298N Motor Driver Module to control the 5V driving motor of the door.

Figure 4.15: Circuit Diagram for the system

Referring to Figure 4.12, we can interpret the mechanism of the system as following. First, when
there are people approaching the door, the PIR sensor will send a HIGH signal to pin 8 of the
Arduino. The Arduino will then send a signal via port 2 & 3 to the L298N to control the motion
of the motor to drive the door.

Since our project need to be affordable in testing process, we decide to follow the simple
automatic door system project to minimize the cost. The article mentioned above meets our
desire. Thus, we will construct our automatic door system with a model as mentioned above.

16
 CHAPTER V

METHODOLOGY

In this chapter, we will first describe generally our system in section 5.1. Then we will discuss in
detail how each objective gets done to build up the complete system in section 5.2

5.1. Overview

Figure 5.1 shows the system overview of our project. There are three main blocks in the system,
which are the mask detection block, the temperature checkup block, and the automatic door
block. In operation, our system will start with the mask detection block, the mask detection
program will load pre-trained face mask classifier from disk, then it will extract the face in video
stream and apply the classifier to check for mask-wearing. Once mask is detected, the system

Figure 5.16: System Overview

will move to the temperature checkup block by requiring user to remove the mask to run the
temperature checkup program. The program based facial landmark with PID control system will
constantly try to align selected spots represent the forehead with the sensor to read the

17
temperature using contactless IR temperature sensor. Finally, the automatic door, which is
modeled by a DC servo, will open when both conditions are met from the previous programs.

5.2. Objectives

In other to build the properly working system as described in section 5.1, we need to complete
the following objectives

5.2.1. Hardware Setup

We first install the Raspberry OS to the SD card and connect it to the port as shown in Figure
5.2. The remain ports should be connected as shown in Figure 5.2 as well.

Figure 5.17: Raspberry setup

5.2.1.1. Set up the camera & touchscreen

Connect the camera and touchscreen to the Raspberry as shown in Figure 5.2. Then, we follow
the instruction from the official forum for the Raspberry to set up them.

Next, the following packages and libraries will need to be installed

line 1 $ sudo apt-get update
line 2 $ sudo apt-get upgrade

Command line 1 and 2 are for updating the Raspberry Pi OS:

line 3 $ pip3 install --upgrade pip3

Command line 3 is for updating the newest python version 3

After updating pip3, we will expand Swap Size and Changing Memory
line 4 $ sudo nano /etc/dphys-swapfile

Command line 4 is for navigating to CONF_SWAOSIZE =100 and changing to

CONF_SWAPSIZE=1024

18
Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system
needs more memory resources and the RAM is full, inactive pages in memory are moved to the
swap space. While swap space can help machines with a small amount of RAM, it should not be
considered a replacement for more RAM. Swap space is located on hard drives, which have a
slower access time than physical memory.

line 5 $ sudo pip3 install numpy

line 6 $ sudo pip3 install scipy
line 7 $ sudo pip3 install scikit-image
line 8 $ sudo pip3 install RPi.GPIO
line 9 $ sudo pip3 install smbus2
line 10 $ sudo pip3 install PyMLX90614

Command lines from 5 to 10 are for installing some necessary libraries

Installing OpenCV:
line 11 $ sudo apt-get install build-essential cmake unzip pkg-config
line 12 $ sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
line 13 $ sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-
dev
line 14 $ sudo apt-get install libxvidcore-dev libx264-dev
line 15 $ sudo apt-get install libgtk-3-dev
line 16 $ sudo apt-get install libcanberra-gtk*
line 17 $ sudo apt-get install libatlas-base-dev gfortran
line 18 $ sudo apt-get install python3-dev

Command lines from 11 to 18 are for installing important dependencies for the OpenCV
line 19 $ sudo apt-get install python3-opencv

Command line 19 then install the OpenCV module.

line 20 $ sudo apt-get install -y libhdf5-dev libc-ares-dev libeigen3-dev gcc
gfortran python-dev libgfortran5 \ libatlas3-base libatlas-base-dev libopenblas-dev
libopenblas-base libblas-dev \ liblapack-dev cython libatlas-base-dev openmpi-bin
libopenmpi-dev python3-dev
line 21 $ sudo pip3 install keras_applications==1.0.8 --no-deps
line 22 $ sudo pip3 install keras_preprocessing==1.1.0 --no-deps
line 23 $ sudo pip3 install h5py==2.9.0
line 24 $ sudo pip3 install pybind11
line 25 $ pip3 install -U --user six wheel mock
line 26 $ wget "https://raw.githubusercontent.com/PINTO0309/Tensorflow-
bin/master/tensorflow-2.2.0-cp37-cp37m-linux_armv7l_download.sh"
line 27 $ chmod u+x tensorflow-2.2.0-cp37-cp37m-linux_armv7l_download.sh
line 28 $ ./tensorflow-2.2.0-cp37-cp37m-linux_armv7l_download.sh
line 30 $ sudo pip3 uninstall tensorflow
31 $ sudo -H pip3 install tensorflow-2.2.0-cp37-cp37m-linux_armv7l.whl

Command line from 20 to 30 are for installing the tensor flow module and libraries.
Installing DLIB module:
line 32 $ pip3 install dlib

Command line 34 is for installing the DLIB module.

Then check the installations:

19
line 33 $ python3
line 34 >>> import dlib
line 35 >>> exit()

Command line 33 to 34 are for checking the DLIB module has already in the python
environment.
Now we will change Swap Size and Memory back
line 36 $ sudo nano /etc/dphys-swapfile

Command line 36 is for changing back the swap size and memory by navigating to the swap file
and then from CONF_SAWPSIZE=1024 we change to CONF_SWAPSIZE=100

line 37 $ sudo /etc/init.d/dphys-swapfile stop

line 38 $ sudo /etc/init.d/dphys-swapfile start
line 39 $ sudo raspi-config

Command line from 37 to 38 are for applying the changes

Finally, reboot the Raspberry

Now, with the system ready, we will create a folder name caps_gr1_proj to store the code of this
project
line 40 $ mkdir caps_gr1_proj
line 41 $ cd caps_gr1_proj

Command line from 40 to 41 are for creating a new folder and storing the programming code of
the systems.
5.2.1.2. Set up the sensor
In this project, we will be using the MLX90614 sensor which is shown in Figure 5.3 (Contact IR
Temperature Sensor) and the protocol to connect this to the Raspberry is I2C protocol.
I2C communication is a feature that allows multiple slaves connect to a single master and you
can have multiple master controlling multiple single, or multiple slaves. I2C only uses two wires
to transmit data between devices: SDA (Serial Data) for transmitting data and SCL (Serial
Clock) for carrying the clock signal. I2C has a way to communicate to the slave device is
addressing them. The master sends the address of the slave it wants to communicate with to
every slave connected to it. Each slave compared the address sent from the master to its own
address.  If the address matches, it sends a low voltage ACK bit back to the master. If the address
doesn’t match, the slave does nothing, and the SDA line remains high.
After connecting the sensor to the Raspberry Pi, we set up the protocol
line 42 $ sudo su -c 'echo "Y" > /sys/module/i2c_bcm2708/parameters/combined'
line 43 $ sudo i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- 5a -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

20
Command line 42 and 43 are for calling out the address of the port that the sensors connect to.
Here, we have the port address connecting to the sensor is 0x5A.

5.2.1.3. Set up the hardware for PID system & the automatic door
We connect the servos X and Y as shown in Figure 5.3. Those servos will take the role to
adjust the angle for the PID system (X moves in horizontal direction while Y moves in
vertical direction). We also connect a third servo Z to model our automatic barrier gate
(see section 5.2.4).

Figure 5.18: Hardware schematic for PID system

21
5.2.2. Constructing Mask Detection Program

Based on section 4.1, before constructing the mask detection program, we first create the
dataset and trained the mask detection classifier.

5.2.2.1. Create dataset

We will use Facial Landmark to detect face in a set of 1000 natural face images of human.

Facial landmark is a method allows us to automatically infer the location of facial structures,
including eyes, eyebrow, nose, mouth, and jawline. There are lots of applications of this method
such as: face alignment, head pose estimation, face swapping, blink detection and much more.
Detecting Facial Landmark has two step processes:

Step 1: Localize the face in the image as shown in Figure 5.4.

Figure 5.19: Localizing face in image

Step 2: Detect the key facial structures on the face ROI (Region of Interest).

Figure 5.20: Detecting ROI

22
After detecting ROI in each image (as shown in Figure 5.5), we spilt the set into two. 500 images
for with_mask data and 500 images for without_mask data. For without_mask data set, Facial
Landmark is applied again to attach a mask image for human face, just as mentioned in section
4.1.1.

5.2.2.2. Trained mask detection classifier

With the dataset prepared, we forward to train the mask classifier. Based on section 4.1.2, we
will fine-tuning MobileNet V2 architecture to train our classifier. This process is done with the
following steps:
1. We load the MobileNet V2 with pre-trained ImageNet weights, not including head
of the network.
2. We, then, construct a new FC head and append to the base in place of the old one
3. Finally, we will tune the head layer weights, leaving the base layers the same during
the backpropagation process

5.2.2.3. Applying face mask detection

According to section 4.1.4, we will develop a program that is able to do the following
task
1. Looping over frame in stream to find human face
1.1. We first grab a frame in stream and resize it.
1.2. Then, we apply Facial Landmark to detect face in the frame and apply mask
classifier to determine if they wear mask or not.
1.3. Return classifier result and location of the face frame.

2. Looping over detected face location inside the initial loop

.1. Labeling result on display as shown in Figure 5.6

Figure 5.21: Display result

23
5.2.3. Constructing Temperature Checkup Program
Based on section 4.2, we will build the program to achieve the following task:

5.2.3.1. Detecting forehead

Once, the system has detected the Mask, user is asked to remove the mask so that it can perform
Facial Landmarking. The system is using DLIB Module for Facial Landmarking mentioned in
section 5.2.2 to find the best spot on the forehead of the person to take temperature from.

5.2.3.2. Processing with PID control system

Then by using PID Control System with Servo Motors, the system tries to align the selected Spot
on the forehead by adjusting servo X and Y mentioned in Figure 5.3. In this part of project, we
use Facial Landmark to detect the forehead spot and face alignment. The facial landmark will
define all the spots on the face and select a spot on the forehead, which is the green dot in Figure
5.7. The PID control system tries to align the selected spot on the forehead to match with the
sensor spot or the users can self-align themselves through the display screen.

Figure 5.22: PID system result

5.2.3.3. Measure temperature & show result

Once aligned the system takes temperature reading using Contact IR Temperature Sensor. On the
display screen, there shall be a spot defined as the spot for the sensor to take the temperature,
which is the blue dot in the Figure 5.8.

Figure 5.23: Measuring temperature process

24
5.2.4. Constructing Auto-Operating Door
In this objective we will construct the door and then develop a program to automatic the door
based on the result form section 5.2.2 & 5.2.3

5.2.4.1. Constructing the door

We will model the door with a DC servo motor, more specifically, we will connect the
third servo to the schematic shown in Figure 5.3. The bar angle will act like the barrier
gate. We will develop a program to adjust the angle of the bar to open and close the door.

5.2.4.2. Programming the door

Figure 5.9 shows the operating condition for the door. When the mask detection program

Figure 5.24: Door operating condition

and temperature checkup program executed, they will return two conditions. If those
condition are met, the door will open.

5.2.5. Integration and Test

In this objective, we will integrate our programs to the hardware and test if they work properly
5.2.5.1 Test the mask detection program
In this test case, we want to test whether the system meets 2 requirements:
Firstly, we want to test whether the systems detect the face mask on a frame by applying the
face mask detection model on some pictures with or without mask. (Expected result as the
figure 5.10)

25
 Figure 5.25: Test the mask detection program

Then we will apply the model with a camera to test the real time and continuum face mask
detection of the system.

5.2.5.2. Test the temperature checkup program

In this test, we divide into 3 tests to check the temperature checkup part of the systems:
First, we will check the facial landmark of the system to see it could detect the face and
detect the forehead of the person. (In this figure 5.11, the systems test to see the green spot is
on the forehead)

Figure 5.26: Test the temperature checkup program

26
Then, we check the temperature checkup program without the pan and tilt module to see it
could take the temperature from a person by aligning their head to match the forehead spot
with the temperature spot on the display screen.
Finally, we test the completed temperature checkup program with the pan and tilt module to
see it could take the temperature of a person by self-alignment to match the temperature spot
with the forehead spot.
Both expected result of the last 2 tests should be like this figure 5.12:

Figure 5.27: Expectation results from testing

5.2.5.3. Test the auto-operating door

In this process case, we divide it into 3 tests:

Firstly, we want to check the door operate well without any conditions.

Secondly, we want to test the door operate well when there is one of the conditions are
met.For instance, the door will not open if the temperature is accepted but there is no mask
detected and vice versa.

Finally, we test the door to see it operate well when two conditions are met, the door must
open when the temperature is accepted, and the mask is detected.

After testing each small program, we will test again the whole system. Finally, we adjust,
fix, and complete the product.

27
 CHAPTER VI

EXPECTED RESULTS

28
 CHAPTER VII

CONCLUSION AND FUTURE WORK

29
REFERENCES
[1] A. Rosebrock, "covid-19-face-mask-detector-with-opencv-keras-tensorflow-and-deep-
learning," pyimagesearch, 4 May 2020. [Online].
[2] S. Mark , H. Andrew, Z. Menglong, Z. Audrey and C. Liang-Chieh, "MobileNetV2:
Inverted Residuals and Linear Bottlenecks," 21 Mar 2019. [Online].
z[3] S. Bhutani, "Touchfree automated temperature checkup and mask detection," hackster, 19
June 2020. [Online].
[4] Don Hun, "Automatic Door Opener using Arduino and PIR Sensor," electronichubs, 23
January 2018. [Online].

30
APPENDICES

31