Professional Documents
Culture Documents
2. System Model
RGB Camera The robot is designed in addition to
2.1. Sensor System the functions mentioned above, it also has the function
Encoder In mobile robotics, the encoder is of guiding the patient to move in the hospital.
used to measure the movement (direction and speed) Accordingly, the robot both moves and tracks the
of each of the wheels of the robot. Determining the patient's face to maintain a distance from the patient
position of the robot by this encoder is a popular during movement. If the patient stops or slows down,
method in the world called the Odometry method [8]– the robot will stop to wait, which is made possible by a
[10]. method that predicts the distance from the image
sensor to the person's face using the monocular
LIDAR sensor (Light Detection and Ranging) To camera [11].
increase the efficiency of positioning we use the Lidar
sensor. Lidar is a method for determining ranges Infrared thermal image sensor The temperature
(variable distance) by targeting an object with a laser sensor used in this research is Panasonic's AMG8833 –
and measuring the time for the reflected light to return Fig. 2. An infrared thermal image sensor with 64
to the receiver. In this research, the research team used thermal pixels distributed into a 2D 8x8 matrix, can
the lidar method to locate the robot in the hospital measure the temperature of an object in the range of
using the RPLIDAR A1 sensor. from 0°C to 80°C. Non-contact infrared sensor with
measuring distance up to 7m under ideal conditions. embedded firmware which permits to control the
Sensor measuring angle up to 60° in both horizontal motor by a PID algorithm.
and vertical. The sampling rate is 10 fps (frame per The sensory information network of the robot is
second). shown in Fig. 4, in which the computer is the central
control unit. Due to the characteristics of different
information rates between sensors (the information
here is point-to-multipoint), a special design for the
communication network in the robot has been
implemented. The RGB camera sensor is connected to
the control unit via a high-speed communication
channel.
(a) (b)
Fig. 2. (a) Infrared thermal image sensor – AMG8833
sensor Node, (b) Case for the AMG8833 sensor Node.
3. Proposed Method
3.1. Guidance the robot to approach the
patient
Positioning for robots using Extended Kalman filters
As shown in Fig. 6 the coordinate system and
parameter symbols of the mobile robot are designed to
move on the flat floor in the room, where (𝑋𝐺 , 𝑌𝐺 ) are
the axes of the global coordinate system of the room,
(𝑋𝑅 , 𝑌𝑅 ) are local coordinate system axes that match
the robot’s center.
• Correction phase: The filter corrects the a priori matched to the local map will be collected into vector
state estimate with 𝒛𝑘 measurements by the 𝒛𝑘 , used as input for the EKF editing step:
following measure update equations:
𝒛𝑘 = [𝑟1(𝑘) , 𝜓1(𝑘) , … , 𝑟𝑁(𝑘) , 𝜓𝑁(𝑘) ]𝑇 (20)
𝑲𝑘 = 𝑷− 𝑇 − 𝑇
𝑘 𝑯𝑘 (𝑯𝑘 𝑷𝑘 𝑯𝑘 + 𝑹𝑘 ) −1
(15)
b) Calculate the matrices R, H and V
̂−
̂𝑘 = 𝒙
𝒙 ̂−
𝑘 + 𝑲𝑘 [𝒛𝑘 − 𝒉(𝒙 𝑘 )] (16) From the coordinates and direction of the robot
estimated by the odometry method, the parameters 𝛽𝑗
𝑘-
𝑷𝑘 = (𝑰 − 𝑲𝑘 𝑯𝑘 )𝑷− (17)
and 𝜌𝑗 of the 𝑗𝑡ℎ line segment in the global map are
In this case we use an absolute measurement using converted to parameters 𝑟𝑖 and 𝜓𝑖 according to the
the RPLIDAR sensor. This sensor allows the detection local coordinate system of the robot. They are
of features of the environment around the robot, which calculated by:
are lines detected in the environment (for example, a
line that cuts across a flat wall in a room) 𝑟𝑖 |𝐶𝑗 |
[𝜓 ] = [ ]
𝑖 𝛽𝑗 − 𝜃 + 𝜋(−0,5𝑠𝑖𝑔𝑛(𝐶𝑗 ) + 0,5) (19)
𝐶𝑗 = 𝜌𝑗 − 𝑥 𝑐𝑜𝑠 𝛽𝑗 − 𝑦 𝑠𝑖𝑛 𝛽𝑗
where r and φ are the range and bearing from robot
position to obstacles respectively.
𝜓𝑖 and 𝑟𝑖 are the measurements that are directly
affected by the measurement noises of the RPLIDAR
sensor. Call these measurement disturbances 𝜀𝑟 and 𝜀𝜓
then the measured component vector from the laser
sensor when including the interferences will be:
𝑟𝑖 |𝐶𝑗 | 𝜀𝑟
[𝜓 ] = [ ] + [𝜀 ](22)
𝑖 𝛽𝑗 − 𝜃𝑟 + (−0,5𝑠𝑖𝑔𝑛(𝐶𝑗 ) + 0,5)𝜋 𝜓
The point with the smallest index would be saved as global map estimated by Odometry. Each local line
one end of the line segment. The slope m and intercept segment 𝐿𝑖 is compared with all transformed global
k of the line were computed using the formula: line segments 𝐺𝑇,𝑗 , and two lines segments are
(∑ 𝑥)(∑ 𝑦) considered as “matched” if their range and bearing to
∑ 𝑥𝑦−
𝑚= 𝑛
; 𝑘 = 𝑦̄ − 𝑚𝑥̄ (27) robot position are approximately the same and the
(∑ 𝑥)2
∑ 𝑥2−
𝑛 overlapping rate between them is less than previous
defined threshold [30].
where n is the total number of data points, x̅ and 𝑦̅ are
The range and bearing of global and local line
the mean of the 𝑥 − and 𝑦 − coordinates of the data
segments to robot position in robot’s coordinate are
points respectively. The algorithm would then be used
to find the remaining points of the cloud to add more presented as (𝜌𝐺,𝑗 , 𝜓𝐺,𝑗 ) and (𝜌𝐿,𝑗 , 𝜓𝐿,𝑗 ) respectively.
points to the line segment. Points that lie too far from
their previous neighbors in the array would be
ignored. The last point that met the distance threshold
was then treated as the other end of the found line
segment. The algorithm was repeated with the new
group until all elements of the collected point cloud
were checked.
Nevertheless, a fixed threshold 𝑑𝑖𝑠𝑇 did not work
for all scans. Experiment results show that in about
10% of the scans, the threshold of the distance from
one point to the best fit line should be larger in order
to detect lines from data points. However, if the
threshold applied for all scans was too large, extracted
line segments would be not smooth and could affect
the matching result. In this paper, a simple dynamic
Fig. 8. The range and bearing of global and local line The inequations below represent conditions for
segments to robot’s position in robot’s coordinate the matching local and transformed global line segments
environment in robot’s coordinate. Two lines segments 𝐿𝑖 and 𝐺𝑇,𝑗
are matched if all following conditions are met:
The overlapping rate between the local line
𝑂𝑘 (𝑎𝑘 , 𝑏𝑘 ) < 𝑇
segment 𝐿𝑖 and the transformed global line segment
𝑂𝑘 (𝑐𝑘 , 𝑑𝑘 ) < 𝑇
𝐺𝑇,𝑗 is defined as follows: (31)
(𝜌𝐿,𝑖 − 𝜌𝐺,𝑗 )2 < 𝑇𝜌
𝑂𝑘 (𝑎𝑘 , 𝑏𝑘 ) = |𝑎𝑘 + 𝑏𝑘 − 𝐺 𝑇,𝑗 | 2
{(𝜓𝐿,𝑖 − 𝜓𝐺,𝑗 ) < 𝑇𝜓
𝑂𝑘 (𝑐𝑘 , 𝑑𝑘 ) = |𝑐𝑘 + 𝑑𝑘 − 𝐺 𝑇,𝑗 | (30) where 𝑇, 𝑇𝜌 , 𝑇𝜓 are predefined threshold.
𝑘 = (𝑗 − 1)𝑛𝐿 + 𝑖 e) Estimation of line parameter’s covariances
𝑘 = 1,2, . . . , 𝑛𝐿 𝑛𝐺 − 1, 𝑛𝐿 𝑛𝐺 In order to compute the measurement covariance
where 𝐺𝑇,𝑗 is the length of transformed global line for the EKF, each extracted line segment could be
segment 𝑎𝑘 , 𝑏𝑘 , 𝑐𝑘 , and 𝑑𝑘 are Euclidean distances presented as (𝜌, 𝜓) parameters where ρ stands for the
between the end points of the line segments 𝐿𝑖 and 𝐺𝑇,𝑗 . perpendicular distance from robot’s position to the
line and ψ is the line orientation:
|−𝑘| 𝑘 −𝑚𝑘
𝜌= ; 𝜓 = 𝑎𝑟𝑐𝑡𝑎𝑛 2 ( , ) (32)
√𝑚2 +1 1+𝑚2 1+𝑚2
2 −𝑏 𝜎 2 +𝑐 𝜎 2
𝑎𝑖 𝜎𝑦𝑦 1 −𝑒 0 0 𝑣𝑎𝑟(𝑟𝑖 ) 0
𝑖 𝑥𝑦 𝑖 𝑥𝑥
𝑅𝑖 = ( 2 ) + (0
2 𝑐𝑜𝑠 2 𝜑 +𝜎 2 𝑠𝑖𝑛 2 𝜑 −2𝜎 2 𝑠𝑖𝑛 𝜑 𝑐𝑜𝑠 𝜑
𝜎𝑦𝑦 𝑖 𝑥𝑥 𝑖 𝑥𝑦 𝑖 𝑖) ≅[ ] (33)
(𝑎𝑖 −𝑐𝑖 )2 +𝑏𝑖 2 −𝑒 𝑒 0 𝑣𝑎𝑟(𝜓𝑖 )
𝑛
𝜋 1 1
where: 𝑒𝑖 = 𝑦̄ 𝑐𝑜𝑠 𝜓𝑖 − 𝑥̄ 𝑠𝑖𝑛 𝜓𝑖 ; 𝜑𝑖 = (𝜓𝑖 + ); 𝑥̄ = 𝑥𝑗 ; 𝑦̄ = 𝑦𝑗 ; 𝑎𝑖 = ∑(𝑥𝑗 − 𝑥̄ )2 ; 𝑏𝑖 = 2 ∑(𝑥𝑗 − 𝑥̄ )(𝑦𝑗 − 𝑦̄ );
𝑥̄ 𝑛 𝑛
𝑐𝑖 = ∑(𝑦𝑗 − 𝑦̄ )2 .
𝑣 𝑣𝑟 𝑐𝑜𝑠 𝑒3 + 𝑘1 𝑒1
[ ]=[ 𝑠𝑖𝑛 𝑒3 ] (38)
𝜔 𝜔𝑟 + 𝑘2 𝑣𝑟 𝑒2 + 𝑘3 𝑒3
𝑒3
a) Motion Goal
First, the robot follows the target direction while
the front does not find an obstacle. This movement
continues to be performed when reading the Lidar
sensor data gives the maximum distance result for the
angle corresponding to the direction from the robot to
the target. The end of this movement when the robot
senses an obstacle in the direction of the target.
After this point, the robot moves on to the second
part of the Motion to Goal behavior. In this behavior,
the robot detects discontinuities on the obstacle and
uses the data from the Lidar sensor to calculate the
distance 𝑑ℎ𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐 . If this distance is reduced compared
to the previous calculation, the robot will continue the
(a) (b) above behavior. If this distance is increased, the robot
Fig. 11. a) Calculating heuristic distance from Lidar’s data; will switch to Following boundary behavior.
b) Actual path of the robot when avoiding obstacles b) Following boundary
Tangent Bug uses a ℎ𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐 function to determine Following boundary behavior is invoked as the
the estimated distance from the current robot position 𝑑ℎ𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐 distance increases. The robot starts to move
to the target through the tangential discontinuity 𝑂𝑖 to in the direction of the last chosen discontinuity from
the obstacle: the direction towards the target. With the selected
direction, the robot moves on the tangent line. This
𝑑ℎ𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐 = 𝑑(𝑟𝑜𝑏𝑜𝑡, 𝑂𝑖 ) + 𝑑(𝑂𝑖 , 𝑔𝑜𝑎𝑙) (39) tangent line is perpendicular to the line connecting the
If there is no obstacle between the robot and the robot and the point closest to the robot on the obstacle.
target then: 𝑑ℎ𝑒𝑢𝑟𝑖𝑠𝑡𝑖𝑐 = 𝑑(𝑟𝑜𝑏𝑜𝑡, 𝑔𝑜𝑎𝑙). This behavior continues to execute when 𝑑𝑓𝑜𝑙𝑙𝑜𝑤𝑒𝑑 ≥
𝑑𝑟𝑒𝑎𝑐ℎ .
Where 𝑑𝑓𝑜𝑙𝑙𝑜𝑤𝑒𝑑 is the shortest distance between
the sensed boundary and goal; 𝑑𝑟𝑒𝑎𝑐ℎ is the shortest
distance between blocking obstacle and goal (or
distance to goal if no blocking obstacle visible).
Whenever 𝑑𝑟𝑒𝑎𝑐ℎ < 𝑑𝑓𝑜𝑙𝑙𝑜𝑤𝑒𝑑 , the robot will switch
back to Motion to Goal behavior.
The algorithm diagram is shown in Fig. 15. Here, a
clear area flag is created to toggle between the two
behaviors. Initially, this flag is active and therefore
Motion to Goal will be applied. If the flag is disabled, (a) (b)
the robot will switch to Following boundary Fig. 13. (a) Voice recognition model; (b) Flowchart of the
behavior. Finally, the robot will stop when it reaches passenger answering program algorithm
the target
3.2. Screening In the final phase, the text data converted from the
passenger's voice will be classified by the model built
Voice interaction with the patient The voice
in phase 1 to determine its label. From there, the
interaction function consists of 3 phases. In the first
computer program on the robot will respond correctly
phase, we build a text classification model by machine
with the necessary information to answer the
learning using the fasttext library [32], an open-
passenger as shown in the flowchart of Fig. 13(b).
source library that allows users to learn text
As shown in Fig. 13(b), in case the program does
representations and text classifiers to classify text
not determine which label the text belongs to, this text
strings. Accordingly, we build a keyword data and label
will be stored in a log file for us to check and add to the
the keyword combinations as follows:
machine learning data.
Tab. 1. Keyword and label datasheet
Measure temperature, blood oxygen concentration,
No. Keyword Tag
sterilization After the patient puts his hand on the
1 hello, hello robot, hi, etc. __label__#greeting#
2 Do you have a fever? __label__#fever# front of the robot, the Spo2 readings and temperature
3 Have you lost your sense __label__#taste# will be checked. In addition, the hand sanitizer spray
of taste? system is also activated after the measurement is
4 Put your finger on the __label__#spo2m# completed.
SpO2 meter. First of all, blood oxygen saturation (SpO2) was
5 Place your palm upside __label__#sterilization# calculated using a formula supplied in the Maxim
down in front of the Integrated™ sample code [33]. The AC and DC
nozzle. component of the pulsative waveform was calculated
6 bye bye, goodbye, thank __label__#goodbye# for both the red and infrared channels and stored in
you … etc integer variables (𝐴𝐶𝑅𝑒𝑑 , 𝐴𝐶𝐼𝑅 , 𝐷𝐶𝑅𝑒𝑑 , 𝐷𝐶𝐼𝑅 ) as a mean
… … … of 5 consecutive peaks/valleys. A ratio (R) of the AC
and DC was then calculated from the mean AC and DC
Initially, the robot greeting "Hello" will guide the values using (40), and the SpO2 value was calculated
patient to put his hand in the measuring position to by the (41):
check the parameters, etc. 𝐴𝐶𝑅𝑒𝑑 ÷𝐷𝐶𝑅𝑒𝑑
According to Table 1, there are more than 100 𝑅= (40)
𝐴𝐶𝐼𝑅 ÷𝐷𝐶𝐼𝑅
labels where each label represents a type of
information provided to the user depending on the 𝑆𝑝𝑂2 = −45,060𝑅2 + 30,345𝑅 + 94,845 (41)
user's question. Next, the AMG8833 sensor measures the
In the second phase, the user's voice is recorded in temperature of an object based on the thermal
real time from the microphone, then using API radiation emitted by that object. The Stefan-Boltzmann
functions to transmit to Google's Speech-to-Text API law describes the power emitted by an object using
audio processing system. The text string will be (42):
returned immediately after and put into the Text
Classifier as shown in Fig. 13(a). 𝑃 = 𝜀𝐴𝜎𝑇 4 (42)
where P is the power radiated from an object; T is the
temperature of the object; A is surface area; σ is Stefan–
Boltzmann constant, ε is the emissivity of the object (it
gets values from 0 to < 1 for objects which do not
absorb all incident radiation - grey object; If it is an
absolute black object then equal 1).
Applying Moghaddam's formula [33], we calculate The robot moves along the rows of seats, to the
the temperature of a grey body as follows: seats in turn, then returns to the starting position and
repeats the cycle.
𝑉𝑜𝑢𝑡 = 𝑘(𝑇𝑜4 − 𝑇𝑠4 ) (43)
When moving through the seats, if the robot uses
where 𝑉𝑜𝑢𝑡 is the output voltage of the sensor, 𝑇0 is the the camera to check if there is someone sitting on the
surface temperature of the object being measured, 𝑇𝑠 is chair, if there is, the robot will stop, if not, it will
the temperature of the sensor's thermistor, k is the continue to the next seats.
empirical constant representing the parameters A, ε, σ At the chair positions with patients, the robot will
as well as the electronic noise that may exist during the turn to the patient, greet, then conduct screening
measurement (calibrating the sensors). including temperature measurement, SpO2, then hand
Based on (43), the surface temperature of the sanitizer spray.
object will be calculated by the formula (44): If the measurement results are normal, the robot
will guide the patient into the clinic, if the result is
𝑇𝑜 = (𝑉𝑜𝑢𝑡 /𝑘 + 𝑇𝑠4 )1/4 (44) abnormal, the robot will guide the patient into the
isolation room.
Actual movement results are shown in Fig. 15. As
4. EXPERIMENTAL RESULTS we can see, the robot only stops at 3 seats with people
4.1. The robot moves to the patient's location sitting, then conducts screening.
During the movement, the robot uses Lidar sensors
In the case of no obstructions Set up the screening to locate. The data from the Lidar will be matched with
clinic environment as shown in Fig. 14(a). Arrange two the clinic's map, where the map is already set up.
rows of patient seats 2m apart, the seats in each row In Fig. 16 we see the data from the lidar is
are also 2m apart, for a total of 8 seats. represented by the green dot, the red lines are the lines
that are matched to the walls of the map.
(a)
In the case of obstructions When there is an • Next, the robot will ask the patient to put their
obstacle in the path, the robot uses the Tangent bug finger on the SpO2 measuring sensor in front of the
algorithm with data from the Lidar to avoid the robot as shown in Figure 1a, the AMG8833 sensor
obstacle, then follows the original trajectory. placed on the screen reads the temperature data of
Fig. 17 show the results of obstacle avoidance when the correct patient. Each patient will have 3 data
a person blocks the robot's path samples read after 5 seconds and the data will be
sent to the central computer.
• After collecting the results, the robot will analyze
the data and predict the patient's status
(Measurement results are shown in Table 2).
Patients will have their hands disinfected from the
robot's nozzle and be directed to the patient to go
to the next clinic.