2019 IEEE Eurasia Conference on IOT, Communication and Engineering

Development of an AI-enabled AGV with robot manipulator

Jin-Siang Shaw¹,2*, Chuin Jiat Liew2, Sheng-Xiang, Xu2 and Zhe-Ming, Zhang2

¹ Research Center of Energy Conservation for New Generation of Residential, Commercial, and Industrial Sectors
2
Institute of Mechatronic Engineering
National Taipei University of Technology
Taipei 10608, Taiwan
*
jshaw@ntut.edu.tw

Abstract
The presented work describes a ROS-based automated guided
vehicle (AGV) applied in usage of an assembly production
line. By reaching the purpose of automatic navigation, two
sets of LIDAR sensors are installed to approach simultaneous
localization and mapping (SLAM). A RGBD camera is used
to perceive the working environment and identify workers
captured in sight, through a trained AI deep learning using
YOLO 3. A robot manipulator is attached on AGV to perform
pick-and-place task. The developed AGV has shown to
perform well in SLAM, automatic navigation, obstacle
avoidance in typical indoor environment.
Keywords: automated guided vehicle, artificial intelligence,
robot operation system, SLAM
Introduction
In the 1990s, the first generation of AGVs was guided by
magnetic strips and only move along magnetic strips. The
following generation of laser guidance mode was developed.
In this circumstances, the AGV only works as an autonomous
cart with low moving speed and insufficient battery. Since
2017, because of the emergence of the Amazon KIVA car,
coupled with the development of China's e-commerce volume,
the network computing technology is improved. The
optoelectronics industry and the electronics business have
also begun to ap
finder to establish the map [1] and localizing, and with the
route planning algorithm global path planning method [2],
through the known map environment, all the path is simulated
once, and the highest scoring path is selected by the scoring
formula to travel to the destination. This research suggests an
approach to develop an unmanned autonomous cart with
artificial intelligence by low cost, which can effectively and
accurately establish maps and navigation, and applied in
industrial production lines.
Design and implementation of AGV
This section starts with an overview of the AGV built,
describing hardware and software structure.
A. Hardware Structure
The AGV presented in this research is built with
30mm*30mm aluminum stretch. A laptop will be embedded
on the AGV to operate ROS system. An Arduino Mega 2560
is connected via USB as microcontroller for DC motors, two
sets of LIDAR sensors too are connected via USB, installed at
the front left and rear right to secure a 360 degree of detection,
receiving laser data to facilitate the AGV to establish maps
and localizing. A RGBD Kinect camera is used to gain point
cloud information, for system image recognition and deep
learning. Lastly, a set of 6 DOF Niryo robot manipulator is
connected via Ethernet cable, applied on the top layer of
AGV to carry out command given through TCP/IP from ROS


  user interface. Figure 1 shows the AGV built and will be
In the early stage of developing deep learning was being
handicapped due to the insufficient of computing speed so as
the internal storage, and the amount of data. In 2012,
Professor Geoffrey Hinton from the University of Toronto use
the deep learning algorithm AlexNet and perform calculations
on the GPU. They won the championship in the ImageNet
image recognition competition held every year at Stanford
University. Not only did the error rate of 6.42% beat the
26.22% of the second place. In results, GPU training become
main stream around the world and used various results to
invest in various industries.
Today, there are many kinds of industrial AGVs on the
market; for example, it is produced by Solomon. The
trackless automatic truck MiR100, and the MR100 trackless
AGV produced by Green Jet Company, although they are all
trackless AGV series, yet have no possessed artificial
intelligence technology. If the AGV is applied to any known
or unknown situation, it is mainly discussed in the following
topics: localizing, mapping and navigation, etc.
Commonly speaking, most AGVs are using the laser range
tested in the research.
B. Software Structure
The software architecture used includes ROS, YOLOv3, and
Arduino are all built under Ubuntu16.04. ROS is mainly used
for navigation and control of AGV. YOLOv3 can achieve a
real-time performance on GPU, using a variety of training
photos to train its own detectors with migration learning.
Through obtaining images from Kinect camera, YOLOv3
transmits the image recognition results to ROS for obstacle
avoidance. Arduino is used for IO input and output, such as
signal input of ultrasonic sensor, signal control of motor
control signal and relay signal control. Finally, by integrating
YOLOv3 and Arduino under ROS system, navigation and
obstacle avoidance are ready.
On the other hand, the Niryo robot manipulator is integrated
with another vision system in order to execute more precise
pick-and-place tasks. Under ROS architecture, we are able to
perform an auto navigation system on AGV with AI
avoidance while a robot manipulator can perform pick-and-

284 ISBN: 978-1-7281-2501-5

Authorized licensed use limited to: Auckland University of Technology. Downloaded on May 28,2020 at 05:09:29 UTC from IEEE Xplore. Restrictions apply.
 2019 IEEE Eurasia Conference on IOT, Communication and Engineering
place tasks in a production line.
Presented Work
A. Niryo Robot Manipulator with Vision
Fig. 1 Design and implementation of AGV
A traditional definition for an industrial robot manipulator is
located at a fixed site in the production line, operating
preloaded commands to fulfill a desired work flow. Since we
are having a robot manipulator which is currently attached
with a moving AGV, an issue is needed to be tackled. As the
AGV has moved to a requested position, assume that the
robot manipulator is going to pick up a targeted object on a
working platform, a vision system is necessary in detecting
and relocating the grapping point [3].
First, a CCD is embedded among the gripper, as the end
effector of Niryo robot, in order to gain vision information
relating gripper and targeted objects. Then after we have done
calibrating the camera, a few more image processing method
are required to get a center point and rotated angle of target
object. It transforms relative robot manipulator world
coordinate due to the position of object is related to the vision
system coordinate. Additionally, the vision system
communicates with Niryo robot using ROS publisher and
subscriber methods, the object position will be conducted to
 ) = p(: |: , :
 )p(|: , : )
execute pick-and-place task, finishing an eye-in-hand
architecture in ROS. Figure 2 shows a CCD embedded on
gripper of Niryo robot.
B. Deep Learning
YOLO (you only see once: unified, real-time object
detection) is a target detection system based on a single neural
convolutional neural network to identify the target. YOLOv3
uses only the convolution layer to make it Full Convolutional
Network (FCN), which has 53 convolutional layers skipping
sampling layers, and without the maxpool layer, YOLOv3
directly splits the original input image into small
segmentation that do not coincide with each other, and then
uses spatial invariance of convolution operations, the final
output of the grid form fmps, each cell of the grid corresponds
to a region of the original image; finally use each cell to
predict those center points in the cell target (GT).
In this study, 10,000 photos of training photos were
downloaded from ImgNet, of which 7,000 photos were used
as training materials and 3,000 photos were used as test
materials. The number of training sessions was set to 250,000
times, which was brought into neural network training.
C. SLAM - Map Building and Navigation
In this study, Gmapping algorithm is used to establish maps
[7]. Multiple sensors are applied in the system, and Gmapping
algorithm obtain better results for indoor environment [8].
Gmapping algorithm is based on RBPF (Rao-Blackwellized
Particle Filter) to get accurate maps through particle filters.
The traditional RBPF uses the known initial posture of the
AGV,  , as the map environment data collected by the
sensor,  . When applied to the SLAM, the map
environment information can be represented by the Markov
process as : =  ,  , … ,  , and the AGV motion
information as : =  ,  , … ,  , thereby estimating the
motion trajectory and The check rate, and according to the
Bell's theorem [9], the posterior probability of the SLAM at
time t can be expressed as the recursive function of the
posterior probability, the environmental state and the motion
state at time t-1, refer to (1).
p(: , |: , :
(1)
After the map is established, it needs to pass through the
feedback signals of the sensor and the map to know the
current location, in order to calculate the correspondence
between the target position and the current position. This
study uses AMCL (Adaptive Monte Carlo localization) to
update particles. The positioning method is to reverse the
known map environment and sensor information to calculate
the current location. For the AGV state,  consists of (x, y,















 
AGV pose, and the set of particles for the dispersion can be
expressed as (2).
Fig. 2 CCD embedded on gripper of Niryo robot. [] [] [
] []
 =  ,  , … ,   =   , ( = 1,2,3, … , ) (2)
network proposed by Redmon and Farhadi in 2015 [4]. At
the 2017 CVPR, Redmon and Farhadi released the YOLO V2,
When AGV determines its position through sensor
which further improved the accuracy and speed of detection
information, it can start using map to navigate function. In
[5]. Since the V1 and V2 versions have great difficulty in
this case, the odometer is an important navigation basis,
identifying small objects, Redmon and Farhadi updated the
knowing the AGV travel distance in the map. It can also
new generation YOLO V3 on the 2018 CVPR. The design of
calculate the walking path through the navigation function
the series improved, using the new model to improve
and avoid obstacles on the map while maintaining a safe
performance and speed to identify smaller objects. Compared
distance. The navigation planning speed is constantly updated.
with SSD, the speed is 3 times higher, and the speed is up to
However, there are many dynamic objects in the actual
3.8 times compared with RetinaNet [6].
environment, and the navigation does not only rely on the
YOLOv3 extracts features from the input image through
previous map anymore, it is capable to update the
the feature extraction network. It uses the features of deep

ISBN: 978-1-7281-2501-5 285

Authorized licensed use limited to: Auckland University of Technology. Downloaded on May 28,2020 at 05:09:29 UTC from IEEE Xplore. Restrictions apply.
 2019 IEEE Eurasia Conference on IOT, Communication and Engineering

Fig. 3 Framework of ROS Navigation.

environmental information by the laser sensor in real-time to shows a 3D map built and presented on laptop ROS interface.
update the following route. The navigation function
architecture is shown as Figure 3 below.

Results and Discussion

A. Eye-in-hand Test
In this experiment, a simple block is set as the target object
for the vision system. Figure 4 shows a bounding box
successfully capture the target and along the upper right

Fig. 5 3D map built results

Fig. 4 Eye-in-hand vision detection results

corner has stated information as the center coordinate and a
rotation angle (theta) of the block, and current coordination
and rotation angle of gripper.
The test of detecting the same target was repeated with
different placement and also variant of angle. Thus the
gripper can successfully grab the target and released at
requested position.
B. Mapping Test
Since we use Gmapping algorithm and particle swarm
optimization to build the map, this experiment needs to
modify the parameters to see the accuracy of map creation. In
this process, the AGV is controlled remotely via TCP/IP to
eliminate dynamic obstacle factors in the field. Figure 5
Fig. 6 Actual photo of experimental field
Figure 6 shows actual photo of the field.
C. Navigation Test
In this experiment, we will use the established map to carry
out the navigation experiment, and repeat the navigation
through the same target point, so as to test the stability of the
AGV. The results of the tests will be shown as Figure 7 to 9
below which is the actual print screen from ROS interface on
laptop.

286 ISBN: 978-1-7281-2501-5

Authorized licensed use limited to: Auckland University of Technology. Downloaded on May 28,2020 at 05:09:29 UTC from IEEE Xplore. Restrictions apply.
 2019 IEEE Eurasia Conference on IOT, Communication and Engineering
Fig. 7 Step 1 for Navigation: Give target position (Green arrow).
Fig. 8 Step 2 for Navigation: Moving toward target and avoiding
obstacles (Red path).
Fig. 9 Step 3 for Navigation: Reaching target position.
D. AI Obstacles Avoidance Test
In test we assume a possible scenario that a random person
might interrupt into the auto simulated navigation path and
the AGV should have instantly identify the dynamic objects
and stop moving. Figure 10 shows a real-time detection result
on screen with a bounding box remarked the detected object.
Fig. 10 AI detection results.
Conclusion
In this paper, we introduced an AI-enabled AGV based on
ROS system to perform SLAM and auto navigation, also
through deep learning, the AGV can make avoidance in the
ISBN: 978-1-7281-2501-5
Authorized licensed use limited to: Auckland University of Technology. Downloaded on May 28,2020 at 05:09:29 UTC from IEEE Xplore. Restrictions apply.
path of navigation. The AGV is concluded in following
points:
x AGV is having the ability to build indoor map.
x AGV is capable of auto navigation.
x AGV can fulfill avoidance task with dynamic
obstacles.
The AI obstacles avoidance test was repeated 500 times and
according to the statistical count, the accuracy rate is among
95%. Although the avoidance test has proven that with deep
learning which is significantly raise the detection ability
comparing to normal machine vision, a few circumstances
still worth discussion as some indistinct feature might still fail
the detection system.
In addition, the pick-and-place test was also repeated in
several times until we manage to debugged the hardware
problem due to some deformation might occurred when the
end effector was trying to reach a longer distance, because the
CCD embedded was an unexpected load for the designed
robot manipulator. Hence we have recalculated the
transformation relation between flange and the elongated
gripper.
Acknowledgments
This paper is financially supported by the “Research Center
of Energy Conservation for New Generation of Residential,
Commercial, and Industrial Sectors” from The Featured Areas
Research Center Program within the framework of the Higher
Education Sprout Project by the Ministry of Education
(MOE) in Taiwan.
References
[1]ġ CC. Peng, YT. Wang and CL. Chen, “LIDAR based scan
matching for indoor localization”, 2017 IEEE/SICE
International Symposium on System Integration (SII), 11-14
Dec. 2017.,
[2] F.  !. Babineca, M. Kajana, et al., “Path Planning with
Modified a Star Algorithm for a Mobile Robot”, Procedia
Engineering 96, 2014, pp 59 – 69.
[3] J.S. Shaw, and W.L. Chi, “Automatic Classification of Moving
Objects on an Unknown Speed Production Line with an Eye-In-
Hand Robot Manipulator,” Journal of Marine Science and
Technology, Vol. 26, No. 3, pp. 387-396, 2018.
[4] J. Redmon, A. Farhadi, “You Only Look Once: Unified, Real-
Time Object Detection”, Computer Vision and Pattern
Recognition (CVPR), 2015
[5] J. Redmon, A. Farhadi, “YOLO9000: Better, Faster, Stronger”,
Computer Vision and Pattern Recognition (CVPR), 2016
[6] J. Redmon, A. Farhadi, “YOLOv3: An Incremental
Improvement”, Computer Vision and Pattern Recognition
(CVPR), 2018
[7] Y.C. Huang, “Development of Automated Guided Vehicle
under ROS Architecture”, National Taipei University of
Technology, Taipei, Taiwan, 2018
[8] J. P. Machado dos Santos, “SmokeNav - Simultaneous
Localization and Mapping in Reduced Visibility Scenarios”,
September 2013.
[9] S.X. Xu居Z.K. Yang居C.L. Chang, “Design and implementation
of an autonomous garbage packing robot” ɾProceedings of the
8th International Symposium on Machinery and Mechatronics
for Agriculture and Biosystems Engineeringɾ Niigata, Japanɾ
2016
287