Original Article

Structural Health Monitoring

2022, Vol. 21(3) 7 88­–802

A novel intelligent inspection  The Author(s) 2021

Article reuse guidelines:

robot with deep stereo vision for sagepub.com/journals-permissions

DOI: 10.1177/14759217211010238
journals.sagepub.com/home/shm
three-dimensional concrete
damage detection and quantification

Cheng Yuan, Bing Xiong, Xiuquan Li, Xiaohan Sang and Qingzhao Kong

Abstract
Crack assessment of reinforced concrete structures using stereo cameras is a potential way for increasing the efficiency
and safety of infrastructure maintenance routines. However, existing damage methods for reinforced concrete structures
are based on the segmentation of two-dimensional planes without consideration to the actual size of concrete damage.
Furthermore, on-site structural monitoring requires the installation of a large number of contact-based sensing devices,
resulting in the potentially excessive consumption of time and financial resources. Therefore, a new vision-based damage
assessment method for reinforced concrete structures using a novel intelligent inspection robot with Internet of things–
enabled data communication system is proposed in this article. In the first part of this article, the data acquisition system
of the inspection robot and the algorithm for three-dimensional structural reconstruction using a stereo camera is dis-
cussed. The discussion is followed by a description of the method for crack quantification based on a new proposed
deep-learning technique. Finally, to accomplish damage localization, the quantified concrete damage with actual size infor-
mation is projected onto a three-dimensional surface point cloud reconstruction of the inspected structure. To verify
the proposed method, a reinforced concrete column that has undergone cyclic loading failure is used as an inspection
subject. The validation experiment demonstrated the ability of the proposed system to segment, localize, and quantify
the damage in three-dimensional space with high accuracy.

Keywords
Structural health monitoring, deep learning, stereo vision, three-dimensional scene reconstruction, concrete damage,
volume quantification

Introduction Traditional facade damage detection relies on visual

Reinforced concrete structures during their service life
can often be subjected to fatigue stresses or cyclic load-
ing that initiates microscopic cracks on the surface
which can grow or coalesce into larger cracks.1–3
Cracking can degrade structural stiffness and generate
material discontinuities.4,5 Potential failures due to
excessive cracking can be prevented by the diligent
implementation of early inspection and monitoring.6–8
In general, two approaches are available for detecting
cracks in concrete, such as contact-based detection and
non-contact inspection.9–13 Although vibration-based
structural damage detection approaches have been
widely applied in recent years, such methods are not
suitable for crack detection due to the fact that they are
not sensitive to local damage.14
inspection and sensor-based structural health monitor-
ing techniques, which mainly depends on the judgement
of trained inspectors and the proper implementation of
sensors or other data acquisition equipment, respec-
tively.15–17 Conventional sensors, such as strain gauges,
accelerometers, and linear variable differential transfor-
mers (LVDTs) are widely used to assess structural
integrity.18–22 For example, the smart-aggregate (SA)
Department of Disaster Mitigation for Structures, Tongji University,
Shanghai, China
Corresponding author:
Qingzhao Kong, Department of Disaster Mitigation for Structures, Tongji
University, Shanghai 200092, China.
Email: qkong@tongji.edu.cn
Yuan
2 et al. 789
method can monitor concrete cracks in real time with
damage index.23–25 However, the application of sensor-
based acquisition networks can be prohibitively time-
consuming and labour-intensive, and long-term moni-
toring may prove impractical due to the need to store
massive amounts of data.26–30
Machine learning in computer vision demonstrated
rapidity and reliability for conducting image-based
inspection of the concrete surface.8,31–36 Deep learning
with robustness against noise disturbance has been
applied to accurately interpret images and sensing data
for crack detection.8,33,37–41 It has demonstrated that
automated damage detection, automated evaluation of
the local damage and safety of the global structure, and
automated data collection using robots belong to the
category of automate vision-based structural inspec-
tion.42 Vision-based inspection devices can be carried
by robotic vehicles that can be remotely controlled from
a distance such as for accessing hard to reach places.43–
46
Oftentimes, deep-learning techniques can benefit
from the inclusion of image pre-processing using estab-
lished two-dimensional (2D) image processing methods
used traditionally for conventional damage detection
and quantification.47 Common processing methods
include edge characterization and threshold segmenta-
tion with advanced filtering.48 Chen et al.49 conducted
corrosion detection of steel structures using support
vector machines (SVMs). A convolutional neural net-
work (CNN)–based crack detection approach was pro-
posed by Cha et al.,50 and experimental results found
that CNNs were able to effectively detect structural
damage.
However, existing deep-learning–based damage
detection approaches cannot automatically quantify
crack length, width, or volume, and on-site damage
assessment requires a large amount of contact sensing
devices.6 In addition, since the integrated simultaneous
localization and mapping (SLAM) requires a large
amount of computing power, existing three-
dimensional (3D) reconstruction and damage detection
methods based on deep learning are not simulta-
neous.51,52 Such an issue is compounded when assessing
large-scale structures and infrastructures. Therefore, a
new and non-contact structural health inspection sys-
tem for reinforced concrete (RC) structures is proposed
in this article to overcome challenges regarding the
practical implementation of deep learning for damage
assessment. The proposed approach expands the quan-
tification of concrete damage from 2D plane to 3D
space. As a vehicle to deliver inspection, a novel inspec-
tion robot with stereo camera was designed and utilized
to perform quasi-real-time damage assessment of RC
structures. The onboard system is linked to quasi-real-
time cloud computing and data visualization algo-
rithms using the Internet of things (IoT). The entire
Structural Health Monitoring 00(0)
inspection system was demonstrated experimentally on
an RC column.
This study is organized as follows. Section
‘‘Methodology’’ details the framework of the proposed
methodology. Following the overall procedure, section
‘‘Design of the novel inspection system’’ shows the
development of the novel inspection system, with the
details of the hardware design and software processing
framework. Following the design of the proposed sys-
tem, section ‘‘Crack detection and quantification’’
explains the crack detection and quantification, describ-
ing the data set generation, deep-learning network used
to segment and detect cracks in 2D plane, proposed
quantification of concrete damage in 3D space coordi-
nates, and estimation of crack volume. Subsequently,
section ‘‘Three-dimensional scene reconstruction’’
describes the approach to reconstruct the real concrete
structure. Finally, the experimental validation and error
analysis are conducted using a lab-scale RC column
with real cracks in section ‘‘Application,’’ describing
the inspection robot and experimental setup, the pro-
posed crack assessment of the RC structure, error anal-
ysis, and the validation results. A brief conclusion is
detailed in the final section ‘‘Conclusion.’’
Methodology
The data pipeline of this study is detailed in Figure 1.
The architecture of the proposed novel inspection sys-
tem has three components, including data acquisition
using stereo camera mounted on the inspection robot,
3D structural reconstruction, and concrete damage
quantification and projection. The main objective of
the inspection task is to automatically detect, localize,
and quantify the crack volume in an RC structure. To
accomplish the objective, a deep-learning–based con-
crete segmentation module is first proposed to localize
and segment concrete damage. To acquire detailed
crack images and point cloud data of 3D representation
of the concrete crack, a novel infinite distance control
inspection robot was designed and tested. The inspec-
tion robot is equipped with a monocular camera, a
stereo camera, a Lidar sensor, an edge computing sys-
tem, and a quasi-real-time data transmission system.
RGB imaging data captured by the robot is routed to a
deep-learning network to segment and localize cracks.
Through visual mapping, 3D convex hull instantiation,
and 3D scene reconstruction, the segmented cracks can
be quantified.
Following validation and testing of the deep-learn-
ing–based 3D damage volume detection model, the
model is placed on an online server to process image
data collected and uploaded by the inspection robot.
The deployed server serves two functions: (1)
790 et al.
Yuan
Figure 1. Flow chart of the proposed 3D concrete damage volume quantification and damage projection.
implement spatial mapping and 3D damage assessment
and (2) visualize the 3D reconstruction model obtained
through edge computing using the Jetson NX
Developer Kit. Finally, damage projection onto the 3D
point cloud surface was conducted to realize the dam-
age localization.
Design of the novel inspection system
Design of the inspection robot
The designed mobile robot includes a control
terminal and edge computing terminal, both of which
serve as the basis of the inspection system, as shown in
Figure 2. The robot is designed to carry a variety of
Structural Health Monitoring 21(3)
3
sensors to gather required inspection data, which is
called MSIL in this study, where M refers to a monocu-
lar camera, S stands for a stereo camera, I represents
the inertial measurement unit (IMU), and L is the light
detection ranging (Lidar) sensor. The monocular cam-
era mounted at the forefront of the robot is used as the
visual sensor of the control board. The stereo camera
(i.e. ZED 2) developed by STEREOLABS with built-
in IMU, Barometer, and Magnetometer sensors is used
to conduct spatial object detection. The IMU provides
6-degree-of-freedom (6-DOF) information, in which
each image can be directly mapped for panoramic
image stitching. Lidar sensor is employed to measure
the distance between the RC structure and the camera.
Consequently, the space and position between the 3D
Yuan
4 et al. 791
Structural Health Monitoring 00(0)

Figure 2. The components of the inspection robot (a) control terminal, (b) data acquisition terminal, and (c) robot internal
structure.

reconstructed structure with local crack and the acqui-
sition system can be tagged under the cooperation
between these three types of sensors. Following the
damage segmentation extracted from a deep-learning–
based model, the obtained damage can be directly pro-
jected on to a global coordinate based on the MSIL.
Since the Raspberry Pi enables control of electronic
components for physical computing and interfacing
through the IoT, the Raspberry Pi served as the main
control port with the remote-control system. A Jetson
NX (Nvidia) which was used to implement embedded
edge computing for artificial intelligence (AI)–based
image processing and manipulation of point cloud
data. Movement of the robot is accomplished
through four-wheel drive differential steering, which
allows in situ rotation of the robot for scene recon-
struction. The STM-32 chip, which belongs to a
family of 32-bit microcontroller integrated circuits by
STMicroelectronics, was used to control robot move-
ment. Jetson NX and the motion control board com-
municate directly through a serial port. Commands for
motions are transmitted to the Jetson NX with a rela-
tively low delay through a WebSocket protocol. The
Jetson NX encodes the instructions and sends them to
792 et al.
Yuan
Figure 3. Pipeline of the data transmission and processing system.
the motion control board. In addition to unlimited dis-
tance control, the robot can also be controlled using a
manual remote controller.
Data transmission and processing system
The robustness and performance of the data acquisi-
tion and transmission system is critical in object detec-
tion due to potential issues and bottlenecks in real-time
data transmission and processing.53 Numerous studies
used offline detection methods in which data are first
acquired and later exported to a computer for post-
processing. This post-processing method reduces the
monitoring efficiency.43 To enhance the monitoring
efficiency, by improving the quasi-real-time processing
and transmission capabilities of data, this article
proposes a partially automated monitoring method.
Figure 3 illustrates the pipeline of the data transmission
and processing system. Following the data acquisition
from the inspection robot using the MSIL, a 4G mod-
ule sends the data to the cloud computing system. An
HTTP protocol transmits reconstructed point cloud
data and captured binocular images, and then pro-
cessed by graphics processing unit (GPU) cluster to
complete the 3D damage detection and quantification.
Following the edge computing of the acquired data on
the Jetson NX, WebRTC and WebSocket protocols
provide low-latency transmission of images and traffic
of remote-control commands, respectively. To facilitate
client control and visualization, a specially designed
web app that includes a graphical user interface (GUI)
was installed on the laptop and a smartphone, as
shown in Figure 4.
Crack detection and quantification
Two types of image databases were set up for storing
images for object detection, damage segmentation,
Structural Health Monitoring 21(3)
5
damage quantification, and 3D reconstruction. The
first database consists primarily of RGB images of con-
crete damage obtained from online open-source data-
bases for 2D object detection and segmentation. The
second database contained point cloud data derived
from images acquired by the stereo camera, and is
mainly used for spatial mapping, damage quantifica-
tion, and damage localization in 3D space.
Image acquisition for 3D concrete damage detection
A total of 400 RGB images of cracks in concrete were
collected from an open-source library to build up the
training data set, as shown in Figure 5(a). The stereo
camera was utilized as a depth sensor to prepare the
testing data set for damage segmentation and quantifi-
cation, as shown in Figure 5(b). The collected RGB
images were then labelled manually using LabelImg for
use in the deep-learning framework. The stereo camera
used in this study is ZED 2, which allows remote
monitoring and recording of 1080p high-definition
(HD) videos at 30 FPS or Wide Video Graphics Array
(WVGA) at 100 FPS. Wide-angle video depth can be
captured with up to 120 field of view.
Crack detection and segmentation
Mask region convolutional neural network (R-CNN)54
is a state-of-the-art method for object detection, and
extends the target detection abilities of the Faster R-
CNN framework by adding an extra branch to achieve
instance segmentation for each output proposal box
using a fully connected (FC) layer. The three stages of
the framework are shown in Figure 6. First, feature
maps are extracted from the input images by the back-
bone network. Second, the generated feature maps are
fed to the region proposal network (RPN) to generate
regions of interest (ROIs). Finally, the generated ROIs
are mapped to extract the corresponding target features
Yuan
6 et al. 793
Structural Health Monitoring 00(0)

Figure 4. GUI of the IoT-enabled inspection system (a) PC platform and (b) smartphone platform.

in the shared feature maps. The ROIs are thereafter

output to the FC and the fully convolutional network
(FCN) for object detection and instance segmentation.
The corresponding bounding boxes and segmentation
masks are generated in this process.
As deeper networks can result in higher accuracy,
ResNet is used as the backbone network for feature
extraction. Image feature extraction is based on shared
convolution layers. Low-level features (i.e. edges and
angles) are extracted by the underlying network. High-
level features that represent target categories are Figure 5. Image acquisition for training of damage detection
extracted at the higher level. To better represent the model. (a) Open-source dataset; (b) Stereo vision dataset.
crack target on multiple scales, a feature pyramid net-
work (FPN) was employed to extend the backbone net-
work. The FPN improves feature extraction using a multiple objects across different scales. The ROI can be
second pyramid that routes high-level features from the generated by sliding the feature maps across the images
initial pyramid towards the lower layers. outputted from the anchors. The trained Mask R-
The RPN scans all FPN top-bottom features and CNN model is integrated into the IoT-based web server
proposes regions that potentially contain objects. using Tensorflow Lite for further damage segmenta-
Anchor boxes are rectangular regions defined to detect tion. To further evaluate the object detection
794 et al.
Yuan Structural Health Monitoring 21(3)
7

Figure 6. Pipeline of the Mask R-CNN for damage detection and segmentation.

Figure 7. Segmented results compared to actual images.

performance of the proposed model, the precision (Q) of instance segmentation, the mean intersection over
and recall (R) rates were used as follows55 union (MIU) rate is used to assess the performance of
image segmentation, and can be expressed as follows
TQ
Q= ð1aÞ
TQ + FQ 1 X k
pii
MIU = ð2Þ
TQ k + 1 i=0 P
k Pk
R= ð1bÞ pij + pji � pii
TQ + FN j=0 j=0

in which TQ, FQ, and FN, respectively, refer to the
number of cases that are positive and detected as posi-
tive, the number of cases that are negative but detected
positive, and the number of cases that are positive but
detected negative. The testing results from 50 images
showed that the general precision and recall rates were
80.10% and 81.98%, respectively. For the evaluation
where k refers to the number of training classes, and pij
refers to the number of pixels that belong to category i
but is misjudged as category j. Thus, pii, pij, and pji,
respectively, refer to the number of pixels with right
classification, false positive, and false negative. The
instance segmentation results from 50 testing images
showed that the MIU rate for crack detection can reach
Yuan
8 et al. 795
Figure 8. Damage object segmentation from 2D to 3D space.
80.97%, and such a rate is adequate for crack instance
segmentation. It is worth noting that this study only
applied instance segmentation instead of semantic seg-
mentation because instance segmentation frameworks
yield an individual mask for each ROI so that individ-
ual instances can be processed separately.
Figure 7 shows the input images and the correspond-
ing segmentation results using Mask R-CNN for vari-
ous concrete crack types (i.e. single crack, multiple
cracks, and concrete spalling). In addition, various
image resolutions, different background complexity,
and different structural scenes are all involved in the
data set. The segmentation results shown on the left
half of Figure 6 were derived from concrete columns
and slabs, and these images were collected by the stereo
camera. The right half shows the detection results of
the open-source database, including the spalling of C
floor, the cracks of pavement, and the shear cracks of
RC beams. This precise detection results show that the
Mask R-CNN used in this study is capable of damage
detection and damage location segmentation in the 2D
plane.
Quantification of concrete damage in 3D space
coordinates
Following segmentation with the Mask R-CNN, spatial
mapping from 2D to 3D space is conducted to quantify
the damage. Spatial mapping uses a single spatial map
to describe real-world geometry. The map can be ini-
tially sparse but continuously updated or can be
updated once after an entire area is mapped. However,
the existing of noise and holes can remarkably impact
the accuracy of 3D reconstruction. To enhance the
accuracy of damage quantification, a statistical outlier
removal (SOR) filter was employed in this study to pre-
process the 3D point cloud. Assuming that the results
follow a Gaussian distribution, all points with mean
Structural Health Monitoring 00(0)
distances outside an interval defined by the global mean
and standard deviation can be outliers and thus
trimmed from the data set. For each point cloud Pi
(i = 1, ..., n) in the data set, the average distance di(k)
can be computed with the consideration to K-nearest
neighbours (KNNs).56 By assuming that the average
distance to the KNN is normally distributed, the
standard-deviation multiplier (N) can be chosen based
upon the cumulative distribution function (CDF) from
the normal distribution.57
After going through point cloud denoising filters, the
projection from the 2D damage mask of the plane dam-
age segmentation to the 3D point cloud data is obtained
as shown in Figure 8. The relationship between a point
in 3D Cartesian space [x, y, z] and its pixel coordinates
[u, v, d] in an image can be expressed as follows
8
< u = xfx z + cx
v = yfy z + cy ð3Þ
:
d = zs
where d refers to the depth, fx and fy refer to the focal
length of the camera on the x-and y-axis, respectively,
cx and cy refer to the centre of the aperture, and s repre-
sents the zoom factor of the depth map. Given the inter-
nal parameters of a stereo camera, the spatial position
and pixel coordinates of each point can be described by
the following matrix
2 3 0 2 3 1
u x
s4 v 5 = C @R4 y 5 + tA ð4Þ
1 z
where C refers to the parameter matrix consisting of fx,
fy, cx, and cy, R represents the rotation matrix, t refers
to the displacement vector, and s is the scaling factor of
the data given in the depth map to the actual distance.
Estimation of crack volume
Open3D, which is an open-source library, is utilized to
compute a 3D convex hull for volumetric quantifica-
tion, as shown in Figure 9. The 3D convex hull is com-
puted based on a finite set of points in space.58
Mathematically, the convex hull of the point set P in
the real vector space is defined as the smallest convex
set that contains P. The convex hull is defined as all
convex combinations of the point set P, and the for-
mula can be expressed as follows
( )
X
n X
n
chð PÞ = ai pi jpi 2 P, ai ø 0, ai = 1, n = 1, 2, . . .
i=1 i=1
ð5Þ
796 et al.
Yuan
Figure 9. 3D concrete crack convex hull construction.
Figure 10. Steps for 3D scene reconstruction.
where a and n are the constant coefficients. Based on
Euler’s formula V 2 E + F = 2, where V refers to the
number of vertices, E refers to the number of edges,
and F represents the number of faces.59 For any convex
polytope P, if the body contains n vertices, then P will
not contain more than 3n 2 6 edges (E) and no more
than 2n 2 4 faces (F). In this study, a random incre-
mental algorithm was employed to construct the con-
vex hull of the point set P.60
3D scene reconstruction
Figure 10 shows computation steps needed for stereo-
based 3D scene reconstruction for real structures. The
Structural Health Monitoring 21(3)
9
ZED camera captured a pair of images at 720p resolu-
tion and a frame rate of 60 FPS for 3D reconstruction
of structural surfaces. The ZED Fusion Application in
the Software Development Kit (SDK) package was
used in real time to simultaneously scan and register
point clouds of an area. The 3D scene reconstruction
process comprised of five main stages, including image
acquisition, stereo calibration, stereo rectification,
stereo block matching, and 3D scene reconstruction, as
shown in Figure 10.
Stereo calibration is required to calibrate the cam-
eras and obtain the required intrinsic and extrinsic
parameters, which can be used to measure distances in
length units rather than in pixels. These parameters are
Yuan
10 et al. 797
Figure 11. Visualization of stereo camera trajectory and the
corresponding 3D-pose estimation.
estimated in an offline calibration process.
Rectification, which ensures distortion removal and
stereo alignment, is an online process. Once the intrin-
sic and extrinsic parameters are obtained, the corre-
spondence between the left and right images need to be
established, such as through a dense disparity map.
The dense disparity map can be computed based on a
method proposed by Geiger et al.61 The depth value zij
can be obtained by the following equation
b�f
zij =
dij
where dij is the disparity of pixel (i, j) in the image, b is
the stereo baseline, and f is the focal length of the
camera.
To estimate the camera pose in the current frame, it
is necessary to register the previous measurement value
based on the coordinates of the camera in the current
frame. This study used the IMU to accurately estimate
the 6-DOF pose (i.e. x, y, z, roll, pitch, and yaw) of the
camera and consequently the pose of the inspection
robot. Most point-based 3D reconstruction methods
map all valid pixels to the 3D space and project them
into a common coordinate system according to the esti-
mated camera pose. A clockwise scanning direction
was chosen for improved reconstruction accuracy. The
scanning path is directly impacted by the field of view
of the stereo camera as well as the maximum range of
the device. The horizontal field of view (FOV) of ZED
2 is 61. Consequently, the vertical FOV can be
obtained using the following equations
  
0:5 � H
FOVV = 2 arctan
fy
  
0:5 � W
FOVH = 2 arctan
fx
where fx and fy refer to the focal length, and W and H
represent the width and the height of the image, respec-
tively. The obtained vertical FOV is approximately 93.
Structural Health Monitoring 00(0)
Figure 12. Experimental setup.
As the horizontal FOV is remarkably larger than the
corresponding vertical FOV, the ZED camera was
rotated 90 horizontally to capture a larger vertical
area in one frame. Since the faster scanning speeds
caused the stereo camera to lose track of the object
with drift errors, and the IMU can be affected by the
ð6Þ
irregular or sudden movements, the movement of
inspection robot during scanning was limited to 0.5 m/
s. The quaternion-based extended Kalman filter (EKF)
was employed to estimate the dynamic pose of the
stereo camera. During the movement of the camera,
the accelerometer of the IMU was used as the long-
term reference for the static horizontal plane (i.e. the
vertical direction), and was fused with the angle change
measured by the gyroscope to achieve a drift-free
dynamic tilt response. The visualization of stereo cam-
era trajectory and the corresponding 3D-pose estima-
tion are detailed in Figure 11.
Application
Inspection robot and setup
Following the successful validation process, a post-
damage RC column recovered from a cyclic loading
test was selected to assess the performance of the pro-
posed inspection system. The prepared RC column has
a shear span of 1000 mm and a 200-mm square cross-
section. Figure 12 shows the field test setup on a scaled
ð7Þ
structural sample. The image reconstruction process
was carried out around the entire RC column at a dis-
ð8Þ tance of 70 cm. To reconstruct more details of the
entire column, the ZED camera was adjusted at differ-
ent viewing angles for data acquisition. A total of four
damage areas were visible as concrete cracks, all of
which occurred at the junction of the column and the
798 et al.
Yuan
Figure 13. Isolated volume of the damage convex hull (a) Crack #1, (b) Crack #2, (c) Crack #3, (d) Crack #4, and (e) damage
convex hull of Crack #3 under different viewing angles.
base. The ROIs for crack reconstruction and quantifi-
cation were concentrated around this area.
Crack assessment of the RC column
Figure 13 shows the isolated damage regions from the
3D damage convex hull construction of the damaged
RC column. The results demonstrate that the proposed
method can accurately segment the contours of con-
crete cracks and the 3D convex hull function can com-
pletely map the volume of the detected concrete
damage, as can be seen in Figure 13(e). Due to the irre-
gularity of the cracks, it is worth noting that the size of
concrete cracks, such as length, width, and depth refer
Structural Health Monitoring 21(3)
11
to the maximum length, width, and depth of a crack
area. Due to the difficulty of quantifying the crack vol-
ume of concrete, only Crack #1 was manually mea-
sured and calculated for damage volume. The crack
was divided into 10 small areas (from C1 to C10 in
Figure 12) to measure the length, width, and depth,
respectively, and finally, the volume of these 10 small
areas was summed to obtain the total volume of the
cracked region.
Error analysis
To assess the accuracy and potential for verifying as-
built conditions, the dimensions of RC column are
Yuan
12 et al. 799
Structural Health Monitoring 00(0)

Table 1. Crack length, width, depth, and corresponding measurements (units: mm).

Crack ID # Measured Proposed Manual RE (%)

type method (mm) measurement
(mm)

#1 Maximum length 201.76 205.90 2.01

Maximum width 55.06 58.80 6.37
Maximum depth 27.00 29.70 9.09
Volume 104,965.65 (mm3) 120,000.00 (mm3) 12.53
#2 Maximum length 214.91 218.00 1.42
Maximum width 91.53 94.50 3.14
Maximum depth 18.00 20.20 10.89
Volume 7,554,144 (mm3) – –
#3 Maximum length 121.37 125.80 3.52
Maximum width 290.69 291.80 0.38
Maximum depth 80.00 84.10 4.88
Volume 211,416.18 (mm3) – –
#4 Maximum length 189.69 192.10 1.25
Maximum width 87.09 89.20 2.37
Maximum depth 24.00 26.20 8.40
Volume 105,629.01 (mm3) – –
ME 5.10
MSE 9.53

RE: relative error; ME: mean error; MSE: mean square error.

extracted from the 3D scene reconstruction model
manually and compared to manual survey dimensions.
The error analysis has also been introduced by previous
studies.6,43 The length of the RC column and the maxi-
mum length, width, and depth of concrete damage were
measured in the point cloud data using MeshLab. For
manual measurement, a vernier calliper was used to
verify the inspection accuracy of the proposed method.
Table 1 lists the dimensions of the scaled model. Values
in Table 1 were obtained by measuring different posi-
tions of the cracked area. Since the crack length, width,
and depth varied greatly along the profile of the cracks,
the multiple measurements in the interior of the crack
and the largest measured model length, width, and
depth are reported for comparison. It is noted that the
volume of the damage region was not measured due to
the irregular shape of the concrete crack region.
Validation results
This study demonstrated an automated method for 3D
identification of concrete cracks. The 360 panoramic
model was reconstructed by the proposed system to
quantify and localize damage regions. According to the
comparison of the measured and computed crack
dimensions of four damage regions listed in Table 1,
the relative errors (REs) between manual measure-
ments and the proposed method are generally less than
13%, and the corresponding mean error (ME), respec-
tively, for crack size is 5.10%. The mean square error
(MSE) is 9.53%, which is acceptable for most practical
applications.
Conclusion
This study presents a novel inspection system to quan-
tify the structural damage based on stereo vision and
deep learning and can serve as a practical and effective
solution for inspection of civil structures. An innova-
tive, partially automated 3D segmentation and quanti-
fication method is proposed to extract damage
information and identify damage in building elements.
The major contributions are given as follows:
(1) Intelligent inspection robots have the potential to
replace human inspectors in conducting on-site
civil structure inspection. The robots also can per-
form quasi-real-time assessment a disaster scene
before rescuers enter a possibly dangerous envi-
ronment. Consequently, the efficiency of civil
inspection work can be greatly improved, and the
safety of inspection tasks can be remarkably
enhanced.
(2) A wireless data transmission method is proposed
to achieve quasi-real-time image display and
online damage detection. The transmission latency
of the reconstructed model after edge computing
is lower than 5 s.
(3) The proposed 3D damage segmentation and quan-
tification model based on Mask R-CNN can
800 et al.
Yuan
accurately localize cracks and quantify the seg-
mented damage volumes.
(4) An IoT-enabled data processing and visualization
system is adopted to process the 3D damage seg-
mentation and quantification. Following data pro-
cessing, the damage is projected into 3D space to
enable damage localization in the 3D reconstruc-
tion model. Each module comes together to form
a comprehensive system that is ready for deploy-
ment in civil structural inspection.
However, limitations of the proposed method
become apparent during deployment. Concrete cracks
smaller than 10 mm could not be detected because they
were not reconstructed in 3D. In addition, some of the
narrowest cracks cannot be detected when the stereo
camera is located beyond a certain distance.
Furthermore, this system is so far not fully automatic
because the 3D reconstruction model obtained by the
edge computing still requires human intervention for
uploading data to the cloud.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) disclosed receipt of the following financial sup-
port for the research, authorship, and/or publication of this
article: This research is supported by the National Key
Research Program of China (grant no. 2020YFC1512500),
the Science and Technology Commission of Shanghai
Municipality (grant no. 19DZ1201200), and the China
National Science Foundation (grant no. 51978507).
ORCID iD
Qingzhao Kong https://orcid.org/0000-0001-9577-4540
References
1. Yuan C, Chen W, Pham TM, et al. Finite element model-
ling of dynamic bonding behaviours between fibre rein-
forced polymer sheet and concrete. Construct Build Mater
2020; 255: 118939.
2. Noorsuhada M. An overview on fatigue damage assess-
ment of reinforced concrete structures with the aid of
acoustic emission technique. Construct Build Mater 2016;
112: 424–439.
3. Narazaki Y, Hoskere V, Hoang TA, et al. Vision-based
automated bridge component recognition with high-level
scene consistency. Comput Aided Civil Infrastruct Eng
2020; 35: 465–482.
Structural Health Monitoring 21(3)
13
4. Chupanit P and Roesler JR. Fracture energy approach to
characterize concrete crack surface roughness and shear
stiffness. J Mater Civil Eng 2008; 20: 275–282.
5. Spencer BF Jr, Hoskere V and Narazaki Y. Advances in
computer vision-based civil infrastructure inspection and
monitoring. Engineering 2019; 5: 199–222.
6. Torok MM, Golparvar-Fard M and Kochersberger KB.
Image-based automated 3D crack detection for post-
disaster building assessment. J Comput Civil Eng 2014;
28: A4014004.
7. Chang PC, Flatau A and Liu S. Health monitoring of
civil infrastructure. Struct Health Monit 2003; 2: 257–267.
8. Xu Y, Bao Y, Chen J, et al. Surface fatigue crack identifi-
cation in steel box girder of bridges by a deep fusion con-
volutional neural network based on consumer-grade
camera images. Struct Health Monit 2019; 18: 653–674.
9. Kharkovsky S, Giri P and Samali B. Non-contact inspec-
tion of construction materials using 3-axis multifunc-
tional imaging system with microwave and laser sensing
techniques. IEEE Instrum Meas Mag 2016; 19: 6–12.
10. Kang D and Cha YJ. Autonomous UAVs for structural
health monitoring using deep learning and an ultrasonic
beacon system with geo-tagging. Comput Aided Civil
Infrastruct Eng 2018; 33: 885–902.
11. Zhao J, Bao Y, Guan Z, et al. Video-based multiscale
identification approach for tower vibration of a cable-
stayed bridge model under earthquake ground motions.
Struct Control Health Monit 2019; 26: e2314.
12. Peng Z, Li J, Hao H, et al. High-resolution time-
frequency representation for instantaneous frequency
identification by adaptive Duffing oscillator. Struct Con-
trol Health Monit 2020; 27: e2635.
13. Fan G, Li J and Hao H. Vibration signal denoising for
structural health monitoring by residual convolutional
neural networks. Measurement 2020; 157: 107651.
14. Zhang J, Guo S and Zhang Q. Mobile impact testing for
structural flexibility identification with only a single refer-
ence. Comput Aided Civil Infrastruct Eng 2015; 30:
703–714.
15. Tu J, Sui H, Feng W, et al. Detecting facade damage on
moderate damaged type from high-resolution oblique
aerial images. IEEE J Select Topic Appl Earth Observ
Remote Sens 2017; 10: 5598–5607.
16. Sony S, Laventure S and Sadhu A. A literature review of
next-generation smart sensing technology in structural
health monitoring. Struct Control Health Monit 2019; 26:
e2321.
17. Kong Q, Fan S, Bai X, et al. A novel embeddable spheri-
cal smart aggregate for structural health monitoring: part
I. Fabrication and electrical characterization. Smart
Mater Struct 2017; 26: 095050.
18. Zhang J, Guo S, Wu Z, et al. Structural identification
and damage detection through long-gauge strain mea-
surements. Eng Struct 2015; 99: 173–183.
19. Lynch JP and Loh KJ. A summary review of wireless sen-
sors and sensor networks for structural health monitor-
ing. Shock Vib Digest 2006; 38: 91–130.
Yuan
14 et al. 801
20. Friswell MI and Penny JE. Crack modeling for structural
health monitoring. Struct Health Monit 2002; 1: 139–148.
21. Yoon H, Shin J and Spencer Jr BF. Structural displace-
ment measurement using an unmanned aerial system.
Comput Aided Civil Infrastruct Eng 2018; 33: 183–192.
22. Li S, Li H, Liu Y, et al. SMC structural health monitor-
ing benchmark problem using monitored data from an
actual cable-stayed bridge. Struct Control Health Monit
2014; 21: 156–172.
23. Yuan C, Kong Q, Chen W, et al. Interfacial debonding
detection in externally bonded BFRP reinforced concrete
using stress wave-based sensing approach. Smart Mater
Struct 2020; 29: 035039.
24. Kong Q, Robert RH, Silva P, et al. Cyclic crack monitor-
ing of a reinforced concrete column under simulated
pseudo-dynamic loading using piezoceramic-based smart
aggregates. Appl Sci 2016; 6: 341.
25. Kong Q, Hou S, Ji Q, et al. Very early age concrete
hydration characterization monitoring using piezocera-
mic based smart aggregates. Smart Mater Struct 2013;
22: 085025.
26. Beckman GH, Polyzois D and Cha YJ. Deep learning-
based automatic volumetric damage quantification using
depth camera. Autom Construct 2019; 99: 114–124.
27. Sony S, Dunphy K, Sadhu A, et al. A systematic review
of convolutional neural network-based structural condi-
tion assessment techniques. Engineering Structures 2021;
226: 111347.
28. Wang F, Mobiny A, Van Nguyen H, et al. If structure
can exclaim: a novel robotic-assisted percussion method
for spatial bolt-ball joint looseness detection. Struct
Health Monit. Epub ahead of print 1 June 2020. DOI:
10.1177/1475921720923147.
29. Wang F and Song G. 1D-TICapsNet: an audio signal
processing algorithm for bolt early looseness detection.
Struct Health Monit. Epub ahead of print 15 December
2020. DOI: 10.1177/1475921720976989.
30. Yuan C, Zhang J, Chen L, et al. Timber moisture detec-
tion using wavelet packet decomposition and convolu-
tional neural network. Smart Mater Struct 2021; 30:
035022.
31. Xue Y and Li Y. A fast detection method via region-
based fully convolutional neural networks for shield tun-
nel lining defects. Comput Aided Civil Infrastruct Eng
2018; 33: 638–654.
32. Zalama E, Gómez Garcı́a Bermejo J, Medina R, et al.
Road crack detection using visual features extracted by
Gabor filters. Comput Aided Civil Infrastruct Eng 2014;
29: 342–358.
33. Ni F, Zhang J and Chen Z. Pixel-level crack delineation
in images with convolutional feature fusion. Struct Con-
trol Health Monit 2019; 26: e2286.
34. Eick BA, Narazaki Y, Smith MD, et al. Vision-based
monitoring of post-tensioned diagonals on miter lock
gate. J Struct Eng 2020; 146: 04020209.
35. Bao Y, Chen Z, Wei S, et al. The state of the art of data
science and engineering in structural health monitoring.
Engineering 2019; 5: 234–242.
Structural Health Monitoring 00(0)
36. Li S, Wei S, Bao Y, et al. Condition assessment of cables
by pattern recognition of vehicle-induced cable tension
ratio. Eng Struct 2018; 155: 1–15.
37. Koziarski M and Cyganek B. Image recognition with
deep neural networks in presence of noise–dealing with
and taking advantage of distortions. Integr Comput Aided
Eng 2017; 24: 337–349.
38. Jang K, Kim N and An YK. Deep learning–based auton-
omous concrete crack evaluation through hybrid image
scanning. Struct Health Monit 2019; 18: 1722–1737.
39. Saleem MR, Park J-W, Lee J-H, et al. Instant bridge
visual inspection using an unmanned aerial vehicle by
image capturing and geo-tagging system and deep convo-
lutional neural network. Struct Health Monit. Epub
ahead of print 1 July 2020. DOI: 10.1177/1475921720
932384.
40. Lei B, Wang N, Xu P, et al. New crack detection method
for bridge inspection using UAV incorporating image
processing. J Aerosp Eng 2018; 31: 04018058.
41. Bao Y and Li H. Machine learning paradigm for struc-
tural health monitoring. Struct Health Monit. Epub
ahead of print 24 November 2020. DOI: 10.1177/147592
1720972416.
42. Yeum CM, Dyke SJ, Ramirez J, et al. Big visual data
analytics for damage classification in civil engineering. In:
Transforming the future of infrastructure through smarter
information: proceedings of the international conference on
smart infrastructure and construction, Cambridge, 27–29
June 2016, pp. 569–574. London: ICE Publishing.
43. Jiang S and Zhang J. Real-time crack assessment using
deep neural networks with wall-climbing unmanned aer-
ial system. Comput Aided Civil Infrastruct Eng 2020; 35:
549–564.
44. Cha YJ, You K and Choi W. Vision-based detection of
loosened bolts using the Hough transform and support
vector machines. Autom Construct 2016; 71: 181–188.
45. Omar T and Nehdi ML. Remote sensing of concrete
bridge decks using unmanned aerial vehicle infrared ther-
mography. Autom Construct 2017; 83: 360–371.
46. Lei B, Ren Y, Wang N, et al. Design of a new low-cost
unmanned aerial vehicle and vision-based concrete crack
inspection method. Struct Health Monit 2020; 19: 1871–1883.
47. Wang C, Wang N, Ho S-C, et al. Design of a new vision-
based method for the bolts looseness detection in flange
connections. IEEE Trans Ind Electron 2019; 67:
1366–1375.
48. Anand R and Kumar P. Flaw detection in radiographic
weldment images using morphological watershed segmen-
tation technique. NDT&E Int 2009; 42: 2–8.
49. Chen PH, Shen HK, Lei CY, et al. Support-vector-
machine-based method for automated steel bridge rust
assessment. Autom Construct 2012; 23: 9–19.
50. Cha YJ, Choi W and Büyüköztürk O. Deep learning-
based crack damage detection using convolutional neural
networks. Comput Aided Civil Infrastruct Eng 2017; 32:
361–378.
51. Li R, Liu J, Zhang L, et al. LIDAR/MEMS IMU inte-
grated navigation (SLAM) method for a small UAV in
802 et al.
Yuan
indoor environments. In: 2014 DGON inertial sensors
and systems (ISS), Karlsruhe, 16–17 September 2014,
pp. 1–15. New York: IEEE.
52. Gomes G. Deep learning-based volumetric damage quan-
tification using an inexpensive depth camera, 2018,
https://mspace.lib.umanitoba.ca/xmlui/handle/1993/33222
53. Ali MI, Ono N, Kaysar M, et al. Real-time data analytics
and event detection for IoT-enabled communication sys-
tems. J Web Semantic 2017; 42: 19–37.
54. He K, Gkioxari G, Dollár P, et al. Mask R-CNN. In:
Proceedings of the IEEE international conference on com-
puter vision, Venice, 22–29 October 2017, pp. 2961–2969.
New York: IEEE.
55. Yu Y, Zhang K, Yang L, et al. Fruit detection for straw-
berry harvesting robot in non-structural environment
based on Mask-RCNN. Comput Electr Agricult 2019;
163: 104846.
56. Rusu RB, Blodow N, Marton Z, et al. Towards 3D object
maps for autonomous household robots. In: 2007 IEEE/
RSJ international conference on intelligent robots and
Structural Health Monitoring 21(3)
15
systems, San Diego, CA, 29 October–2 November 2007,
pp. 3191–3198. New York: IEEE.
57. Carrilho A, Galo M and Santos R. Statistical outlier
detection method for airborne lidar data. In: Interna-
tional Archives of the Photogrammetry, Remote Sensing &
Spatial Information Sciences, Karlsruhe, 10–12 October
2018.
58. Barber CB, Dobkin DP and Huhdanpaa H. The Quic-
khull algorithm for convex hulls. ACM Trans Math Softw
1996; 22: 469–483.
59. Kabluchko Z, Last G and Zaporozhets D. Inclusion–
exclusion principles for convex hulls and the Euler rela-
tion. Discrete Comput Geometry 2017; 58: 417–434.
60. Chan TM. Output-sensitive results on convex hulls,
extreme points, and related problems. Discrete Comput
Geometry 1996; 16: 369–387.
61. Geiger A, Roser M and Urtasun R. Efficient large-scale
stereo matching. In: Asian conference on computer vision,
Queenstown, New Zealand, 8–12 November 2010, pp.
25–38. New York: Springer.