Automation in Construction 140 (2022) 104302

Contents lists available at ScienceDirect

Automation in Construction
journal homepage: www.elsevier.com/locate/autcon

Review

Deep learning-based data analytics for safety in construction

Jiajing Liu a, b, Hanbin Luo a, b, *, Henry Liu c
a
National Center of Technology Innovation for Digital Construction, Huazhong University of Science and Technology, 430074 Wuhan, China
b
School of Civil Engineering and Hydraulic Engineering, Huazhong University of Science and Technology, 430074 Wuhan, China
c
School of Design and the Built Environment, University of Canberra, 11 Kirinari St., Bruce, ACT 2617, Australia

A R T I C L E I N F O A B S T R A C T

Keywords: Deep learning has been acknowledged as being robust in managing and controlling the performance of con­
Safety struction safety. However, there is an absence of state-of-the-art review that examines its developments and
Deep learning applications from the perspective of data utilization. Our review aims to fill this void and addresses the following
Computer vision
research question: what developments in deep learning for data mining have been made to manage safety in con­
Natural language processing
struction? We systematically review the extant literature of deep-learning-based data analytics for construction
Knowledge graph
safety management, including: (1) image/video-based; (2) text-based; (3) non-visual sensor-based; and (4) multi-
modal-based. The review revealed three challenges of existing research in the construction industry: (1) lack of
high-quality database; (2) inadequate ability of deep learning models; and (3) limited application scenarios.
Based on our observations for the prevailing literature and practice, we identify that future research on safety
management is needed and focused on the: (1) development of dynamic multi-modal knowledge graph; and (2)
knowledge graph-based decision-making for safety. The application of deep learning is an emerging line of in­
quiry in construction, and this study not only identifies new research opportunities to support safety manage­
ment, but also facilitates practicing deep learning for construction projects.

1. Introduction emerged. It refers to a method that is robust in automatically extracting

Worldwide, safety in the construction industry is a challenge for
practitioners [1,2,3]. For example, in the UK, approximately 30% of
workplace accidents are from the construction sector [4]. According to
the U.S. Bureau of Labor Statistics, the death toll of the construction
industry in the United States climbed up to 1008 in 2018, i.e., 37 more
than in that of 2017 [5]. In China, 904 people were killed during the
construction of housing and municipal engineering projects in 2019 [6].
Despite an intensive attention being paid to safety management, the
construction-related accident rate has remained to be high.
To manage safety in construction, emerging information technolo­
gies, such as Internet of Things (IoT) and computer vision, have been in
place to monitor daily on-site activities [7,8,9,10]. As a result, sub­
stantial data (e.g., digital images and safety reports) have been recorded
and stored, containing significant information for resolving such prob­
lems as unsafe behavior and structural defeats [7,11]. However,
extracting useful information knowledge from such a large amount of
data for safety management and decision-making is challenging [12].
Consequently, Deep learning, a subset of Artificial Intelligence (AI), has
sophisticated features at high levels of abstraction and, therefore, is
promising for enabling a higher effective and efficient management of
construction safety [13].
Over the past decade, deep-learning techniques have been further
developed to enhance construction on-site safety [14,15,7,16]. For
example, Fang et al. [15] and Ding et al. [14] developed deep learning
and computer vision approaches to monitor and identify people’s unsafe
behavior. Despite the research above, there has been no state-of-the-art
review with an attempt to address: What developments in deep learning for
data mining have been made to manage safety in construction? [17,18,19].
Addressing the research question above requires a systematic review to
garner an understanding of and examine deep learning and construction
safety in terms of the data-mining perspective, so that headway can be
made to engender and enact a robust focus on research and practice
enhance of construction safety management [7,16].
This paper commences with an introduction of the data source for
safety in construction (Section 2). Then, Section 3 introduces the
development of deep learning in computer science. Next, the research
approach used to form the study’s line of inquiry is presented (Section

* Corresponding author.
E-mail address: luohbcem@hust.edu.cn (H. Luo).

https://doi.org/10.1016/j.autcon.2022.104302
Received 13 September 2021; Received in revised form 13 February 2022; Accepted 26 April 2022
Available online 10 May 2022
0926-5805/© 2022 Elsevier B.V. All rights reserved.
J. Liu et al.
4). In Section 5, we review the existing deep learning applications for
safety in construction industry to provide a contextual backdrop of the
research. Finally, the research challenges, recommendation of future
research, and conclusion are discussed and presented.
2. Data source for safety in construction
Within the context of safety, we focus on unsafe behavior and rele­
vant conditions that pervade on-site practice. It has been observed that
88% of the accidents that occurred on construction sites were as a result
of unsafe behaviors, while 10% of that was attributable to unsafe work
conditions (e.g., plant and structural monitoring). As addressed, AI re­
quires different types of data (i.e., digital images and safety reports)
useful for the management of safety. These include:
• Image/video: In the construction industry, video surveillance system
is installed on the sites to record daily activities in real-time with
permission. Additionally, the safety inspectors use cameras or
phones to record hazards during their on-site inspection ((Fig. 1 (a)).
• Safety report: As noted from Fig. 1 (b), the safety report data (e.g.,
quality and safety reporting notice, weekly supervision form) are
recorded by engineers who are experienced in documenting safety
incidents, as is shown in Fig. 1(b).
• Non-visual sensor data: Having non-visual sensors (e.g., RFID, UWB,
GPS) in place is also a common practice to monitor unsafe behavior
and conditions. For example, positioning sensors are used to track
people’s location in construction sites. Likewise, sensors (e.g.,
displacement, deformation) are used to monitor surrounding settle­
ment during excavation. This type of data is stored in numeric
format, as shown in Fig. 1 (c).
3. Deep learning in computer science
The advent of deep learning algorithms has enabled a wide appli­
cation of deep learning to various areas such as healthcare [20], self-
driving cars [21], and language translations [22]. In line with the
research question above, the deep learning techniques and the devel­
opment of two prevailing applications (e.g., computer vision and natural
language processing) are presented. We examine the deep-learning-
related technologies in computer science, as they form the underlying
basis for deep learning-based analytics in construction.
3.1. Deep learning techniques
Having benefitted from large-scale data training has advanced deep
learning. However, in the industries where training data are lacking, the
applications of deep learning are limited due to its rigorously required
(a) (b)
Fig. 1. Example of the data source in the construction industry.
2
Automation in Construction 140 (2022) 104302
amount of inputted data. Transfer learning can leverage knowledge and
skills learned in similar domains/tasks, and it is being applied as a
common approach to overcome the absence of domain data sets [23,24].
Based on the representation of knowledge to be transferred, transfer
learning algorithms can be categorized into: (1) instance-transfer; (2)
feature-representation-transfer; (3) parameter-transfer; and (4)
relational-knowledge-transfer [25].
Unlabeled data, essentially, is adequate or can be easily collected in
many cases, while high-quality labeled data is normally limited and
expensive to acquire. More importantly, the model’s accuracy may not
be significantly improved, though additional time is consumed with an
enlarged training data set. In this regard, active learning has been
developed, with an aim of iteratively selecting information samples from
the unlabeled pool to minimize the cost of constructing datasets and
then improve model performance [26,27]. Emerging from the applica­
tion scenarios engenders three types of active learning: (1) membership
query synthesis; (2) stream-based selective sampling; and (3) pool-based
active learning [26]. Membership query synthesis actively generates
new unlabeled instances to be queried. Stream-based selective sampling
is based on a hypothesis that an unlabeled instance is observed at a time,
and query selection is performed in order by identifying whether each
instance is queried. Additionally, pool-based active learning performs
query selection by prioritizing instances using acquisition functions and
assumes that there are many unlabeled instances.
Data augmentation is another technique widely used in deep
learning tasks. It deals with overfitting and contributes to improving the
performance of model that is associated with unseen samples [28]. Data
augmentation focuses on enhancing the value of limited data and can be
divided into the supervised and unsupervised (data augmentation)
methods. The supervised data augmentation adopts preset data trans­
formation rules. It performs data augmentation depending on existing
data, e.g., (1) single-sample data augmentation, which is based on the
geometric or color transformation of the sample; and (2) multi-samples
data augmentation, which integrates multiple samples to generate new
samples. By contrast, the unsupervised data augmentation learns the
distribution of data through the model, randomly generating samples
that are consistent with the distribution of the training data set. Notably,
a typical application of the unsupervised data augmentation algorithm is
Generative Adversarial Network.
3.2. Deep learning for computer vision
Computer vision is an interdisciplinary field of science, which fo­
cuses on constructing computational models to capture contextual in­
formation from digital images or videos for automating the tasks of the
human visual system [29]. Deep-learning-based computer vision has
contributed to producing impressive results and even exceeded human-
(c)
J. Liu et al. Automation in Construction 140 (2022) 104302

level performance in some cases [30,31,32]. The typical tasks of com­
puter vision include:
• Object detection: Object detection describes the ability to detect or
identify objects in any given image correctly along with their spatial
position in the given image. It is in the form of rectangular boxes
(known as Bounding Boxes) within which the object it is bounded.
Cases in point regarding object detection networks are relation to
Faster R-CNN [33], F-RCN [34], SSD [35], YOLO [36] and RetianNet
[37].
• Object tracking: Object tracking is to predict the size and location of
the target in subsequent frames, given the size and location of the
target in the initial frame of a video sequence. As a result, the posi­
tion relationship of the object to be tracked in the continuous video
sequence is established; thus, the complete trajectory of the object
can be obtained. Typical attempts (tracking algorithms) that were
developed based on deep learning include C-COT [38], DLT [39],
FCNT [40], and MDNet [41].
• Instance segmentation: It aims to locate the individual objects and
segment their boundaries (i.e., lines and curves) within a scene,
regardless of whether they are under the same type. Well-known
instance segmentation networks comprise Mask R-CNN [42], Mas­
kLab [43], and YOLACT [44].
• Pose/Activity recognition: Pose/Activity recognition is used to un­
derstand human actions and behaviors by determining human joints’
location from images, for example, CPM [45], Hourglass [46], CPN
[47], and HRNet [48].
Table 1 presents the examples of deep-learning-based computer
vision algorithms in the context of computer science.
3.3. Deep learning for natural language processing
Natural language processing (NLP) is a series of theoretically
inspired computing techniques used to represent and analyze human
language automatically [59,60]. The general pipeline of practicing NLP
tasks is indicated in Fig. 2. Corpus is a collection of texts, which could be
either standard public data sets or existing electronic texts. However,
corpus preprocessing is time-consuming and laborious, including corpus
cleaning, word segmentation, and stop words removal. Word embedding
is adopted to convert words into vector representations prior to feature
extraction.
The common tasks of NLP include:
• Name entity recognition (NER): The NLP model is used to identify,
extract and categorize entities from sentences. It comprises the
extracting names of people, locations and things, which are derived
from the text and placed under certain categories, such as person,
location and time. For instance, BERT and BiLSTM-CRF are the ap­
plications of NER [61,62].
• Text classification: The NLP model is trained to classify documents/
text reports according to specific attributes, e.g., subject, document
type and time. The model extracts the features of the text and then
matches them with the predetermined categories for classification. A
popular case in point regarding text classification models is BERT
[61].
• Information retrieval: The NLP model can locate the required infor­
mation among unstructured data. An information retrieval system
indexes a collection of documents and analyzes the user’s query.
Then, it compares the description of each document with the query
and finally generates and presents the relevant results. The prevail­
ing methods for information retrieval involve DRMM [63] and
SNRM_PRF [64].
• Text summarization: An NLP algorithm can be used to create a
shortened version of a text report with main points. Fundamentally,
there are two general approaches for text summarization, i.e.,
abstractive and extractive summarizations. In the abstractive case,
the NLP model engenders an entirely new summary in terms of
phrases and sentences used in the analyzed text. Regarding the
extractive summarization, the model extracts phrases and sentences
from the existing text and groups them into a summary.

Table 1
Examples of deep learning-based computer vision algorithms in computer
4. Research approach
science.
The systematic review, as addressed above, has been performed for
Task Algorithms Performance Reference
this research. It is initially built on retrieving the relevant research pa­
SAPD model can achieve a pers from Scopus database. The use of this database is robust for opti­
Zhu et al.
SAPD single-model single-scale AP of
[49] mizing search results for a systematic review [65]. A series of keywords,
47.4% on COCO database
FCOS with ResNeXt-64x4d-101 such as “deep learning”, “construction safety” or “safety management”
Tian et al.
Object detection FCOS achieves 44.7% in AP on COCO or “safety”, and “construction”, were inputted into the search engine of
[50]
database the selected database. Keywords were applied in the Title/Abstract/
FSAF model can achieve a state- Keyword’ (T/A/K) field of the Scopus. In this research, the filters were
Zhu et al.
FSAF of-the-art 44.6% mAP on COCO
database
[51] applied as follows: (1) peer-reviewed articles; (2) language: English; and
GMTracker achieves 63.8 IDF1 (3) timespan: between 2012 and 2021.
GM- He et al.
on MOT17 and 63.9 IDF1 on To avoid missing influential papers, the Top 5 journals in terms of the
Tracker [52]
MOT16 number of their articles relating to deep learning-based construction
SiamMOT (DLA-169) achieves
Shuai et al. safety were further searched to filter out the papers containing “deep
Object tracking SiamMOT 53.2 MOTA and 51.7 IDF1 on
HiEve challenge
[53] learning” in their titles, or keywords or abstracts. These include Auto­
DyGLIP achieves 84.6 MOTA and mation in Construction (AIC), Advanced Engineering Informatics (AEI),
Quach et al.
DyGLIP 39.9 IDF1 in S05 on AI City
[54]
ASCE Journal of Construction Engineering and Management (JCEM), ASCE
challenge database Journal of Computing in Civil Engineering (JCCE), and Computer-Aided Civil
DC- NAS achieves 84.3% on
Dcnas Cityscapes, and 86.9% on
Zhang et al. and Infrastructure Engineering (CACIE). The filtered papers were then
[55] manually reviewed to determine their relevance. In summary, a total of
Object PASCAL VOC 2012
segmentation DystaB achieves 58.9 and 47.2 185 articles were identified for our next analysis.
Yang et al.
DystaB on DAVIS17 and Youtube-VOS
[56] Fig. 3 illustrates the distribution of the 185 bibliographic records
database
from 2012 to 2021. There is a trend that the number of publications in
SimPoE achieves 56.7 MPJPE
and 26.6 MPJPE on Human3.6 M Yuan et al. this topic has significantly increased over the past five years. Accord­
SimPoE ingly, we conclude that deep-learning-based safety management has
Pose/Activity and MPII- INF-3DHP dataset, [57]
recognition respectively. become and continues being a significant research scheme and, there­
FCPose achieves 64.8% APkp on Mao et al. fore, a plethora of scholarly works is expected in this area in the near
FCPose
the COCO dataset [58]
future.

3
J. Liu et al. Automation in Construction 140 (2022) 104302

Fig. 2. General pipeline of NLP application tasks.

relationships from a micro to a macro perspective; that is, between

80 scholars and institutions. Fig. 4 shows the co-authorship network of the
Number of Publications

70 67 bibliographic records of the observed period (2012− 2021). As noted,

there have been strong collaborative relationships between multiple
60
institutions, e.g., the research institute of Prof. Heng Li, Prof. Lieyun
50 Ding and Prof. Hanbin Luo.
40 Fig. 5 presents the results of the co-occurrence keywords. The min­
31 imum number of occurrences of a keyword is set to 5. The size of the
30 nodes represents the frequency of the keywords, which can be split into
20 13 two categories: (1) method-related (e.g., deep learning, convolutional
neural network and computer vision); and (2) topic-related (e.g., con­
10
0 0 0 0 0 4 struction safety, construction workers and structural health monitoring).
0
2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021
5. Deep learning-based data analytics for safety
Year

Fig. 3. The year profile of publications.
Research collaboration describes collaborative research works un­
dertaken to achieve the common goal of producing new scientific
knowledge. A collaboration analysis in this study was run to illustrate
As noted from the above sections, deep-learning-related techniques
relate to data analytics. Therefore, they have the potential to accurately
detect unsafe behavior and unsafe condition. Considering this point of
view, we examine the progress of deep-learning-based data analytics for
safety in construction.

Fig. 4. Co-authorship network.

4
J. Liu et al.
Fig. 5. Keywords co-occurrence network.
5.1. Image/video-based data analytics
Exploring the relevant literature has identified that previous studies
were focused on using image/video to manage construction safety. They
primarily concentrated on: (1) prevention of people’s unsafe behavior;
(2) structural health monitoring.
5.1.1. Prevention of people’s unsafe behavior
Research has shown that approximately 88% of accidents are caused
by people’s unsafe behavior [66,67]. An accurate detection for people’s
unsafe behavior in real-time is thus pivotal for managing safety
behavior. As demonstrated above, deep-learning-based computer vision
approaches have been adopted by past studies to identify people’s un­
safe behavior automatically. According to the methods used for behavior
monitoring, we categorized such studies into (1) object detection; (2)
proximity measurement; (3) activity-based; and (4) semantic reasoning.
5.1.1.1. Object detection. Since studies suggest that a considerable
number of injuries occurred as the result of the absence of personal
protective equipment (PPE) [68], and detecting whether workers wear
PPE as required has been the hotspot, including the detection of the hard
hat [69,70,71], vest [69] and safety harness [72]. Emerging from Nath
et al.’s research [69], for example, is related to three deep learning
approaches that are built on You-Only-Look-Once (YOLO) V3 for
determining if a person wears a hard hat, vest, or both properly. The
result showed that the one-step end-to-end approach ensures the best
performance with a mean average precision of 72.3% and a processing
speed of 11 frames per second.
5.1.1.2. Proximity measurement. The improper spatial relationship on
construction sites engenders risks for people who enter the hazard area.
Fang et al. [73] utilized Mask R-CNN to detect people traversing struc­
tural support according to whether their masks overlap. Continuing with
this theme, more studies have developed deep-learning-based computer
vision methods to detect people entering dynamic hazard areas, such as
the working area of heavy vehicles, which can lead to struck-by acci­
dents [74,75,8,76]. For instance, Luo et al. [8] applied YOLO v2 to
detect plant and people and then measured the distance between them
based on perspective transformation. Similarly, Yan et al. [76] estimated
5
Automation in Construction 140 (2022) 104302
the 3D spatial relationship between people and heavy vehicles using the
3D bounding box reconstructed from the video that is recorded by the
monocular camera, aiming to address the spatial relationship distortion
caused by two-dimensional image pixels-based estimation.
5.1.1.3. Activity-based. People-activity monitoring enables the appli­
cation of automatic detection for unsafe behavior, which relies on
mining spatiotemporal information in video frames [14,77,78,79]. For
example, Ding et al. [14] applied a CNN model to extract the visual
features from videos and utilized an LSTM model to sequence the
learning features for unsafe behavior recognition, which outperformed
the descriptor-based methods. In response to the characteristics of far-
field surveillance videos, Luo et al. [78] adapted a deep action recog­
nition method with a fusion strategy to recognize workers’ actions and
then proposed a Bayesian nonparametric hidden semi-Markov model to
infer people’s activities based on action sequences.
5.1.1.4. Semantic reasoning. The task-oriented model developed based
on predetermined rules cannot identify multiple types of unsafe be­
haviors. This is because it is unable to accommodate the adjustments of
safety regulations either. To solve this issue, the semantic-reasoning-
based approach has been developed [80,81,82,83]. For example, Tang
et al. [84] proposed a human-object interaction (HOI) recognition model
to detect spatial relationships among objects in digital images so that
hazards implied in the image can be determined and detected.
Continuing with this, Fang et al. [80] combined computer vision algo­
rithms with ontology to develop a knowledge graph to understand
construction scene and then infer whether there is an existence of unsafe
behavior. Likewise, Pan et al. [81] integrated zero-shot learning tech­
niques with a knowledge graph to improve the accuracy and timeless
knowledge graph updates. Table 2 summarizes prior key studies on
computer vision and deep-learning-based unsafe behavior recognition.
5.1.2. Structural health monitoring
Structural health monitoring (SHM) is a strategy and process for
damage identification and characterization of engineering structures
[89]. Structural damage refers to the change of structural material pa­
rameters and geometric characteristics, which can lead to structural
failure as well as significant safety accidents under unforeseen
J. Liu et al. Automation in Construction 140 (2022) 104302

Table 2 for example, crack detection [93,94,91,95,96,104,97,98,99], pothole

Examples of computer vision and deep-learning-based unsafe behavior
recognition.
Task Description
Object detection- Three deep learning models were proposed to
based detect if workers are wearing hard hat and vest
A Faster-R-CNN and deep CNN model was
developed to identify workers and their
harness
A deep learning model for non-hardhat-use
(NHU) detection is developed
A real-time smart video surveillance
monitoring system was developed to recognize
people entering dangerous areas on a
construction site
Proximity
A computer vision method for UAV-assisted
measurement-
remote proximity monitoring was presented
based
A real-time warning system using visual data is
proposed to protect workers from potentially
dangerous situations involving equipment
operations
A hybrid deep learning model is proposed to
automatically recognize workers’ abnormal
actions
Activity-based
Using deep learning approach to identify
irregular construction activities from image
data
A knowledge graph integrating computer
vision with ontology is developed for
Semantic reasoning- identifying hazards on construction sites
based The framework integrates computer vision and
ontology is developed for semantically
reasoning hazards and mitigations
circumstances. As shown in Table 3, existing literature is replete with
studies applying deep learning to SHM.
Extensive efforts have been put into vision-based damage detection,
detection [100,101,92], corrosion detection [102], bolt-loosening
detection [103] and sewer pipe defect detection [104,106,105]. Such
Author works above have enabled developing a taxonomy comprising: (1)
(Year) classification to determine whether there are defects in the image; (2)
Nath et al. object detection to locate the defects in the image; and (3) segmentation
[69] to measure the geometric size of defects.
Fang et al. To take crack detection as an example, attempts have been made to
[85] explored deep learning models to determine whether an image/image
Fang et al. patch contains cracks. For instance, Gopalakrishnan et al. [23] used the
[15] VGG-16 DCNN, pre-trained on a massive ImageNet database with
transfer learning to search out the images containing cracks. Kang and
Luo et al. Cha [91] introduced a deep CNN with a sliding window technique to
[8]
analyze video data collected from UAVs. In particular, an autonomous
UAV method using ultrasonic beacons and a geotagging method for the
Kim et al.
[75] localization of damage were proposed.
Other researchers applied semantic segmentation techniques to
Son et al.
provide a pixel-wise prediction of cracks in the images. Emerging from
[86]
Bang et al. [93] is a deep convolutional encoder-decoder network, which
is useful for identifying road cracks in black-box images at the pixel
Ding et al.
[14]
level. The encoder is composed of the convolutional layer of the residual
network to extract crack features, and the decoder consists of the
Lin et al. deconvolution layer to locate cracks’ edges in the input image. In
[87] addition, the estimation of structural strength based on computer vision
techniques has also received attention. Wang et al. [108] demonstrated
Fang et al.
[80] the feasibility of the faster region-based convolutional neural network to
evaluate the residual strength of corroded textile-reinforced concrete, i.
Wu et al. e., a mean average precision of 98.98%.
[88] While there has been a significant achievement of computer-vision-
based people’s unsafe behavior detection and SHM, issues associated
with identifying multiple defects or multiple unsafe behaviors in real
projects remain unaddressed because of “noise” in the construction
industry.

5.2. Text-based data analytics

Table 3
Safety reports record valuable sources of information about how and
Prior works on deeplearning-based structural health monitoring.
why an event occurred. The existing attempts to analyze such reports
Application/ Description Type of data Reference depend on manual processing, while the unstructured free-text format
Task
could make the work be lengthy and challenging. With the development
1) Whether there are Image, of deep learning in NLP, there is a trend to explore deep learning tech­
[23,90]
cracks video
niques for automatic safety report analysis. The examples of typical
Crack 2) Locate cracks in Image,
detection the image video
[91,92] application scenarios are shown in Table 4.
3) Geometric Image,
[93,94,95,96,97,98,99]
measurement video Table 4
1) Locate potholes in Image,
[92] Prior works on deep learning-based safety report analysis.
the image video
Pothole
Image, Application/ Description Method Reference
detection 2) Geometric
video, point [100,101] Task
measurement
cloud
Accident causes 1) Classify accident Word2Vec skip-gram [113]
Corrosion 1) Locate corrosion Image,
[102] analysis causes model and a hybrid
detection in the image video
structured deep neural
1) Locate bolts in the
Bolt-loosening Image, network
image and estimate [103]
detection video Convolutional [114]
the rotational angles
bidirectional long short-
1) Whether there are Image,
[104] term memory model
defects video
Sewer 2) Identify the most CNN and hierarchical [115]
2) Locate defects in Image,
pipe defect [105] impactful hazards attention networks
the image video
detection and behaviors
3) Geometric Image,
[106] Hazard 1) Classify hazard BERT [116]
measurement video
classification types Word embedding, CNN, [117]
Structure
1) Concrete strength and LDA
strength Image [107,108]
evaluation LDA, CNN, Word Co- [118]
estimation
occurrence network and
1) Provide fined-
Structure word cloud technology
detailed defect Vibration
damage [109,110,111,112] Accident 1) Critical BiLSTM-CRF [119]
information for signal
diagnosis information information
structure
extraction extraction from
accident reports

6
J. Liu et al.
The analysis of accident causes extracted from accident reports is
useful for understanding, predicting, and preventing the occurrence of
construction accidents. In this instance, Zhang et al. [114] proposed a
convolutional bidirectional long short-term memory-based method to
classify the construction accident narratives according to the type of the
accident cause. Moreover, Baker et al. [115] applied Hierarchical
Attention Networks to learn valid injury precursors from accident re­
ports to identify the most significant hazards and behaviors in a fully
automated and data-driven manner. In contrast, classifying safety re­
ports based on hazard types has received attention [116,117,118]. For
example, Zhong et al. [118] applied a Latent Dirichlet Allocation algo­
rithm to identify hazard topics, then used a Convolution Neural Network
algorithm to classify hazard records, and finally generated a Word Co-
occurrence Network to determine the interrelations between hazards
and exploited Word Cloud technology to provide a visual overview of
hazard records. In addition to the classification task focusing on the
singular dimension, Feng and Chen [119] adopted the BiLSTM-CRF
model to extract critical information from accident reports, including
date, location, accident causes, type of accident fatality, and injury
situations.
Despite significant contribution above made to the text-based in­
formation extraction for safety, with the following issues, such as open
information extraction, long text-based information extraction and dy­
namic information update, are still being questioned. Further, using
information that has been extracted for safety decision-making is being
challenging for research.
5.3. Non-visual sensor-based data analytics
Managing construction safety through the use of non-visual sensor is
one of foci in the existing literature. The research scopes of relevant
studies include: (1) prevention of plant accident; and (2) structural
health monitoring.
5.3.1. Prevention of plant accident
Although the plant-related accidents can result in severe casualties
and significant economic loss, the existing research on applying deep
learning to manage such issues is limited. In essence, several studies
have made headways to the operation status control of the shield ma­
chine [120,121,122,123]. For example, Zhou et al. [123] proposed a
hybrid deep learning model composed of the wavelet transform, CNN,
and LSTM, to predict shield machine’s attitude and position during
Tunnel Boring Machine (TBM) excavation. Their model overwhelmed
the three widely used predictive models (ARIMA, LSTM, and WLSTM) in
terms of prediction accuracy. Subsequently, Zhang et al. [122] identified
a cyber-physical system (CPS)-based hierarchical autonomous control
scheme to support pressure balance control of slurry shield tunnel
boring machines. Emerging from Zhang et al. [122]’s work is a deep
learning model for the coordination-level controller. However, their
result showed that its performance was not better than an autonomous
control system based on the proposed switched model predictive
controller.
In the case of plants, there has been a paucity of research examining
their location and status. This is due to the difficulties in the identifi­
cation of their mechanical parameters and the extraction of attributes.
Moreover, AI-based plant status prediction using historical and real-time
data remains a grand challenge in real projects.
5.3.2. Structural health monitoring
Vibration signal is another data source widely used for SHM, and the
underlying assumption is that the dynamic characteristics and responses
will be changed when there damages exist. In this instance, Sajedi and
Liang [112] applied a fully convolutional encoder-decoder neural
network for vibration-based SHM technique and performed semantic
damage segmentation in a grid environment framework available for
large-scale SHM. The proposed model achieved an accuracy of 96.3%
7
Automation in Construction 140 (2022) 104302
and 93.2% in locating damage and classifying sixteen damage mecha­
nisms, respectively.
5.4. Multi-modal-based data analytics
Multiple data fusion is the process of integrating data from different
sources to produce more consistent, accurate, and useful information
than that collected from a singular ‘supplier’. Some research has tended
to focus on how multi-modal can be fused and used to manage safety in
construction, attempting to leverage reports to improve the accuracy of
managing safety. For example, Fang et al. (2021) applied a matching
approach by fusing image features and text features to detect people’s
unsafe behavior from images [124]. While headway is being made to
manage safety by leveraging reports to improve the accuracy, multi-
source data has not been fully utilized and fused for safety.
6. Research challenges
Deep learning has been a important technique to support safety
management in the construction industry. Nonetheless, there is an ex­
istence of challenges that hinder its applications.
6.1. Lack of high-quality database
Deep learning for safety management in the construction industry
requires an extensive database for model training. However, the current
database is limited to specific tasks such as worker and excavator
detection [125,126]. One of the main challenges hindering the creation
of a high-quality database is about establishing the ground-truth label.
The ground-truth labels are critical to the performance of deep
learning methods. Nevertheless, it is difficult to obtain such high-quality
labels due to the limited availability of experienced safety engineers as
well as huge manual ground-truth labeling workloads. For example, the
ground-truth label of psychological signals used for occupational safety
requests people to complete questionnaires, which can be subjective and
error-prone [127,128]. Likewise, the supervised deep learning-based
computer vision algorithms for hazard identification need thousands
of images and corresponding labels. Such training processes requires
human experts to label images and indicate the relationship between
construction resources (e.g., worker, equipment and environment in the
scenes), which may result in ‘false alarms’ in those manual labels. The
false labels can significantly mislead the machine learning process and
generate an issue that computer vision models produce false hazard
detections.
Collecting quality data in complex and dynamic working conditions
is another challenge for high-quality database creation. The harsh work
environment and people’s perspiration and movement can adversely
affect the signal quality of the physiological sensors for occupational
health and safety monitoring. Put simply, a model that performs well in
the laboratory may not effectively work on a construction site in
practice.
Strategies, such as data augmentation (e.g., noise addition, genera­
tive models) and transfer learning, can contribute addressing and
resolving the above issues. Further research into augmentation tech­
niques would produce more extensive data sets, thereby enhancing the
performance of deep learning. In addition, semi-supervised learning or
unsupervised learning is also a promising way to accommodate the
challenges above, because they require smaller samples for training. As
such, future work is needed to develop new semi-supervised learning or
unsupervised learning algorithms for safety.
6.2. Limited ability of deep learning models
Extant studies tend to use a relatively small database to train deep
learning. Therefore, the generalization of the models needs to be
enhanced. Generalization describes the ability of a trained machine
J. Liu et al.
learning model to predict unseen samples accurately during the model
testing phase [129]. Thus, a good generalization is needed for any
practical deep learning algorithms. Normally, the training and testing
samples are different, and the performance of a deep learning model
during testing stage is worse than that in the training process.
Multiple factors may determine the generalization performance of
deep learning models, such as cross-site and cross-device. For example,
there are many wristbands for psychological signal collection on the
market. Although many of them can measure PPG signals, the signal
quality varies by brand. Even for devices from the same company, the
quality of signals is varied with the generation of different models. Thus,
signal processing and machine learning approaches need to handle
cross-device variations effectively. Transfer learning techniques can be
used to alleviate the above limitations by making use of data or machine
learning models from an auxiliary domain or task.
Compared with traditional machine learning algorithms, the supe­
riority of deep learning lies in the elimination of feature engineering
design. However, deep learning models have been identified as being
“black boxes”. In this instance, it is difficult for engineers to understand
the model’s learning processes and how detection is made and to iden­
tify the parameters that need to be adjusted for improving detection
accuracy. To address the above challenges, there is a need for deep
learning models to integrate engineering knowledge for characterizing
engineering.
6.3. Limited to specific applications
Whilst significant development has been made to identify hazards
using deep learning, it is essential to realize that these approaches are
being specific for some certain tasks of safety only, such as defect (e.g.,
crack) detection, or not wearing PPE detection. Put simply, a wide range
of problems has not been well addressed by using deep learning tech­
niques. With the advancement of big data, cloud computing, deep
learning, and other informatics technologies, developing new methods is
needed to discover knowledge from multi-source data and train them to
detect on-site safety issues in construction within a wider context.
7. Recommendation for future work
Addressing the challenges discussed above requires future research.
Noteworthy, existing studies on deep-learning-based safety are
detection-focused and have been limited to certain safety-related prob­
lems. With this in mind, future research is needed to be conducted
within a wider context so that outcomes generated from deep learning
can be learned by site managers to better manage safety issues and then
identify and take actions to control and minimize them. Accordingly, we
have proposed a framework to illustrate how deep learning can be
Fig. 6. Proposed framework.
8
Automation in Construction 140 (2022) 104302
further developed and utilized to mine data for safety management in
the construction industry (Fig. 6).
7.1. Construction of dynamic multi-modal knowledge graph
Currently, safety-related information is stored by different stake­
holders in the form of heterogeneous data (e.g., images, videos and
texts). The data is scattered in multiple systems with diverse, complex
and isolated characteristics. The value of a single data is not prominent.
To address the emerging challenges, future research should focus on
developing new solutions to integrate and use multi-source data for
knowledge extraction and decision-making.
One of the feasible solutions for data integration is developing a
multi-modal knowledge graph, where entities and their interdepen­
dency (relationships) are represented and captured as nodes and edges,
respectively [130]. Both the nodes and edges in a knowledge graph are
high-dimensional. As addressed above [130], Fang et al. [80] are
claimed as a pioneer to embed computer vision into ontology to estab­
lish a knowledge graph for unsafe behavior recognition that is under­
pinned by rule-based reasoning. Multiple unsafe behaviors in the scene
can be identified at one time, however, the developed knowledge graph
is small-scale with very limited reasoning ability.
As noted from Fig. 6, the first step for constructing a multi-modal
knowledge graph is to develop a domain ontology. We have identified
that safety codes, historical safety reports, and expert experience are
pivotal for enabling the ontology (Fig. 6).
The second step is of extracting knowledge from multi-source data.
Deep-learning-based computer vision model is robust in extracting
knowledge (from images), understanding the objects, object-object
interaction, and implying unsafe events (from images/ videos). Simi­
larly, the NLP technique should be developed to identify entities and
entity-entity/attribute relationships. Then, different types of knowledge
graph can be identified from the extracted knowledge accordingly. The
well-defined entities enable filtering relevant information from multi-
source data and constructing precise knowledge graphs [130].
The final stage of the Step 2 relates to the fusion of different
knowledge graph. Put simply, it is regarding how to link different
knowledge graphs and fuse them. We have constructed different
knowledge graphs based on the previous discussion, such as the image/
video knowledge and safety report-based event knowledge graphs.
Additionally, the update of the multi-modal knowledge graph should
also be considered, as multi-source data can be collected using sensors
(e.g., CCTV) and manual inspection daily. Hence, accurately updating
the knowledge graph and reducing the “noise” of the data will be
research challenges in our future work.
J. Liu et al.
7.2. Knowledge graph-based decision-making for safety
With an accurate multi-modal knowledge graph, the final step of the
proposed framework above (Step 3) is with regard to making effective
and efficient decisions to reduce or prevent accidents. Decision-making
in the construction sites needs to consider different relief stakeholders
(e.g., construction unit, supervision). In addition to identifying and
assessing hazards from as-built multi-modal knowledge graph in con­
struction site, future work on decision making needs to focus on: (1)
dynamic prediction of hazards on construction sites; (2) recommenda­
tion of knowledge for hazards prevention; (3) recommendation of
knowledge for hazard intervention (e.g., worker safety training).
We suggest that hazards on construction sites can be predicted in an
early manner. Noteworthy, the characteristics and patterns of an unsafe
event can be noted before it occurs. For example, in the case where a
person enters the excavator dangerous area, their direction will be to­
ward the excavator, which is subject to a closer distance to the worker.
Based on the as-built multi-modal knowledge graph, the characteristics
and patterns of a hazard can be extracted and analyzed so that hazards
can be predicted in an early manner. Nevertheless, the following
research challenges need to be considered: (1) how can an event’s
characteristics and patterns be defined before it occurs? (2) how to
predict the hazard according to the extracted information? and (3) how
early can the hazard be predicted?
Knowledge for different stakeholders can also be recommended to
take actions to prevent accidents as per the as-built multi-modal
knowledge graph. Through the consideration for the type of hazards, the
multi-modal knowledge graph can be used to recommend possible so­
lutions to prevent hazards. In this instance, it is essential for future
research to address three questions: (1) how to define the hazard from
the perspective of knowledge? (2) how to design a recommendation
system and then recommend knowledge to the stakeholders based on the
multi-modal knowledge graph? and (3) how to evaluate the effective­
ness and feasibility of the recommendation system on hazards solution?
8. Conclusion
We have placed a systematic review in this paper to understand the
developments of the deep-learning-related studies and their applications
to construction safety, focusing on four key areas according to different
types of data: (1) image/video-based; (2) text-based; (3) non-visual
sensor-based; (4) multi-modal-based. Furthermore, our review has
provided an insight into the challenges confronted by research and
practice. These involve: (1) lack high-quality database; (2) inadequate
ability of deep learning models; and (3) limited to specific applications.
To further manage safety through the use of deep learning, we have
identified that our future study needs to focus on utilizing multi-source
data and extracting knowledge from the data. In doing so, there is also a
need of paying attention to: (1) the construction of dynamic multi-modal
knowledge graph; and (2) knowledge graph-based decision-making for
safety management. To this end, the contributions of our research are
threefold, as we have: (1) provided novel insights into examining the
development of deep learning for studying safety in construction from
the perspective of data utilization; (2) identified the research challenges
of adopting deep learning; and (3) presented new research opportunities
to support and facilitate deep learning-based data analytics for safety in
construction.
Declaration of Competing Interest
This manuscript has not been published or presented elsewhere in
part or in entirety and is not under consideration by another journal. We
have read and understood your journal’s policies, and we believe that
neither the manuscript nor the study violates any of these. There are no
conflicts of interest to declare.
9
Automation in Construction 140 (2022) 104302
Acknowledgement
The authors would like to acknowledge the financial support pro­
vided by the National Natural Science Foundation of China (Grant No.
51978302, U21A20151, 71732001) and the China Scholarship Council.
References
[1] S. Guo, L. Ding, Y. Zhang, M.J. Skibniewski, K. Liang, Hybrid recommendation
approach for behavior modification in the Chinese construction industry,
J. Constr. Eng. Manag. 145 (6) (2019) 04019035, https://doi.org/10.1061/
(ASCE)CO.1943-7862.0001665.
[2] P.E.D. Love, L. Ika, B. Luo, Y. Zhou, B. Zhong, W. Fang, Rework, failures, and
unsafe behavior: Moving toward an error management mindset in construction,
IEEE Trans. Eng. Manag. (2020) 1–13, https://doi.org/10.1109/
TEM.2020.2982463. In press.
[3] P.E. Love, P. Teo, J. Smith, F. Ackermann, Y. Zhou, The nature and severity of
workplace injuries in construction: engendering operational benchmarking,
Ergonomics 62 (10) (2019) 1273–1288, https://doi.org/10.1080/
00140139.2019.1644379.
[4] Health and Safety Executive, Workplace Fatal Injuries in Great Britain 2021,
2021. http://www.hse.gov.uk/statistics/pdf/fatalinjuries.pdf (accessed Feburary
14, 2022).
[5] U.S. Bureau of Labor Statistics, Census of Fatal Occupational Injuries. htt
ps://www.bls.gov/iif/oshwc/cfoi/cftb0330.htm, 2019 (accessed Feburary 14,
2022).
[6] Ministry of Housing and Urban-Rural Development of the People’’s Republic of
China, Circular of the General Office of the Ministry of Housing and Urban-Rural
Development on the production safety accidents of housing and municipal
engineering in 2019. https://www.mohurd.gov.cn/gongkai/fdzdgknr/tzgg/202
006/20200624_246031.html, 2020 (accessed Feburary 14, 2022).
[7] W. Fang, L. Ding, P.E.D. Love, H. Luo, H. Li, F. Peña-Mora, B. Zhong, C. Zhou,
Computer vision applications in construction safety assurance, Autom. Constr.
110 (2020), 103013, https://doi.org/10.1016/j.autcon.2019.103013.
[8] H. Luo, J. Liu, W. Fang, P.E. Love, Q. Yu, Z. Lu, Real-time smart video
surveillance to manage safety: a case study of a transport mega-project, Adv. Eng.
Inform. 45 (2020) 101100, https://doi.org/10.1016/j.aei.2020.101100.
[9] C. Zhou, H. Luo, W. Fang, R. Wei, L. Ding, Cyber-physical-system-based safety
monitoring for blind hoisting with the internet of things: a case study, Autom.
Constr. 97 (2019) 138–150, https://doi.org/10.1016/j.autcon.2018.10.017.
[10] S. Tang, D.R. Shelden, C.M. Eastman, P. Pishdad-Bozorgi, X. Gao, A review of
building information modeling (BIM) and the internet of things (IoT) devices
integration: present status and future trends, Autom. Constr. 101 (2019)
127–139, https://doi.org/10.1016/j.autcon.2019.01.020.
[11] C.J. Anumba, A. Akanmu, X. Yuan, C. Kan, Cyber—physical systems development
for construction applications, Front. Eng. Manag. 8 (1) (2021) 72–87, https://doi.
org/10.1007/s42524-020-0130-4.
[12] T. Hartmann, A. Trappey, Advanced engineering informatics-philosophical and
methodological foundations with examples from civil and construction
engineering, Develop. Built Environ. 4 (2020), 100020, https://doi.org/10.1016/
j.dibe.2020.100020.
[13] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015)
436–444, https://doi.org/10.1038/nature14539.
[14] L. Ding, W. Fang, H. Luo, P.E. Love, B. Zhong, X. Ouyang, A deep hybrid learning
model to detect unsafe behavior: integrating convolution neural networks and
long short-term memory, Autom. Constr. 86 (2018) 118–124, https://doi.org/
10.1016/j.autcon.2017.11.002.
[15] Q. Fang, H. Li, X. Luo, L. Ding, H. Luo, T.M. Rose, W. An, Detecting non-hardhat-
use by a deep learning method from far-field surveillance videos, Autom. Constr.
85 (2018) 1–9, https://doi.org/10.1016/j.autcon.2017.09.018.
[16] B. Zhong, H. Wu, L. Ding, P.E. Love, H. Li, H. Luo, L. Jiao, Mapping computer
vision research in construction: developments, knowledge gaps and implications
for research, Autom. Constr. 107 (2019) 102919, https://doi.org/10.1016/j.
autcon.2019.102919.
[17] T.D. Akinosho, L.O. Oyedele, M. Bilal, A.O. Ajayi, M.D. Delgado, O.O. Akinade, A.
A. Ahmed, Deep learning in the construction industry: a review of present status
and future innovations, J. Build. Eng. 32 (2020), 101827, https://doi.org/
10.1016/j.jobe.2020.101827.
[18] L. Hou, H. Chen, G.K. Zhang, X. Wang, Deep learning-based applications for
safety management in the AEC industry: a review, Appl. Sci. 11 (2) (2021) 821,
https://doi.org/10.3390/app11020821.
[19] A. Pal, S.-H. Hsieh, Deep-learning-based visual data analytics for smart
construction management, Autom. Constr. 131 (2021), 103892, https://doi.org/
10.1016/j.autcon.2021.103892.
[20] A. Esteva, A. Robicquet, B. Ramsundar, V. Kuleshov, M. DePristo, K. Chou, C. Cui,
G. Corrado, S. Thrun, J. Dean, A guide to deep learning in healthcare, Nat. Med.
25 (1) (2019) 24–29, https://doi.org/10.1038/s41591-018-0316-z.
[21] S. Ramos, S. Gehrig, P. Pinggera, U. Franke, C. Rother, Detecting unexpected
obstacles for self-driving cars: Fusing deep learning and geometric modeling, in:
2017 IEEE Intelligent Vehicles Symposium (IV), 2017, pp. 1025–1032, https://
doi.org/10.1109/IVS.2017.7995849.
[22] D.W. Otter, J.R. Medina, J.K. Kalita, A survey of the usages of deep learning for
natural language processing, IEEE Trans. Neural Networks Learn. Syst. 32 (2)
(2020) 604–624, https://doi.org/10.1109/TNNLS.2020.2979670.
J. Liu et al.
[23] K. Gopalakrishnan, S.K. Khaitan, A. Choudhary, A. Agrawal, Deep convolutional
neural networks with transfer learning for computer vision-based data-driven
pavement distress detection, Constr. Build. Mater. 157 (2017) 322–330, https://
doi.org/10.1016/j.conbuildmat.2017.09.110.
[24] L. Torrey, J. Shavlik, Transfer learning, handbook of research on machine
learning applications and trends: algorithms, methods, and techniques, IGI Global
(2010) 242–264, https://doi.org/10.4018/978-1-60566-766-9.ch011.
[25] S.J. Pan, Q. Yang, A survey on transfer learning, IEEE Trans. Knowl. Data Eng. 22
(10) (2009) 1345–1359, https://doi.org/10.1109/TKDE.2009.191.
[26] J. Han, S. Kang, Active learning with missing values considering imputation
uncertainty, Knowl.-Based Syst. 224 (2021), 107079, https://doi.org/10.1016/j.
knosys.2021.107079.
[27] J. Kim, J. Hwang, S. Chi, J. Seo, Towards database-free vision-based monitoring
on construction sites: a deep active learning approach, Autom. Constr. 120
(2020), 103376, https://doi.org/10.1016/j.autcon.2020.103376.
[28] C. Shorten, T.M. Khoshgoftaar, A survey on image data augmentation for deep
learning, J. Big Data 6 (1) (2019) 1–48, https://doi.org/10.1186/s40537-019-
0197-0.
[29] W. Fang, P.E. Love, H. Luo, L. Ding, Computer vision for behaviour-based safety
in construction: a review and future directions, Adv. Eng. Inform. 43 (2020)
100980, https://doi.org/10.1016/j.aei.2019.100980.
[30] A. Esteva, B. Kuprel, R.A. Novoa, J. Ko, S.M. Swetter, H.M. Blau, S. Thrun,
Dermatologist-level classification of skin cancer with deep neural networks,
Nature 542 (7639) (2017) 115–118, https://doi.org/10.1038/nature21056.
[31] H.A. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, A. Kalloo,
A.B.H. Hassen, L. Thomas, A. Enk, Man against machine: diagnostic performance
of a deep learning convolutional neural network for dermoscopic melanoma
recognition in comparison to 58 dermatologists, Ann. Oncol. 29 (8) (2018)
1836–1842, https://doi.org/10.1093/annonc/mdy166.
[32] A. Jamaludin, T. Kadir, A. Zisserman, SpineNet: automatically pinpointing
classification evidence in spinal MRIs, in: International Conference on Medical
Image Computing and Computer-Assisted Intervention, Springer, 2016,
pp. 166–175, https://doi.org/10.1007/978-3-319-46723-8_20.
[33] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: towards real-time object detection
with region proposal networks, IEEE Trans. Pattern Anal. Machine Intell. 39 (6)
(2017) 1137–1149, https://doi.org/10.1109/TPAMI.2016.2577031.
[34] J. Dai, Y. Li, K. He, J. Sun, R-fcn: object detection via region-based fully
convolutional networks, Adv. Neural Inf. Proces. Syst. 29 (2016), in: https://proc
eedings.neurips.cc/paper/2016/file/577ef1154f3240ad5b9b413aa7346a1e-
Paper.pdf.
[35] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A.C. Berg, Ssd: single
shot multibox detector, in: European Conference on Computer Vision, Springer,
2016, pp. 21–37, https://doi.org/10.1007/978-3-319-46448-0_2.
[36] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: unified, real-
time object detection, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2016)
779–788, doi:110.1109/CVPR.2016.91.
[37] T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object
detection, in: Proceedings of the IEEE International Conference on Computer
Vision, 2017, pp. 2980–2988. https://openaccess.thecvf.com/content_ICCV_2
017/papers/Lin_Focal_Loss_for_ICCV_2017_paper.pdf.
[38] M. Danelljan, A. Robinson, F.S. Khan, M. Felsberg, Beyond correlation filters:
learning continuous convolution operators for visual tracking, in: European
Conference on Computer Vision, Springer, 2016, pp. 472–488, https://doi.org/
10.1007/978-3-319-46454-1_29.
[39] N. Wang, D.Y. Yeung, Learning a deep compact image representation for visual
tracking, Adv. Neural Inf. Proces. Syst. 26 (2013) 809–817, in: https://proc
eedings.neurips.cc/paper/2013/file/dc6a6489640ca02b0d42dabeb8e46bb7-Pa
per.pdf.
[40] L. Wang, W. Ouyang, X. Wang, H. Lu, Visual tracking with fully convolutional
networks, in: Proceedings of the IEEE International Conference on Computer
Vision, 2015, pp. 3119–3127. https://openaccess.thecvf.com/content_iccv_
2015/papers/Wang_Visual_Tracking_With_ICCV_2015_paper.pdf.
[41] H. Nam, B. Han, Learning multi-domain convolutional neural networks for visual
tracking, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2016) 4293–4302.
https://openaccess.thecvf.com/content_cvpr_2016/papers/Nam_Learning_Multi
-Domain_Convolutional_CVPR_2016_paper.pdf.
[42] K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask r-cnn, in: Proceedings of the IEEE
International Conference on Computer Vision, 2017, pp. 2961–2969. https://op
enaccess.thecvf.com/content_ICCV_2017/papers/He_Mask_R-CNN_ICCV_2017_p
aper.pdf.
[43] L.-C. Chen, A. Hermans, G. Papandreou, F. Schroff, P. Wang, H. Adam, Masklab:
instance segmentation by refining object detection with semantic and direction
features, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2018) 4013–4022. http
s://openaccess.thecvf.com/content_cvpr_2018/papers/Chen_MaskLab_Instance_
Segmentation_CVPR_2018_paper.pdf.
[44] D. Bolya, C. Zhou, F. Xiao, Y.J. Lee, Yolact: real-time instance segmentation, in:
Proceedings of the IEEE/CVF International Conference on Computer Vision,
2019, pp. 9157–9166. http://openaccess.thecvf.com/content_ICCV_2019/pape
rs/Bolya_YOLACT_Real-Time_Instance_Segmentation_ICCV_2019_paper.pdf.
[45] S.-E. Wei, V. Ramakrishna, T. Kanade, Y. Sheikh, Convolutional pose machines,
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (2016) 4724–4732. https://ope
naccess.thecvf.com/content_cvpr_2016/papers/Wei_Convolutional_Pose_Mach
ines_CVPR_2016_paper.pdf.
[46] A. Newell, K. Yang, J. Deng, Stacked hourglass networks for human pose
estimation, in: European Conference on Computer Vision, Springer, 2016,
pp. 483–499, https://doi.org/10.1007/978-3-319-46484-8_29.
10
Automation in Construction 140 (2022) 104302
[47] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, J. Sun, Cascaded pyramid network for
multi-person pose estimation, Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(2018) 7103–7112. https://openaccess.thecvf.com/content_cvpr_2018/papers/
Chen_Cascaded_Pyramid_Network_CVPR_2018_paper.pdf.
[48] K. Sun, B. Xiao, D. Liu, J. Wang, Deep high-resolution representation learning for
human pose estimation, in: Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 2019, pp. 5693–5703. https://opena
ccess.thecvf.com/content_CVPR_2019/papers/Sun_Deep_High-Resolution_Rep
resentation_Learning_for_Human_Pose_Estimation_CVPR_2019_paper.pdf.
[49] C. Zhu, F. Chen, Z. Shen, M. Savvides, Soft anchor-point object detection, in:
Computer Vision–ECCV 2020: 16th European conference, Glasgow, UK, August
23–28, 2020, Proceedings, Part IX 16, 2020, pp. 91–107, https://doi.org/
10.1007/978-3-030-58545-7_6.
[50] Z. Tian, C. Shen, H. Chen, T. He, Fcos: fully convolutional one-stage object
detection, in: Proceedings of the IEEE/CVF International Conference on Computer
Vision, 2019, pp. 9627–9636. https://openaccess.thecvf.com/content_ICCV_
2019/papers/Tian_FCOS_Fully_Convolutional_One-Stage_Object_Detection_ICCV_
2019_paper.pdf.
[51] C. Zhu, Y. He, M. Savvides, Feature selective anchor-free module for single-shot
object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, 2019, pp. 840–849. https://openaccess.thecvf.com/con
tent_CVPR_2019/papers/Zhu_Feature_Selective_Anchor-Free_Module_for_Single
-Shot_Object_Detection_CVPR_2019_paper.pdf.
[52] J. He, Z. Huang, N. Wang, Z. Zhang, Learnable graph matching: incorporating
graph partitioning with deep feature learning for multiple object tracking, in:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2021, pp. 5299–5309. https://openaccess.thecvf.com/content
/CVPR2021/papers/He_Learnable_Graph_Matching_Incorporating_Graph_
Partitioning_With_Deep_Feature_Learning_CVPR_2021_paper.pdf.
[53] B. Shuai, A. Berneshawi, X. Li, D. Modolo, J. Tighe, SiamMOT: Siamese multi-
object tracking, in: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, 2021, pp. 12372–12382. https://openaccess.thecvf.co
m/content/CVPR2021/papers/Shuai_SiamMOT_Siamese_Multi-Object_Tracking_
CVPR_2021_paper.pdf.
[54] K.G. Quach, P. Nguyen, H. Le, T.D. Truong, C.N. Duong, M.T. Tran, K. Luu,
DyGLIP: a dynamic graph model with link prediction for accurate multi-camera
multiple object tracking, in: Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 2021, pp. 13784–13793. https
://openaccess.thecvf.com/content/CVPR2021/papers/Quach_DyGLIP_A_Dyna
mic_Graph_Model_With_Link_Prediction_for_Accurate_CVPR_2021_paper.pdf.
[55] X. Zhang, H. Xu, H. Mo, J. Tan, C. Yang, L. Wang, W. Ren, Dcnas: densely
connected neural architecture search for semantic image segmentation, in:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, 2021, pp. 13956–13967. https://openaccess.thecvf.com/content
/CVPR2021/papers/Zhang_DCNAS_Densely_Connected_Neural_Architecture
_Search_for_Semantic_Image_Segmentation_CVPR_2021_paper.pdf.
[56] Y. Yang, B. Lai, S. Soatto, DyStaB: Unsupervised object segmentation via
dynamic-static bootstrapping, in: Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 2021, pp. 2826–2836. https://opena
ccess.thecvf.com/content/CVPR2021/papers/Yang_DyStaB_Unsupervised_Obje
ct_Segmentation_via_Dynamic-Static_Bootstrapping_CVPR_2021_paper.pdf.
[57] Y. Yuan, S.E. Wei, T. Simon, K. Kitani, J. Saragih, SimPoE: simulated character
control for 3D human pose estimation, in: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2021, pp. 7159–7169.
https://openaccess.thecvf.com/content/CVPR2021/papers/Yuan_SimPoE_
Simulated_Character_Control_for_3D_Human_Pose_Estimation_CVPR_2021_paper.
pdf.
[58] W. Mao, Z. Tian, X. Wang, C. Shen, FCPose: fully convolutional multi-person pose
estimation with dynamic instance-aware convolutions, in: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021,
pp. 9034–9043. https://openaccess.thecvf.com/content/CVPR2021/papers
/Mao_FCPose_Fully_Convolutional_Multi-Person_Pose_Estimation_With_Dynamic_
Instance-Aware_Convolutions_CVPR_2021_paper.pdf.
[59] N. Indurkhya, F.J. Damerau, Handbook of Natural Language Processing, CRC
Press, 2010, https://doi.org/10.1201/9781420085938.
[60] Y. Zou, A. Kiviniemi, S.W. Jones, Retrieving similar cases for construction project
risk management using natural language processing techniques, Autom. Constr.
80 (2017) 66–76, https://doi.org/10.1016/j.autcon.2017.04.003.
[61] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of Deep
Bidirectional Transformers for Language Understanding, arXiv preprint. arXiv:1
810.04805, 2018. https://arxiv.org/pdf/1810.04805.pdf.
[62] Z. Huang, W. Xu, K. Yu, Bidirectional LSTM-CRF Models for Sequence Tagging,
arXiv Preprint. arXiv:1508.01991, 2015. https://arxiv.org/pdf/1508.01991.pdf.
[63] J. Guo, Y. Fan, Q. Ai, W.B. Croft, A deep relevance matching model for ad-hoc
retrieval, in: Proceedings of the 25th ACM International on Conference on
Information and Knowledge Management, 2016, pp. 55–64, https://doi.org/
10.1145/2983323.2983769.
[64] H. Zamani, M. Dehghani, W.B. Croft, E. Learned-Miller, J. Kamps, From neural re-
ranking to neural ranking: Learning a sparse representation for inverted indexing,
in: Proceedings of the 27th ACM International Conference on Information and
Knowledge Management, 2018, pp. 497–506, https://doi.org/10.1145/
3269206.3271800.
[65] W.M. Bramer, M.K. Rethlefsen, J. Kleijnen, O.H. Franco, Optimal database
combinations for literature searches in systematic reviews: a prospective
exploratory study, Syst. Rev. 6 (1) (2017) 1–12, https://doi.org/10.1186/s13643-
017-0644-y.
J. Liu et al.
[66] H.W. Heinrich, Industrial Accident Prevention. A Scientific Approach, Industrial
Accident Prevention. A Scientific Approach, Second edition, 1941, https://doi.
org/10.2105/ajph.22.1.119-b.
[67] H. Li, M. Lu, S.-C. Hsu, M. Gray, T. Huang, Proactive behavior-based safety
management for construction safety improvement, Saf. Sci. 75 (2015) 107–117,
https://doi.org/10.1016/j.ssci.2015.01.013.
[68] H. Li, X. Li, X. Luo, J. Siebert, Investigation of the causality patterns of non-helmet
use behavior of construction workers, Autom. Constr. 80 (2017) 95–103, https://
doi.org/10.1016/j.autcon.2017.02.006.
[69] N.D. Nath, A.H. Behzadan, S.G. Paal, Deep learning for site safety: real-time
detection of personal protective equipment, Autom. Constr. 112 (2020), 103085,
https://doi.org/10.1016/j.autcon.2020.103085.
[70] J. Shen, X. Xiong, Y. Li, W. He, P. Li, X. Zheng, Detecting safety helmet wearing
on construction sites with bounding-box regression and deep transfer learning,
Comp. Aided Civil Infrastruct. Eng. 36 (2) (2021) 180–196, https://doi.org/
10.1111/mice.12579.
[71] J. Wu, N. Cai, W. Chen, H. Wang, G. Wang, Automatic detection of hardhats worn
by construction personnel: a deep learning approach and benchmark dataset,
Autom. Constr. 106 (2019), 102894, https://doi.org/10.1016/j.
autcon.2019.102894.
[72] Q. Fang, H. Li, X. Luo, L. Ding, H. Luo, C. Li, Computer vision aided inspection on
falling prevention measures for steeplejacks in an aerial environment, Autom.
Constr. 93 (2018) 148–164, https://doi.org/10.1016/j.autcon.2018.05.022.
[73] W. Fang, B. Zhong, N. Zhao, P.E. Love, H. Luo, J. Xue, S. Xu, A deep learning-
based approach for mitigating falls from height with computer vision:
convolutional neural network, Adv. Eng. Inform. 39 (2019) 170–177, https://doi.
org/10.1016/j.aei.2018.12.005.
[74] S. Bang, Y. Hong, H. Kim, Proactive proximity monitoring with instance
segmentation and unmanned aerial vehicle-acquired video-frame prediction,
Comp. Aided Civil Infrastruct. Eng. 36 (6) (2021) 800–816, https://doi.org/
10.1111/mice.12672.
[75] D. Kim, M. Liu, S. Lee, V.R. Kamat, Remote proximity monitoring between mobile
construction resources using camera-mounted UAVs, Autom. Constr. 99 (2019)
168–182, https://doi.org/10.1016/j.autcon.2018.12.014.
[76] X. Yan, H. Zhang, H. Li, Computer vision-based recognition of 3D relationship
between construction entities for monitoring struck-by accidents, Comp. Aided
Civil Infrastruct. Eng. 35 (9) (2020) 1023–1038, https://doi.org/10.1111/
mice.12536.
[77] S. Han, S. Lee, F. Peña-Mora, Comparative study of motion features for similarity-
based modeling and classification of unsafe actions in construction, J. Comput.
Civ. Eng. 28 (5) (2014) A4014005, https://doi.org/10.1061/(ASCE)CP.1943-
5487.0000339.
[78] X. Luo, H. Li, X. Yang, Y. Yu, D. Cao, Capturing and understanding workers’
activities in far-field surveillance videos with deep action recognition and
Bayesian nonparametric learning, Comp. Aided Civil Infrastruct. Eng. 34 (4)
(2019) 333–351, https://doi.org/10.1111/mice.12419.
[79] D. Roberts, W.T. Calderon, S. Tang, M. Golparvar-Fard, Vision-based construction
worker activity analysis informed by body posture, J. Comput. Civ. Eng. 34 (4)
(2020) 04020017, https://doi.org/10.1061/(ASCE)CP.1943-5487.0000898.
[80] W. Fang, L. Ma, P.E. Love, H. Luo, L. Ding, A. Zhou, Knowledge graph for
identifying hazards on construction sites: integrating computer vision with
ontology, Autom. Constr. 119 (2020) 103310, https://doi.org/10.1016/j.
autcon.2020.103310.
[81] Z. Pan, C. Su, Y. Deng, J. Cheng, Video2Entities: a computer vision-based entity
extraction framework for updating the architecture, engineering and construction
industry knowledge graphs, Autom. Constr. 125 (2021), 103617, https://doi.org/
10.1016/j.autcon.2021.103617.
[82] R. Xiong, Y. Song, H. Li, Y. Wang, Onsite video mining for construction hazards
identification with visual relationships, Adv. Eng. Inform. 42 (2019), 100966,
https://doi.org/10.1016/j.aei.2019.100966.
[83] B. Zhong, H. Li, H. Luo, J. Zhou, W. Fang, X. Xing, Ontology-based semantic
modeling of knowledge in construction: classification and identification of
hazards implied in images, J. Constr. Eng. Manag. 146 (4) (2020) 04020013,
https://doi.org/10.1061/(ASCE)CO.1943-7862.0001767.
[84] S. Tang, D. Roberts, M. Golparvar-Fard, Human-object interaction recognition for
automatic construction site safety inspection, Autom. Constr. 120 (2020),
103356, https://doi.org/10.1016/j.autcon.2020.103356.
[85] W. Fang, L. Ding, H. Luo, P.E. Love, Falls from heights: a computer vision-based
approach for safety harness detection, Autom. Constr. 91 (2018) 53–61, https://
doi.org/10.1016/j.autcon.2018.02.018.
[86] H. Son, H. Seong, H. Choi, C. Kim, Real-time vision-based warning system for
prevention of collisions between workers and heavy equipment, J. Comput. Civ.
Eng. 33 (5) (2019) 04019029, https://doi.org/10.1061/(ASCE)CP.1943-
5487.0000845.
[87] Z.H. Lin, A.Y. Chen, S.H. Hsieh, Temporal image analytics for abnormal
construction activity identification, Autom. Constr. 124 (2021), 103572, https://
doi.org/10.1016/j.autcon.2021.103572.
[88] H. Wu, B. Zhong, H. Li, P. Love, X. Pan, N. Zhao, Combining computer vision with
semantic reasoning for on-site safety management in construction, J. Build. Eng.
42 (2021), 103036, https://doi.org/10.1016/j.jobe.2021.103036.
[89] D. Balageas, C.-P. Fritzen, A. Güemes, Structural Health Monitoring, John Wiley
& Sons, 2010. ISBN:978-1-905209-01-9.
[90] S. Park, S. Bang, H. Kim, H. Kim, Patch-based crack detection in black box images
using convolutional neural networks, J. Comput. Civ. Eng. 33 (3) (2019)
04019017, https://doi.org/10.1061/(ASCE)CP.1943-5487.0000831.
11
Automation in Construction 140 (2022) 104302
[91] D. Kang, Y.-J. Cha, Autonomous UAVs for structural health monitoring using deep
learning and an ultrasonic beacon system with geo-tagging, Comp. Aided Civil
Infrastruct. Eng. 33 (10) (2018) 885–902, https://doi.org/10.1111/mice.12375.
[92] C. Zhang, C.-C. Chang, M. Jamshidi, Concrete bridge surface damage detection
using a single-stage detector, Comp. Aided Civil Infrastruct. Eng. 35 (4) (2020)
389–409, https://doi.org/10.1111/mice.12500.
[93] S. Bang, S. Park, H. Kim, H. Kim, Encoder–decoder network for pixel-level road
crack detection in black-box images, Comp. Aided Civil Infrastruct. Eng. 34 (8)
(2019) 713–727, https://doi.org/10.1111/mice.12440.
[94] K. Jang, Y.-K. An, B. Kim, S. Cho, Automated crack evaluation of a high-rise
bridge pier using a ring-type climbing robot, Comp. Aided Civil Infrastruct. Eng.
36 (1) (2021) 14–29, https://doi.org/10.1111/mice.12550.
[95] S. Jiang, J. Zhang, Real-time crack assessment using deep neural networks with
wall-climbing unmanned aerial system, Comp. Aided Civil Infrastruct. Eng. 35 (6)
(2020) 549–564, https://doi.org/10.1111/mice.12519.
[96] R. Kalfarisi, Z.Y. Wu, K. Soh, Crack detection and segmentation using deep
learning with 3D reality mesh model for quantitative assessment and integrated
visualization, J. Comput. Civ. Eng. 34 (3) (2020) 04020010, https://doi.org/
10.1061/(ASCE)CP.1943-5487.0000890.
[97] J. Liu, X. Yang, S. Lau, X. Wang, S. Luo, V.C.S. Lee, L. Ding, Automated pavement
crack detection and segmentation based on two-step convolutional neural
network, Comp. Aided Civil Infrastruct. Eng. 35 (11) (2020) 1291–1305, https://
doi.org/10.1111/mice.12622.
[98] Q. Song, Y. Wu, X. Xin, L. Yang, M. Yang, H. Chen, C. Liu, M. Hu, X. Chai, J. Li,
Real-time tunnel crack analysis system via deep learning, IEEE Access 7 (2019)
64186–64197, https://doi.org/10.1109/ACCESS.2019.2916330.
[99] X. Zhang, D. Rajan, B. Story, Concrete crack detection using context-aware deep
semantic segmentation network, Comp. Aided Civil Infrastruct. Eng. 34 (11)
(2019) 951–971, https://doi.org/10.1111/mice.12477.
[100] F. Wei, G. Yao, Y. Yang, Y. Sun, Instance-level recognition and quantification for
concrete surface bughole based on deep learning, Autom. Constr. 107 (2019),
102920, https://doi.org/10.1016/j.autcon.2019.102920.
[101] H. Wu, L. Yao, Z. Xu, Y. Li, X. Ao, Q. Chen, Z. Li, B. Meng, Road pothole extraction
and safety evaluation by integration of point cloud and images derived from
mobile mapping sensors, Adv. Eng. Inform. 42 (2019), 100936, https://doi.org/
10.1016/j.aei.2019.100936.
[102] D.J. Atha, M.R. Jahanshahi, Evaluation of deep learning approaches based on
convolutional neural networks for corrosion detection, Struct. Health Monit. 17
(5) (2018) 1110–1128, doi:10.1177%2F1475921717737051.
[103] T.-C. Huynh, J.-H. Park, H.-J. Jung, J.-T. Kim, Quasi-autonomous bolt-loosening
detection method using vision-based deep learning and image processing, Autom.
Constr. 105 (2019), 102844, https://doi.org/10.1016/j.autcon.2019.102844.
[104] S.S. Kumar, D.M. Abraham, M.R. Jahanshahi, T. Iseley, J. Starr, Automated defect
classification in sewer closed circuit television inspections using deep
convolutional neural networks, Autom. Constr. 91 (2018) 273–283, https://doi.
org/10.1016/j.autcon.2018.03.028.
[105] M. Wang, S.S. Kumar, J.C.P. Cheng, Automated sewer pipe defect tracking in
CCTV videos based on defect detection and metric learning, Autom. Constr. 121
(2021), 103438, https://doi.org/10.1016/j.autcon.2020.103438.
[106] G. Pan, Y. Zheng, S. Guo, Y. Lv, Automatic sewer pipe defect semantic
segmentation based on improved U-net, Autom. Constr. 119 (2020) 103383,
https://doi.org/10.1016/j.autcon.2020.103383.
[107] Y. Jang, Y. Ahn, Y. Kim Ha, Estimating compressive strength of concrete using
deep convolutional neural networks with digital microscope images, J. Comput.
Civ. Eng. 33 (3) (2019) 04019018, https://doi.org/10.1061/(ASCE)CP.1943-
5487.0000837.
[108] W. Wang, P. Shi, L. Deng, H. Chu, X. Kong, Residual strength evaluation of
corroded textile-reinforced concrete by the deep learning-based method,
Materials 13 (14) (2020) 3226, https://doi.org/10.3390/ma13143226.
[109] M. Azimi, G. Pekcan, Structural health monitoring using extremely compressed
data through deep learning, Comp. Aided Civil Infrastruct. Eng. 35 (6) (2020)
597–614, https://doi.org/10.1111/mice.12517.
[110] Y.-Z. Lin, Z.-H. Nie, H.-W. Ma, Structural damage detection with automatic
feature-extraction through deep learning, Comp. Aided Civil Infrastruct. Eng. 32
(12) (2017) 1025–1046, https://doi.org/10.1111/mice.12313.
[111] F. Ni, J. Zhang, M.N. Noori, Deep learning for data anomaly detection and data
compression of a long-span suspension bridge, Comp. Aided Civil Infrastruct. Eng.
35 (7) (2020) 685–700, https://doi.org/10.1111/mice.12528.
[112] S.O. Sajedi, X. Liang, Vibration-based semantic damage segmentation for large-
scale structural health monitoring, Comp. Aided Civil Infrastruct. Eng. 35 (6)
(2020) 579–596, https://doi.org/10.1111/mice.12523.
[113] F. Zhang, A hybrid structured deep neural network with Word2Vec for
construction accident causes classification, Int. J. Constr. Manag. (2019) 1–21,
https://doi.org/10.1080/15623599.2019.1683692.
[114] J. Zhang, L. Zi, Y. Hou, D. Deng, W. Jiang, M. Wang, A C-BiLSTM approach to
classify construction accident reports, Appl. Sci. (Switzerland) 10 (17) (2020)
5754, https://doi.org/10.3390/app10175754.
[115] H. Baker, M.R. Hallowell, A.J.P. Tixier, Automatically learning construction
injury precursors from text, Autom. Constr. 118 (2020), 103145, https://doi.org/
10.1016/j.autcon.2020.103145.
[116] W. Fang, H. Luo, S. Xu, P.E.D. Love, Z. Lu, C. Ye, Automated text classification of
near-misses from safety reports: An improved deep learning approach, Adv. Eng.
Inform. 44 (2020), 101060, https://doi.org/10.1016/j.aei.2020.101060.
[117] B. Zhong, X. Pan, P.E.D. Love, L. Ding, W. Fang, Deep learning and network
analysis: classifying and visualizing accident narratives in construction, Autom.
Constr. 113 (2020), 103089, https://doi.org/10.1016/j.autcon.2020.103089.
J. Liu et al.
[118] B. Zhong, X. Pan, P.E.D. Love, J. Sun, C. Tao, Hazard analysis: a deep learning and
text mining framework for accident prevention, Adv. Eng. Inform. 46 (2020),
101152, https://doi.org/10.1016/j.aei.2020.101152.
[119] D. Feng, H. Chen, A small samples training framework for deep learning-based
automatic information extraction: case study of construction accident news
reports analysis, Adv. Eng. Inform. 47 (2021), 101256, https://doi.org/10.1016/
j.aei.2021.101256.
[120] D. Guo, J. Li, S.-H. Jiang, X. Li, Z. Chen, Intelligent assistant driving method for
tunnel boring machine based on big data, Acta Geotech. (2021) 1–12, https://doi.
org/10.1007/s11440-021-01327-1.
[121] Q. Wang, X. Xie, H. Yu, M.A. Mooney, Predicting slurry pressure balance with a
long short-term memory recurrent neural network in difficult ground condition,
Comp. Intel. Neurosci. 2021 (2021), https://doi.org/10.1155/2021/6678355.
[122] Y. Zhang, G. Gong, H. Yang, W. Li, J. Liu, Precision versus intelligence:
autonomous supporting pressure balance control for slurry shield tunnel boring
machines, Autom. Constr. 114 (2020), 103173, https://doi.org/10.1016/j.
autcon.2020.103173.
[123] C. Zhou, H. Xu, L. Ding, L. Wei, Y. Zhou, Dynamic prediction for attitude and
position in shield tunneling: a deep learning method, Autom. Constr. 105 (2019),
102840, https://doi.org/10.1016/j.autcon.2019.102840.
[124] W. Fang, P.E. Love, L. Ding, S. Xu, T. Kong, H. Li, Computer vision and deep
learning to manage safety in construction: matching images of unsafe behavior
12
Automation in Construction 140 (2022) 104302
and semantic rules, IEEE Trans. Eng. Manag. (2021), https://doi.org/10.1109/
TEM.2021.3093166.
[125] D. Roberts, M. Golparvar-Fard, End-to-end vision-based detection, tracking and
activity analysis of earthmoving equipment filmed at ground level, Autom.
Constr. 105 (2019), 102811, https://doi.org/10.1016/j.autcon.2019.04.006.
[126] B. Xiao, S.-C. Kang, Development of an image data set of construction machines
for deep learning object detection, J. Comput. Civ. Eng. 35 (2) (2021) 05020005,
https://doi.org/10.1061/(ASCE)CP.1943-5487.0000945.
[127] C.R. Ahn, S. Lee, C. Sun, H. Jebelli, K. Yang, B. Choi, Wearable sensing technology
applications in construction safety and health, J. Constr. Eng. Manag. 145 (11)
(2019) 03119007, https://doi.org/10.1061/(ASCE)CO.1943-7862.0001708.
[128] I. Awolusi, E. Marks, M. Hallowell, Wearable technology for personalized
construction safety monitoring and trending: review of applicable devices,
Autom. Constr. 85 (2018) 96–106, https://doi.org/10.1016/j.
autcon.2017.10.010.
[129] M.T. Musavi, K.H. Chan, D.M. Hummels, K. Kalantri, On the generalization ability
of neural network classifiers, IEEE Trans. Pattern Anal. Mach. Intell. 16 (6) (1994)
659–663, https://doi.org/10.1109/34.295911.
[130] C. Fan, C. Zhang, A. Yahja, A. Mostafavi, Disaster City digital twin: a vision for
integrating artificial and human intelligence for disaster management, Int. J. Inf.
Manag. 56 (2021), 102049.