Journal of Building Engineering 33 (2021) 101816

Contents lists available at ScienceDirect

Journal of Building Engineering

Machine learning applications for building structural design and

performance assessment: State-of-the-art review
Han Sun a, *, Henry V. Burton b, Honglan Huang b
a Yahoo Research, USA
b Department of Civil and Environmental Engineering, University of California Los Angeles, USA

ARTICLE INFO ABSTRACT

Keywords: Machine learning models have been shown to be useful for predicting and assessing structural performance,
Machine learning identifying structural condition and informing preemptive and recovery decisions by extracting patterns from
Artificial intelligence
data collected via various sources and media. This paper presents a review of the historical development and
Building structural design and performance
recent advances in the application of machine learning to the area of building structural design and perfor-
assessment
Supervised learning
mance assessment. To this end, an overview of machine learning theory and the most relevant algorithms is
Unsupervised learning provided with the goal of identifying problems suitable for machine learning and the appropriate models to
use. The machine learning applications in building structural design and performance assessment are then re-
viewed in four main categories: (1) predicting structural response and performance, (2) interpreting experi-
mental data and formulating models to predict component-level structural properties, (3) information retrieval
using images and written text and (4) recognizing patterns in structural health monitoring data. The chal-
lenges of bringing machine learning into structural engineering practice are identified, and future research op-
portunities are discussed.

1. Introduction
Machine learning (ML) refers to a set of methodologies that are ca-
pable of automatically detecting patterns in data, which can then be
used to develop forecasting models and support decision making un-
der uncertain conditions (Murphy, 2012) [1]. There are three main
types of learning: supervised, unsupervised and reinforcement. Super-
vised learning is used to develop predictive models where the goal is
to map a set of inputs (also known as features, attributes or covari-
ates) to one or more outputs (also known as the response variable).
Supervised learning problems are described as classification or pat-
tern-recognition when the response variables are categorical and re-
gression when the outputs are numerical variables. Unsupervised or
descriptive learning is associated with much less well-defined prob-
lems, where the goal is to discover underlying relationships in the
data. Both supervised and unsupervised learning can be achieved us-
ing parametric and non-parametric models. Whereas the former uti-
lizes a fixed number of parameters, the size of the training dataset de-
termines the number of parameters in the latter. Parametric models
are often easier to construct and implement but are constrained by the
assumptions that they make about the data distribution. Non-
parametric models are much more flexible but their complexity in-
creases with the size of the dataset. Reinforcement learning, the least
popular of the three categories, is used to acquire knowledge on how
to act or behave (i.e. make decisions) under uncertainty (Hastie et al.,
2009 [2]; Murphy, 2012 [1]). Note that semi-supervised learning,
which, for the purposes of this paper, is not included as a primary cat-
egory, combines elements of both supervised and unsupervised learn-
ing.
ML methods are not foreign to building structural design and per-
formance assessment (SDPA) as applications in this area can be traced
back to as early as the late 1980's when Adeli and Yeh (1989) [3] de-
veloped and applied an ML-based methodology to a beam design
problem. This pioneering work was followed by several studies during
the 1990's that applied artificial neural networks (ANNs) (Hopfield,
1982) [4] to building SDPA problems. One of the first in this series of
studies was conducted by Vanluchene and Sun (1990) [5], who ap-
plied back-propagation neural networks (Rumelhart et al., 1986 [6])
to three distinct building SDPA problems related to locating the load
on a beam, designing a reinforced concrete beam and analyzing a sim-

* Corresponding author.
E-mail addresses: hansun2014@ucla.edu (H. Sun), hvburton@seas.ucla.edu (H.V. Burton), honglanhuang@ucla.edu (H. Huang).

https://doi.org/10.1016/j.jobe.2020.101816
Received 21 March 2020; Received in revised form 17 July 2020; Accepted 11 September 2020
Available online 21 September 2020
2352-7102/© 2020 Elsevier Ltd. All rights reserved.
H. Sun et al.
ply supported plate. This study was closely followed by several others
(Hajela and Berke, 1991; Ghaboussi et al., 1991; Wu et al., 1992;
Masri et al., 1993; Kang and Yoo, 1994; Messner et al., 1994; Elkordy
et al., 1994) [7–13], most of which utilized the back-propagation net-
work. Recognizing the growing popularity of ANNs in building SDPA,
Gunaratnam and Gero (1994) [14] conducted a detailed examination
of the factors that influence their performance, some of which were
domain-specific, while others were domain-independent. The authors
highlighted the importance of reduced dimensionality (i.e. the num-
ber of features or predictors) and embedment of domain-specific
knowledge in achieving effective learning. To address specific chal-
lenges associated with the back-propagation methodology such as the
slow rate of learning, Adeli and Park (1995) [15] explored the use of
counter-propagation algorithms to address building SDPA problems.
Whereas back-propagation networks utilized only supervised learning,
the counter-propagation algorithm combined both supervised and un-
supervised learning. In the Adeli and Park study, the two algorithms
were applied to four building SDPA problems including the concrete
beam design and simply supported plate analysis defined by Vanluch-
ene and Sun and two others involving the analysis of a steel beam.
In the late 1990's, Reich (1997) [16] conducted a review of the lit-
erature on the application of ML to civil engineering problems. In ad-
dition to building SDPA, the review included other civil engineering
domains such as transportation, construction management, water re-
sources, environmental and materials. In fact, only sixteen of the
ninety-seven citations were specific to building SDPA. In addition to
reviewing the literature, the author highlighted several issues to be
addressed towards the practical application of ML in civil engineering.
They include (1) having a deep understanding of the learning prob-
lems, (2) knowing which ML technique is most suitable for the prob-
lem at hand, (3) the ease of implementation or availability of various
ML techniques, (4) proper evaluation of trained models and (5) the
availability of efficient information management systems.
Due to the limited availability of computational power and stor-
age, early ML applications in building SDPA (such as the ones de-
scribed in the previous two paragraphs) were limited to a few rela-
tively simple problems involving small datasets. In contrast, the in-
crease in computational resources and resurgence of artificial intelli-
gence over the past two decades has led to the development of more
sophisticated tools and techniques that can harness these new data
streams and solve highly nonlinear learning problems. Within build-
ing SDPA, the revitalization of ML has been fueled by the complexity
of modern systems, which requires the generation and/or manipula-
tion of large datasets to rigorously assess their performance under var-
ious loading conditions. These datasets can be produced from (1) re-
connaissance and remote sensing from past extreme events, (2) mea-
surement data from large-scale (or multiple small-scale) physical ex-
periments, (3) response of instrumented systems under normal operat-
ing loading conditions, (4) large-scale computational simulations and
(5) relevant audio-visual media (e.g. images, videos and written text).
The abundance of studies on ML methods applied to building
SDPA problems since the Reich paper warrants a more current state-
of-the-art review. The goal of this paper is to synthesize past research
on this topic towards a common understanding of the types of prob-
lems that are suited to ML applications, the characteristics of ML
methods, the challenges associated with applying ML to building
SDPA (ML-SDPA) and opportunities for the future. The review begins
with a brief introduction to ML that includes a general problem for-
mulation and discussion of relevant sub-topics (feature engineering
and model training and performance evaluation). Next, the mathemat-
ical details of some ML algorithms that are increasingly being applied
to building SDPA problems are presented. This is followed by a review
of the existing ML-SDPA literature categorized in terms of the follow-
ing four application areas: (1) predicting structural response and per-
formance, (2) interpreting experimental data and formulating models
2
Journal of Building Engineering 33 (2021) 101816
to predict component-level structural properties, (3) information re-
trieval using images and written text and (4) recognizing patterns in
structural health monitoring data. Subsequently, a discussion of spe-
cific challenges and future research opportunities related to the avail-
ability and collection of useful data, the explainability and inter-
pretability (or lack thereof) of some ML models and challenges with
overfitted models are presented.
2. Overview of machine learning problems
This section presents an overview of ML beginning with a general-
ized formulation of supervised (classification and regression) and un-
supervised learning problems. A brief discussion of feature engineer-
ing and model training and performance evaluation is also included.
The material presented in this section is obtained from several statis-
tics and ML sources that provide more details.
2.1. General formulation
For supervised learning, the dataset of feature variables can be de-
scribed by a matrix X with dimension n×p, where n is the total num-
ber of observations (data points) and p is the number of features (or
independent variables). The response variable y is described by an
n×1 vector containing the label for each observation. For a classifica-
tion problem,y is a categorical variable and for regression, y is a nu-
merical variable. Unsupervised learning problems include the feature
matrix X but not the response variable. The objective of supervised
learning is to solve the generalized optimization problem by minimiz-
ing the empirical loss function defined by Equation (1) (Murphy,
2012) [1].
(1)
where yi is the response variable for observation i, f(xi;θ)(also com-
monly denoted as including later in this paper) is the predicted re-
sponse from the ML model based on the feature xi and θ represents
the set of model parameters. φ is a loss measure between the true (yi)
and predicted value of the response variable. λΩ(θ)is a regulariza-
tion term that penalizes the model based on its complexity by restrict-
ing the parameter set θ through some regularizing function Ω(θ). λ is a
model parameter that is determined as part of the optimization
process. The objective is to find the set of model parameters that
minimizes the empirical loss over the training data with the regular-
ization penalty considered. Note that θ is only used in parametric
models, which have a finite number of parameters. For example, lin-
ear regression models always have p + 1 model parameters. On the
other hand, rather than using a finite number of parameters to define
the data distribution, non-parametric models utilize a flexible parame-
ter set whose size, in theory, can be infinite, and is often treated as a
function. The Support Vector Machine with radial basis function ker-
nel is an example of a non-parametric model whose parameter set de-
pends on the training data. Equation (1) is a convenient generalized
formulation that is adopted by many supervised learning methods in-
cluding ordinary least squares, ridge, least absolute shrinkage and se-
lection operator (LASSO) and logistic and kernel regression. Depend-
ing on the ML method, the minimization problem can be solved using
a closed form solution, gradient-based optimization, or convex relax-
ation.
The goal of unsupervised learning is to infer the underlying struc-
ture and parameters of the model that generated the data, which can
then be used to group the data into clusters, generating new instances
and drawing inferences. The objective function for unsupervised
learning is shown in Equation (2), where α is the set of model para-
meters that characterizes a learned structure for the given dataset.
H. Sun et al. Journal of Building Engineering 33 (2021) 101816

The objective function can take the variant forms of negative log-
likelihood and Kullback-Leibler divergence. In clustering analysis, φ
quantifies the cost of assigning a data point xi to a particular cluster.
Examples of ML methods that follow this generalized formulation in-
clude the Gaussian Mixture Model, K-means and K-Nearest Neighbors
(Buhmann and Held, 1999) [17].
deep learning approaches are extremely popular because they auto-
mate feature engineering to achieve state-of-the-art level performance
in many pattern recognition and data mining domains. This approach
has gained widespread popularity in recent years because of the in-
crease in computation power, which made complex neural net archi-
tecture trainable, thus achieving superb feature extraction.

2.3. Model training and performance assessment

(2)

There are many well-established procedures for training ML mod-

In ML theory, the objective function expressed in Equations (1) els that attempt to achieve stable and effective prediction perfor-
and (2) is defined as the empirical loss over the training dataset, de- mance for new data given a training dataset. One common strategy is
noted as Ln(f) for a given model f. The theoretical solution to the ML k-fold cross validation (also discussed in Reich 1997 [16]), which ran-
problem, which is shown in Equation (3), is the set of parameters that domly splits a dataset into k different subsets and trains the model k
minimizes the loss function over the entire data space,L(f). However, times using the kth subset as testing data and the remaining k − 1 sub-
real problems are almost always limited by the amount of data sam- sets for model training. The best performing of the k models over the
pled from the entire space. Therefore, the ideal solution is often ap- testing dataset is selected. This procedure is intended to reduce over-
proximated by minimizing the empirical loss over the training data in- fitting on the training dataset. Another popular technique that is used
stead. to avoid overfitting is Bootstrapping, which randomly samples a sub-
set of the data with replacement and trains the model M times. The fi-
(3) nal model is selected as an average over the predicted results (regres-
sion) or based on a majority vote (classification) (Bunke and Droge,
1984 [23]) from the M models. Both bootstrapping and k-fold cross
where p(x,y) is the theoretical joint probability density function of the
validation effectively reduces model variances and removes bias and
feature (x) and response (y) variable over the entire data space, D.
are the primary training techniques used to develop data-driven mod-
els. These training procedures are evaluated by using various perfor-
2.2. Feature engineering
mance metrics for model selection. For example, performance metrics
for binary classification models include accuracy, precision and recall.
Prior to training a ML model, the features that are found to influ-
Precision refers to the number of correct “positive” (e.g. building is
ence model performance, improve training efficiency, and increase
red-tagged) predictions normalized by the number of positive predic-
flexibility, must be selected and extracted. Most ML methods deploy
tions. Recall is the number of correct positive predictions normalized
standard feature selection and extraction algorithms. However, some
by the number of actual positive classes. Accuracy is the number of
also have the ability to adjust features to achieve the best possible
correct predictions normalized by the total number of predictions
prediction performance.
(Powers, 2011) [24]. For multi-class classification problems, in addi-
Feature selection can be categorized into three methods: filter,
tion to the aforementioned three metrics, a confusion matrix and
wrapper, and embedded. The filter method ranks the original features
top-k class accuracy is also used (Krizhevsky et al., 2012) [25]. Re-
according to an importance measure such as the scores from a Chi-
gression models are typically evaluated using the mean squared error
square test or correlation coefficients between individual features and
(MSE), root mean squared error (RMSE), adjusted root mean square
the response variable and selects a subset to be used for model train-
error, coefficient of multiple determination, the median absolute error
ing. The wrapper method recursively includes or excludes features
and the median absolute relative deviation (MARD) (Mack et al.,
from an initial pool and selects the best performing feature set based
2007; Burton et al., 2017) [26,27].
on feedback from the ML model. Embedded methods are used by
those algorithms that incorporate automatic feature selection (e.g.,
3. Machine learning models commonly used for building SDPA
LASSO regression). Both filter and wrapper methods are good at
problems
avoiding overfitting issues by reducing model complexity and improv-
ing training efficiency by reducing highly correlated features.
Some ML algorithms that are commonly employed for building
Feature extraction consists of two major tasks that increase the ef-
SDPA problems are introduced. Only supervised learning (classifica-
fectiveness of ML models. The first is dimensionality reduction, which
tion and regression) models are included because very few ML-SDPA
is achieved by applying methods such as Principle Component Analy-
studies involving unsupervised learning can be found in the literature.
sis (PCA), which performs a linear mapping from the original data
The next section summarizes the recent (mostly within the last
space to a lower dimensional space such that the data variance over
decade) ML-SDPA research. Most of the methods included in the liter-
each resulting orthogonal component is maximized. The second in-
ature review are covered in this section. However, some of the more
volves transforming the data into a higher dimensional space such
advanced algorithms such as recurrent or convolutional neural net-
that the patterns become sparse and separable, such as in kernel-
works, while included in the ML-SDPA review, are not discussed in
based ML algorithms (Huang et al., 2006) [18].
this section. The relevant references are provided for readers who
Besides the earlier-described general feature engineering tech-
would like to become more familiar with the details of these method-
niques, specific feature designs have been proven to be very successful
ologies.
for domain-specific problems. For instance, the use of HAAR-like fea-
tures achieved human-level face-recognition accuracy with far less
3.1. Linear regression: ordinary least squares, LASSO, ridge and
computational effort (Viola and Jones, 2001; Lienhart and Maydt,
polynomial basis functions
2002) [19,20], SIFT features (Lowe, 2004) [21] are very effective for
object detection within images and HOG features (Dalal and Triggs,
Ordinary Least Squares (OLS) is one of the simpler regression
2005) [22] are particularly good for human detection. However, these
methods and is very commonly used in building SDPA. It is included
domain-specific feature engineering techniques require considerable
in this section because it provides a basis for understanding some of
trial and error testing and are designed to only work for very specific
the less common regression techniques (e.g. LASSO and ridge), which
problems and data structures. Neural networks and the associated

3
H. Sun et al. Journal of Building Engineering 33 (2021) 101816

are described later in this section. In the OLS formulation, the re- variables by employing nonlinear basis functions. One strategy that
sponse variables are approximated using Equation (4). has been adopted in several of the ML-SDPA studies discussed later in
the paper is the use of kernel basis functions. The word “kernel” has
(4) several interpretations; however, it is often described as a measure of
the similarity between two observations xi and xj. This similarity (or
where y is the observed response variables, represents the predicted
lack therefore) is quantified using a kernel function, k(xi,xj) (note that
response variables, X is the feature matrix described earlier and ∈ is
xi and xj are feature vectors of an individual observation). Examples
an n×1 vector of residuals, which is taken as the difference between
of commonly used kernel functions include linear, polynomial, sig-
the observed and predicted values of the response variables. β (same
moid and Gaussian or squared exponential. Equation (6) describes the
as the model parameters θ, in Equation (1)) is a p×1 vector
Gaussian kernel, which was commonly used in the reviewed ML-SDPA
( if the intercept is included) of predictor coefficients,
studies.
which is derived by minimizing the residual sum of squares, RSS = (y
− Xβ)T(y − Xβ). Note that for OLS regression, RSSrepresents the loss
function (φ) described earlier and no regularization term is included.
The OLS minimizing predictors are computed using the closed form (6)
solution in Equation (5a).
LASSO (Tibshirani, 1996) [28] and ridge (Hoerl and Kennard,
1970) [29] are two linear regression methods that, as noted earlier, where the bandwidth, σ, is a model parameter that is determined dur-
incorporate feature engineering as part of the overall formulation. The ing the training process (e.g. using k-fold cross validation or based on
LASSO method integrates both feature selection (by setting some of AIC). In the context of kernel ridge regression (i.e. ridge regression
the predictor coefficients to zero) and shrinking the OLS coefficients performed using a kernel basis function), the feature matrix X is re-
by including a penalty on the RSS loss function. As shown in Equation placed by a kernel matrix K (Equation (7)) and the minimization
(5b), the regularizing function (Ω(θ) in Equation (1)) is taken as the problem takes on the form shown in Equation (8).
sum of the absolute values (the L1norm) of the predictor coefficients.
Like LASSO, ridge adds a penalty to the RSS loss function. However,
the root sum of the square of the predictor coefficients (the L2 norm)
is used as the regularization function. Also, ridge regression does not (7)
incorporate feature selection i.e. none of the predictor coefficients are
shrunk to zero.

(5a) (8)

where In is an n×n identity matrix and α is the vector of regression

(5b)
coefficients in kernel space. Equation (9) is used to compute the re-
sponse function for a test data point (xtest).

(5c)
(9)

where I is a p×p identity matrix. Note that while the predictor coeffi-
3.3. Tree-based algorithms: Decision Trees, random forests and adaptively
cients for OLS and ridge can be solved analytically, boosted trees
there is no closed form solution for the LASSO coefficients .
Tree-based algorithms can be used for both classification and re-
For both ridge and LASSO, the value of the regularization parameter
gression. The models that belong to this category recursively divide
can be determined using k-fold cross validation and/or by minimizing
the training dataset while exploring and learning its structure towards
some information criterion such as the Akaike Information Criterion
creating subspaces that are mutually exclusive or have high levels of
(AIC) (Akaike, 1974) [30] and Bayesian Information Criterion (BIC)
purity (the ratio between the class with the most samples and the size
(Schwarz, 1978) [31].
of the data subset). The rules used to grow and prune each tree (e.g.
By replacing X in Equation (4) with a nonlinear transformation of
node splitting and stopping criteria), the number of considered trees
itself (also referred to as a basis function, e.g., g(X)), linear regression
and the approach to aggregating information from multiple trees, are
can be used to create models that capture a nonlinear relationship be-
what distinguishes the different types of tree-based algorithms.
tween the response and feature variables. It is important to note that
The structure of a tree can be described using nodes, each of
the regression model itself is still linear because the parameters (β)
which represents a data subset of predictors and response variables.
are linear (Murphy, 2012) [1]. In the literature review section pre-
The lowest level node, which is the one that comprises the entire
sented later in the paper, several studies have utilized linear regres-
dataset, is called the root node. Additional nodes are created when a
sion with higher-order polynomial basis functions i.e. X in Equation
parent node is divided based on some criteria (described later) to cre-
(4) is replaced with . The complexity of the model
ate child nodes. The highest level nodes, which are also referred to as
can be increased by using higher values of d or by utilizing multiple
the leaf nodes, represent the data subsets created at the very end of
piece-wise polynomial basis functions as in multiple adaptive regres-
the data-division process. In other words, no further splitting of the
sion splines (MARS) (Friedman, 1991) [32].
data occurs beyond the leaf nodes, whose associated subspaces pro-
vide the response variable prediction. All other nodes (excluding the
3.2. Kernel regression root and leaf nodes) are referred to as interior nodes. For a classifica-
tion problem, the prediction is taken as the dominant class (or catego-
As noted earlier, linear regression models can be adapted to cap- rial variable) within the leaf node that meets the data-division criteria
ture complex non-linear relationships between features and response for all nodes leading to it. For a regression problem, the predicted

4
H. Sun et al.
value of the response variable is taken as the mean value at the corre-
sponding leaf node.
The Decision Tree (DT), which is the simplest of the tree-based al-
gorithms, considers all features when splitting each data subset and
choses the one that minimizes the impurity measure. The Gini index
(GI) (Equation (10)) is commonly used to measure impurity.
(10)
where represents the fraction of observations
belonging to the cth class within the region (or data subset) Rm. I(⋅) is
an indicator function. Several alternative criteria such as the mini-
mum number of samples needed at a given node for additional split-
ting and the maximum depth of the tree are used to terminate the
growth (or data-division) process (Breiman et al., 1984; Hastie et al.,
2009) [33,34].
Adaptive boosting seeks to improve the performance of DTs by it-
eratively creating new models that correct the errors of the previous
one. This is achieved by applying weights to the training datapoints
based on some set of criteria. The first model is created using uniform
weights for the training data
{(xi,yi):i = 1,2,…,n}, yi∈{1,2,…,C}. The error (ε(m))
associated with it-
eration m is computed and the weights are updated according to
Equations (11a) and (11b).
(11a)
wi + 1 = wi⋅exp(α(m)⋅I(yi≠T(m)(xi))) (11b)
where α(m) is the weight factor assigned to the base model T(m)(x), and
I(⋅) is the indicator function. Based on Equation (11a), the misclassi-
fied data points in iteration m are assigned a higher α(m) and a higher
weight in iteration m + 1. A linear aggregation of the weighted base
models is used to give the final prediction (Freund and Schapire,
1997; Hastie et al., 2009 Abu-Mostafa et al., 2012) [35–37].
(12)
The Random Forests (RF) model uses an aggregated set of Decision
Trees, which are constructed by applying bootstrap sampling to the
training dataset. For regression problems, the model prediction is
taken as the average of the considered DTs and the class that is pre-
dicted by the majority of trees is used for classification. During the
data-division process at each node, a randomly selected subset of fea-
tures is considered, thus reducing the correlation across the different
trees and avoiding overfitting (Breiman, 2001) [38].
3.4. Logistic regression
Logistic regression is often used as a classification algorithm be-
cause it is fairly easy to implement and interpret the final results.
Given the feature vector, x and assuming a binary response for each
observation, y∈{0,1}, the probability of the y = 1 class is computed
using the sigmoid transformation of the linear function of x (Equation
(13)).
(13)
where β = [β1,β2,…,βp]T and β0 are the predictor coefficients vector
and the bias term, respectively. Multinomial logistic regression is used
for problems with more than two classes i.e. y∈{1,2,…,C}, where C
5
Journal of Building Engineering 33 (2021) 101816
represents the number of classes. The probability that y belongs to the
cth class is computed using Equation (14).
(14)
where βc and βc0 are the predictor coefficients vector and the bias
term associated with computing the probability of the cth class. The
predicted class is taken as the one with the highest probability
(Bishop, 2006) [39]. The training process used to retrieve β is similar
that of the OLS method discussed above with a closed-form solution.
3.5. Support vector machines
Originally developed as a binary classifier, support vector ma-
chines (SVM) seek to determine the hyperplane that separates a
dataset into two classes with the widest possible gap between them. If
the training data is linearly separable, a hard-margin version of SVM
is applied. Otherwise, the hinge loss function (Rosasco et al., 2004)
[40] is introduced to maintain a soft margin for the decision bound-
ary, which begins by defining the L2 regularized objective function
shown in Equation (15).
(15)
where the response approximation function in the original feature
space is . By adopting the ε-insensitive loss function to-
gether with slack variables (because the objective function is not dif-
ferentiable) (Cortes & Vapnik, 1995) [41] and an appropriate kernel
function k(x,xi), the response approximation becomes
where are the regression co-
efficients in kernel (high-dimensional) space.
For a binary class problem with the training dataset defined by
{(xi,yi):i = 1,2,…,n}, yi∈{ − 1,1} and feature vector observation xi,
the classification is based on For a multi-class problem, an
one-versus-all or one-versus-one approach can be adopted. For a set of
C classes i.e. yi∈{1,2,…,C}, the data from class c is treated as positive
and the data from the other classes is treated as negative in the one-
versus-all approach. In the one-versus-one approach, classifiers
are trained and a prediction is established for each pair. The class
with the highest number of votes is then used as the prediction
(Bishop, 2006; Murphy, 2012) [1,39].
3.6. K-Nearest Neighbors
K-Nearest Neighbors (KNN) is a non-parametric algorithm that is
used for both classification and regression. First, the "K" observations
in the training data that are nearest to the observation x (Nk) are
identified based on some pre-defined distance metric. An empirical
function is then created based on the number of each class in that
subset of datapoints (defined as Nk). For y∈{1,2,…,C}, the probability
that observation y belongs to the cth class is computed using Equation
(15).
(16)
where I(e) is an indicator function that is equal to 1 if e is true and 0
if e is false where e represents whether the observation yi belongs to
class c. The Euclidean distance is often used as the distance metric in
KNN and the value of K can be chosen using k-fold cross validation.
The observation x is assigned to the class with the highest empirical
probability. KNN can also be used for regression where the value of
H. Sun et al.
the response variable is taken as the average (or median) value of the
K nearest distance (Murphy, 2012) [1].
3.7. Discriminant analysis
Discriminant analysis is a technique that is used to address binary
or multi-class classification problems. The methodology assumes that
feature variables within a particular class take on a multi-variate nor-
mal (MVN) distribution. More specifically, the distribution of the fea-
ture vector x conditioned on class c is defined by , 4. Prior studies on applying machine learning to building SDPA
where μc and Σc are the mean vector and covariance matrix computed
using that data subset associated with class c. The probability that ob-
servation x belongs to class c is obtained by applying Bayes theorem
to the class-conditioned multi-variate normal distribution (Equation
(17)).
(17)
where fc(x) and fi(x) are the class-conditioned MVN probability den-
sity function (pdf) for the feature vector x and πc is the prior probabil-
ity of being in class c (estimated as the fraction of class c observations
in the training dataset). The classification (or discriminant) function is
obtained by substituting the MVN pdf into Equation (17). In linear
discriminant analysis (LDA), the same covariance matrix (Σ) is used
across all classes (i.e. computed using the feature vectors for the en-
tire training dataset), which produces a linear decision boundary be-
tween each pair of classes. The LDA classifier (δc(x)) is shown in
Equation (18).
(18)
The predicted class is taken as the one with the highest δc(x)
value. Quadratic Discriminant Analysis (QDA) uses class-conditioned
covariance matrices (Σc) (i.e. computed using only the feature vectors
from the class c data subset), which produces a quadratic decision
boundary between each pair of classes. The QDA classifier is shown in
Equation (19) (Hastie et al., 2009; Murphy, 2012) [1,36].
(19)
3.8. Artificial neural networks and its variants
ANNs refer to a category of pattern recognition algorithms that are
inspired by the biological nervous system. The network takes a set of
features x as inputs and applies complex feature fusion operations
through a series of layers. hl, which denotes the lth hidden layer and
consists of pl neurons, is computed based on a linear combination of
the neurons at the previous layer via the pl×pl − 1 weight matrix Wl
and the pl×1 bias vector bl, followed by the activation al(⋅) (Wu et al.,
2018) [42].
h l = a l( W lh l − 1 + b l) (20)
The final layer, which could be a linear layer for regression prob-
lems or a softmax layer for classification problems (Bishop, 2006)
[39], outputs the predicted response . Loss function choices include
RSS for regression and cross entropy loss for classification. The ANN
model is trained through backpropagation, which is a gradient-based
algorithm that calculates error gradients over each model parameter
based on the chain rule (Rumelhart et al., 1986) [6]. Numerous vari-
ants of ANNs have been developed to achieve faster convergence, bet-
ter prediction performance and less memory usage. ANN variants can
differ based on the activation function (e.g., leaky rectified linear
unit), the type of layer-connections (e.g., dropout, max-pooling) or the
6
Journal of Building Engineering 33 (2021) 101816
connection mechanism (e.g., Recurrent Neural Network (RNN)). The
term deep learning (DL) is used to describe ANNs with many layers.
The success of DL started with Krizhevsky et al. (2012) [25], who for-
mulated a deep convolutional neural network (CNN) ImageNet classi-
fication model that achieved superb performance. Because of its pat-
tern recognition capability, DL has since been successfully used in
many domains including computer vision, speech recognition and nat-
ural language processing.
problems
A broad range of relatively recent (mostly within the last decade)
ML-SDPA publications are summarized based on the four categories
identified earlier, which are also schematically illustrated in Fig. 1:
(1) predicting structural response and performance, (2) models devel-
oped using data from physical experiments, (3) information retrieval
using images and written text and (4) models developed using field
reconnaissance and structural health monitoring data. It is recognized
that some of the reviewed studies belong to more than one category.
For instance, models developed with the intent of using measured
structural responses (from instrumented buildings) and/or observed
damage (during reconnaissance) to estimate building performance
states belong to categories (1) and (4) (e.g. Ghiasi et al., 2016 [43];
Zhang and Burton, 2019 [44]). Another example is text-feature-based
building damage prediction models trained using field reconnaissance
data, which belong to categories (3) and (4) (Mangalathu and Burton,
2019) [45]. The reason for the categories is to help researchers and
practitioners make decisions about which types of ML problems are
suited for specific SE problems.
4.1. Predicting structural response and performance
Nonlinear structural response simulation is recognized as the ideal
approach to assessing the performance of built systems under extreme
loading. Prior studies have used ML to complement or expand the pre-
dictive capabilities of “mechanics-based” or “physics-based” structural
response simulations. These so-called surrogate (or meta-) models
serve as compact statistical representations of the relationship be-
tween a set of input variables (e.g. structural properties, loading char-
acteristics) and the response or performance quantities of interest.
They are useful for reducing the number of mechanics-based simula-
tions needed for computationally intensive applications such as uncer-
tainty quantification and performance optimization within a high-
dimension parameter space.
Table 1 summarizes the studies that have used ML models to esti-
mate structural response demands or performance metrics (e.g. col-
lapse fragility). For each study, the structural system type, category of
response and predictor variables, adopted ML algorithm(s) and model
performance evaluation methods are shown. The listed studies fo-
cused on building seismic systems including steel (Seo et al., 2012;
Khojastehfar et al., 2014; Jough and Sensoy, 2016; Kiani et al., 2019)
[46–49] and concrete moment frames (with and without masonry in-
fill) (Mitropoulou and Papadrakakis, 2011; Burton et al., 2017; Mor-
fidis et al., 2017; Zhang et al., 2018) [27,50–52], steel braced frames
(Moradi and Burton, 2018; Moradi et al., 2018) [53,54] and rein-
forced concrete shear walls (Sun et al., 2019; Zhang and Burton,
2019) [44,55]. Most of the reviewed studies developed regression
models with engineering demand parameters (e.g. story drift ratio,
peak floor acceleration) as the response variables and in some of those
cases, this prediction was as an intermediate step towards developing
limit state fragility functions (e.g. Seo et al., 2012) [46]. Utilizing a
slightly different approach, a few studies directly incorporated the
limit state fragility parameters (e.g. median and dispersion of the col-
lapse intensity) as the response variable (Khojastehfar et al., 2014;
H. Sun et al.
Fig. 1. Schematic illustration of the four ML-SDPA categories.
Jough and Sensoy, 2016; Burton et al., 2017) [27,47,48]. Whereas
most of the studies sought to predict mainshock demands and limit
state parameters, three (Burton et al., 2017; Zhang et al., 2018; Zhang
and Burton, 2019) [27,44,52] focused on aftershock performance. Bi-
nary classification models were used in only two of the fourteen stud-
ies (Zhang et al., 2018; Kiani et al., 2019) [49,52]. While some studies
used only ground motion intensity measures as the model features
(e.g. Mitropoulou and Papadrakakis, 2011; Kiani et al., 2019) [49,50],
others also included structural configuration (e.g. number of stories in
building), modeling (e.g. damping ratio) and material properties (e.g.
yield strength of steel).
Several different algorithms were used to develop the ML-based
structural response and limit state parameter prediction models. For
regression, ANN and linear models with polynomial basis functions
were most widely used. Other adopted regression algorithms include
ridge (conventional and kernel), LASSO, elastic net, PCA and SVM.
Some authors used a single algorithm (e.g. Mitropoulou and Pa-
padrakakis, 2011) [50] while others compared the performance of
multiple algorithms (e.g. Burton et al., 2017) [27]. Similarly, of the
two studies that developed classification models, one used RF (Zhang
et al., 2018) [52] while the other compared the performance several
algorithms (Kiani et al., 2019) [49]. Most studies used training-testing
splits to evaluate the performance the developed ML model. These
splits ranged from 33% to 67% (i.e. 33% training and 67% testing) on
one extreme to 80%–20% on the other. None of the studies evaluated
the effect of the training-testing partition point on model perfor-
mance. The ratio of the predicted to actual value of the response vari-
able was the most widely used performance metric in the regression
studies. Others included the coefficient of determination (R2), MSE,
MARD and mean absolute error (MAE). For the classification studies,
accuracy, precision, recall and F-measure were used as the perfor-
mance metrics.
4.2. Models developed using experimental data
There is a long history of using empirical data from physical ex-
periments to develop statistical models for predicting structural para-
7
Journal of Building Engineering 33 (2021) 101816
meters (e.g. component stiffness, strength and/or deformation capac-
ity). Many of the earlier models, which were developed using very
small datasets (on the order 10's of datapoints), adopted relatively
simple analytical expressions with one or two input parameters
(Hobbs, 1972; Bažant & Zebich, 1983; Bažant & Chern, 1984; Bažant
et al., 1991; Bažant & Kim, 1991; Carpinteri et al., 1995) [56–61]. As
the size of the datasets increased (on the order of 100's), more com-
plex multi-variate analytical expressions have been adopted (e.g.
Haselton et al., 2016; Lignos & Krawinkler, 2010) [62,63]. Some of
the most recent studies, which are summarized in the next paragraph,
have implemented advanced ML models.
Table 2 summarizes the studies that have developed ML models
using experimental data. The categories are the same as Table 1 with
one exception: the models developed to predict structural response
demands and limit state parameters are based on the results from
nonlinear analyses, which, for the most part, meant that the authors
had the ability to control the size of the dataset. However, for the ML
models developed using experimental data, the size of the dataset is
controlled by the number of experiments conducted in prior studies.
For this reason, the number of sample points is also documented in
Table 2. Only reinforced concrete components were used in the re-
viewed studies including columns (Naeej et al., 2013; Luo and Paal,
2018; Luo and Paal, 2019; Mangalathu and Jeon, 2018) [64–67],
beam-column joints (Jeon et al., 2014; Luo and Paal, 2018; Luo and
Paal, 2019; Mangalathu and Jeon, 2018) [65–68], slabs (Vu and
Hoang, 2016) [69], infilled frames (Huang and Burton, 2019) [70]
and shear walls (Mangalathu et al., 2020) [71]. The number of speci-
mens ranged from 65 (Naeej et al., 2013) [64] to 536 (Mangalathu
and Jeon, 2018) [67]. Both classification and regression models have
been developed using experimental data. The former has been used to
predict the failure mode in specific components (Huang and Burton,
2019; Mangalathu and Jeon, 2019; Mangalathu et al., 2020)
[67,70,71] while the latter was employed to predict component-level
structural parameters including column confinement coefficients,
beam-column shear strengths, punching shear strength in slab, beam-
column drift capacity and backbone parameters (e.g. strength, stiff-
ness and deformation capacity). The Luo and Paal (2018) [65] model
is the only one that was developed to predict multiple output parame-
H. Sun et al. Journal of Building Engineering 33 (2021) 101816

Table 1
Summary of ML models developed in prior studies to predict building structural response and/or performance.

Study Structure Type Response Variable Predictor Variables ML Algorithm(s) ML Model Performance
(s) Evaluation

Training/Testing Performance
Split Metric

Mitropoulou RC Frames Structural Ground motion parameters ANN 33–67 Ratio of

and response predicted to
Papadrakakis, parameters actual
2011 response
Seo et al., 2012 Steel Moment Frames Structural Earthquake direction, steel properties, Linear Regression with NA NA
response damping and building age and configuration Polynomial Basis Function
parameters
Khojastehfar et Steel Moment Frames Collapse fragility Ground motion parameters ANN 65–30 Mean
al., 2014 parameters absolute
error and
MSE
Jough and Steel Moment Frames Collapse fragility Frame strength and ductility parameters Linear Regression with NA R2, MSE and
Sensoy, 2016 parameters Polynomial Basis Function mean
absolute
error
Burton et al., RC Infilled Frames Aftershock Mainshock response demands (e.g. SDR) and OLS, PC, LASSO, Ridge 80–20 MARD
2017 collapse fragility component damage levels (conventional and kernel)
parameters regression
Morfidis et al., RC Frames SDR and Ground motion (e.g. peak ground acceleration) ANN 70–30 MSE
2017 associated damage and structural parameters (e.g. fundamental
period)
Moradi and Controlled Rocking Structural Rocking frame design parameters (e.g. post- Linear Regression with 35–65 R2, ratio of
Burton, 2018 Steel Braced Frames response tensioning force and frame aspect ratio) Polynomial Basis Function predicted to
parameters actual
response
Moradi et al., Controlled Rocking Performance limit Rocking frame design parameters (e.g. post- Linear Regression with 80–20 R2, ratio of
2018 Steel Braced Frames states (e.g. life tensioning force and frame aspect ratio) Polynomial Basis Function predicted to
safety) actual
response
Zhang et al., RC Frames Post- earthquake Mainshock response demands (e.g. SDR) and Random Forests 75–25 Confusion
2018 safety state component damage levels Matrix
Kiani et al., Steel Moment Frames Damage fragility Ground motion parameters Logistic, LASSO, SVM, Naïve 70–30 Recall,
2019 parameters Bayes, DT, RF, KNN, DA and precision
ANN and F-
measure
Zhang and Tall RC Building with Aftershock Mainshock response demands Support Vector Machines 75–25 MSE
Burton, 2019 Moment Frame and damage fragility
Core Walls parameters

ters. The input parameters always comprise the cross section, rein-
forcement and material properties of the component.
Linear (ordinary and piecewise linear) models, DT, MARS, sym-
bolic methods, least square SVM and ANNs have been used for regres-
sion. The methods adopted for the classification studies include adap-
tive boosting, logistic regression, ANNs, RF, SVM and DT. Again, most
studies utilized training-testing splits to evaluate model performance,
however, because of the generally small size of the datasets, the parti-
tion point was mostly skewed towards a much larger proportion for
the training subset (e.g. 90%–10% training-testing). MSE, RMSE and
R2 were most commonly used to evaluate the performance of regres-
sion models. Bias, scatter index, correlation coefficient, agreement in-
dex, coefficient of variance, mean average percentage and absolute er-
ror were also used for this purpose. Similar to the structural response
and limit state assessment studies, accuracy, recall and precision were
used to evaluate the classification models.
4.3. Information retrieval using images, video and written text
Techniques for automatically extracting information from images,
video and written text, have been broadly applied to several fields, in-
cluding engineering, medicine and the physical, natural and social sci-
ences. In building SDPA, large numbers of images, written text and (to
a lesser extent) videos are often generated during laboratory experi-
ments, field reconnaissance and routine inspections. With systematic
collection, curation and organization, ML models can be developed to
extract useful information from these three types of media.
Computer Vision (CV) is a sub-category of artificial intelligence
that seeks to empower computers to extract meaningful information
from images and videos (Szeliski, 2010) [72]. It is worth noting that
while ML methods can be incorporated, some CV tasks can be per-
formed using non-ML algorithms. In fact, most of the existing CV ap-
plications related to the built environment did not utilize ML algo-
rithms. These studies focused on (i) visual identification and retrieval
of concrete (Zhu et al., 2011; German et al., 2012; German et al.,
2013; Koch et al., 2014; Koch et al., 2015) [73–77] and steel (Kong
and Li, 2018) [78] crack properties (e.g. crack width, length, and ori-
entation) and spalling (concrete only) from images and videos, (ii) au-
tomatically developing as-built models using images (Brilakis et al.,
2011; Koch et al., 2014; Koch et al., 2015) [76,77,79] and (iii) struc-
tural component-level damage classification (German et al., 2013;
Paal et al., 2015) [75,80].
Unlike the previously mentioned studies that explicitly make use
of designated features for visual content detection/classification, the
more modern CV applications such as the ones summarized in Table
3, utilize ML methods to automatically extract visual features. ML-
based computer vision has been used to detect RC cracks and spalling
(Cha et al., 2018; Kucuksubasi and Sorgucb, 2018; Hoang et al.,
2019b) [81–83], detect loosened and corroded steel bolts (Cha et al.,
2016; Cha et al., 2018) [81,84] and identify and classify structure and
component types and the presence and severity of damage (Gao and

8
H. Sun et al. Journal of Building Engineering 33 (2021) 101816

Table 2
Summary of ML models developed using data from physical experiments of building components.

Study Component Number Response Variable(s) Predictor Variables ML Algorithm(s) ML Model Performance
Type of Evaluation
Sample
Points
Training/Testing Performance
Split Metric

Naeej et al., RC Column 65 Lateral confinement Material and geomteric M5 Regression Tree 37–28 Bias, Scatter
2013 coefficient properties Index,
correlation
coefficient
and
agreement
index
Jeon et al., RC Beam- 516 Shear strength Material and geometric Linear, MARS and symbolic regression NA RMSE, CC and
2014 Column Joint properties Coefficient of
Variance
Luo and RC Column 262 Backbone curve Material and geometric Least square SVM regression 90–10 R2 and RMSE
Paal parameters (e.g., yield properties, applied axial loads
2018 and maximum shear and failure mode
force)
Luo and RC Column 160 Drift capacity Material and geometric Locally weighted least square SVM 70–30 R2, RMSE and
Paal properties regression mean absolute
2019 prediction
error (MAPE)
Hoang et RC 218 Reinforcement- concrete Steel and concrete properties Least square SVM regression NA R2, RMSE and
al., 2019 Components ultimate bond strength MAPE
Hoang RC Slab 140 Punching shear Material and geometric Piecewise linear and ANN regression NA R2
2019 capacity properties and fibre volume
Vu and FRP RC Slab 82 Punching shear Material and geomteric Least square SVM and ANN regression 72–10 R2, RMSE and
Hoang capacity properties MAPE
2016
Mangalathu RC Beam- 311 Shear strength and Material and geomteric Linear, logistic, LASSO, ridge, stepwise 70–30 R2, MSE and
and Jeon Column Joint failure mode properties and elastic net for regression. KNN, Naïve Absolute Error
2018 Bayes, SVM, DT and RF for classification for regression
and accuracy
for
classification
Huang and RC Frames 114 In- plane failure mode Infill material and geometric AdaBoost, DT, logistic, multi- layer 70–30 Recall score
Burton with infills properties, stiffness of the perceptron, RF and SVM
2019 system and axial loading
Mangalathu RC Core Wall 393 Failure mode Wall geometric properties, Naïve Bayes, KNN, DT, RF, AdaBoost, 70–30 Confusion
et al., material properties of rebar XGBoost, LightGBM and CatBoost matrix
2020 and concrete and design
parameters

Table 3
Summary of ML models developed using images, videos and written text.

Study Structure and/or Component Task Media ML ML Model Performance

Type Type Algorithm(s) Evaluation

Training/Testing Performance
Split Metric

Cha et al., 2016 Steel Bolts Detecting loosened bolts Images SVM ~90- 10 Accuracy
Cha et al., 2018 Steel and concrete Detecting concrete cracks, steel corrosion, bolt corrosion and Images Faster R– ~80- 20 Precision
components steel delamination CNN
Gao and Mosalam Multiple building systems and Structural component and system classification. Damage type Images CNN ~80- 20 Confusion
2018 components and level classification matrix
Kucuksubasia and RC buildings RC crack Images CNN ~2- 98 Accuracy
Sorgucb, 2018
Hoang et al., 2019b RC building components RC spalling Images Logistic ~90- 10 Recall and
regression F1 score
Mangalathu and Multiple building types ATC- 20 Tag (red, yellow, green) Text RNN 75–25 Confusion
Burton 2019 matrix
Gonzalez et al., 2020 Multiple building types Building materials and lateral support resisting system types Images CNN N/A Precision
and recall
Naito et al., 2020 Multiple building types Seismic damage level Images CNN N/A Precision
and recall

Mosalam 2018; Gonzalez et al., 2020; Naito et al., 2020) [85–87].

While CNN is the most widely used method among these studies, SVM
and logistic regression has also been implemented. The training-
testing splits ranged from as high as (in terms of the relative size of

9
H. Sun et al. Journal of Building Engineering 33 (2021) 101816

the training set) 90%–10% to as low as 2%–98%. It is worth noting
that the latter involved a study that utilized transfer learning which
typically does not require large amounts of training data. The confu-
sion matrix, accuracy, precision, recall and the F1 score were used as
performance metrics.
The Mangalathu and Burton (2019) [45] study, which is also sum-
marized in Table 3, is the only one that utilized text-based media. The
authors trained a long short-term memory (LSTM) deep learning
model (Graves and Schmidhuber, 2005; Hochreiter, and Schmidhuber,
1997) [88,89] to classify building damage based on the ATC-20 cate-
gories (red, yellow and green) (ATC, 1995) using natural language
damage descriptions as the features. The dataset included 3423 build-
ings affected by the 2014 South Napa earthquake, with written docu-
mentation of the damage and the assigned ATC-20 tags. A 75%–25%
training-testing split was used and the model performance was also
assessed using the confusion matrix.
utilized training-testing splits and two of them (Rafiei and Adeli,
2017; Sun et al., 2019) [55,95] evaluated the effect of the partition-
point on model performance. The Figueredo et al. study was the only
one to utilize the receiving operator characteristic (ROC) curve to
evaluate model performance. The other studies utilized some of the
metrics from prior sections (e.g. accuracy, MARD, mean absolute er-
ror).
The only study to develop ML models using post-event field recon-
naissance data (besides the ones mentioned in the earlier sections) is
by Mangalathu et al. (2020) [96]. Using a similar dataset from the
2014 South Napa earthquake (discussed in the previous section), the
authors developed a second damage classification model based on
ATC-20 tags. However, instead of written damage descriptions, this
model utilized features related to the building (e.g. age, number of
stories), site (closest distance to surface projection of the fault rup-
ture) and shaking intensity.

4.4. Models developed using structural health monitoring and field
reconnaissance data
Structural health monitoring (SHM) and post-event field reconnais-
sance have been central to the advancement of building SDPA. The
data generated from both of these activities provide insights into the
performance of different types of structures, especially under extreme
loading conditions, and are well-suited for ML applications.
SHM is generally concerned with using various types of sensors to
detect the type, location and extent of damage to a structure. Some of
the more traditional techniques that have been used to detect damage
from SHM data include auto-regressive model fitting (e.g. Sohn and
Farrar, 2001) [90], Fast Fourier Transform (e.g. Lynch, 2002 [91])
and wavelet transformation (e.g. Noh et al., 2011; Hwang and Lignos,
2018) [92,93]. Some noticeable ML-SDPA-SHM studies are summa-
rized in Table 4. A broad range of structure types were considered, in-
cluding an aluminum frame test specimen (Figueiredo et al., 2011)
[94], a steel frame and truss (Ghiasi et al., 2016) [43] and tall RC
building structures (Rafiei and Adeli, 2017; Sun et al., 2019) [55,95].
All but one of the studies involved damage detection, localization
and/or classification. The Sun et al. study was focused on reconstruct-
ing seismic structural responses in tall buildings. This was also the
only study that utilized regression (i.e. all others incorporated classifi-
cation). In most cases, the predictor variables (e.g. auto-regressive and
frequency-domain parameters, wavelet features) were extracted from
accelerometer recordings. However, while the Sun et al. model was
developed with the intention of being applied to accelerometer mea-
surements, it was demonstrated using data generated from nonlinear
response history analyses. The adopted ML methods include SVM,
ANN and kernel ridge. Also, the Rafiei and Adeli (2017) [95] study
implemented a neural dynamic algorithm that was previously devel-
oped by the second author (Adeli and Park, 1995) [15]. Most studies
5. Discussion
As evidenced by the previous section, the application of ML to
building SDPA problems has regained significant momentum within
the past decade since its dormancy from the late 1990's to the late
2000's. Most (if not all) of the reviewed studies have been exploratory
and there is no evidence that any of the applications have made their
way into practice. For ML-SDPA to advance from conception and re-
search into practice, there are several challenges that must be over-
come. A synthesis of those challenges as well as opportunities for fu-
ture work are presented in this section.
5.1. Data
One big contributor to the success of ML in other fields is the ac-
cess to adequate data. Although the amount of data required to
achieve reasonable performance for ML models depends on the prob-
lem and goal, it is essential to have sufficient high-quality data that
the sampled group could represent the true distribution. This enables
the adopted ML algorithm(s) to discover underlying patterns and pro-
duce predictive models that are truly generalizable within the prob-
lem scope. One of the major challenges in ML-SDPA applications is
that the datasets are often limited in quantity and diversity. In the
studies that sought to predict structural response and performance us-
ing ML models, the data was generated from nonlinear response his-
tory analyses by the researchers performing the study (e.g. Seo et al.,
2012; Moradi and Burton, 2018) [46,53]. However, to the author's
knowledge, none of these datasets have been made publicly available.
To have a truly representative dataset of structural response demands,
an open access repository should be instituted with rigorous quality
control measures. The recently established DesignSafe (Rathje et al.,
2017) [97] platform makes the creation of such a repository more fea-

Table 4
Summary of ML models developed using structural health monitoring and field reconnaissance data.

Study Structure Type Task Predictor Variables ML Algorithm(s) ML Model Performance Evaluation

Training/Testing Performance
Split Metric

Figueiredo et Aluminum Frame Damage detection and Auto- regressive parameters extracted from ANN NA ROC Curve
al., 2011 Test Specimen classification accelerometer recordings
Ghiasi et al., Steel Frame and Damage detection and Wavelet features extracted from accelerometer SVM 66–34 Mean Absolute
2016 Truss localization recordings Error
Rafiei and RC Tall Building Damage classification Frequency domain parameters from acceleration Neural Dynamic Multiple Accuracy
Adeli 2017 recordings Algorithm
Sun et al., RC Tall Building Structural response Structure (e.g. wall thickness) and building Kernel ridge and Kernel Multiple MARD
2019 prediction properties (e.g. height and location) SVM regression
Mangalathu Portfolio of Damage classification Building (e.g., number of stories) and site (e.g. DT, RF, KNN and LDA 70–30 Accuracy,
et al., 2019 Buildings shaking intensity) properties Precision and
Recall

10
H. Sun et al.
sible. The studies related to automatic information retrieval from vi-
sual media and models developed using field reconnaissance and SHM
data face similar challenges with lack of diversity in the adopted
datasets. Resources such as Structural ImageNet (Gao and Mosalam,
2018) [85], the National Hazard Engineering Infrastructure (NHERI)
RAPID facility (https://rapid.designsafe-ci.org/) and the DataCenter-
Hub (http://datacenterhub.org) will, over time, help alleviate this
challenge. Despite being relatively small (on the order of hundreds of
datapoints), there is more diversity in the datasets that have been
generated from physical experiments. In other words, the prior studies
in this area have utilized data generated from a broad range of experi-
ments conducted by many researchers (e.g. Jeon et al., 2014; Huang
and Burton, 2019; Hoang et al., 2019a) [68,70,98].
One partial solution to the shortage of data from physical experi-
ments is to incorporate domain knowledge within the ML algorithm,
which will reduce the complexity of the model space and conse-
quently reduce the amount of data that is needed to achieve good per-
formance. Transfer learning is another technique that can be used to
address the data-shortage issue. The basic idea behind transfer learn-
ing is that the knowledge acquired from training one model for a spe-
cific problem or domain can be “transferred” to another. Additionally,
there are procedures such as Monte Carlo Simulation, and generative
models such as Generative Adversarial Networks (GAN) (Goodfellow
et al., 2014) [99] and Variational Autoencoders (VAE) (Kingma and
Welling 2014) [100], which can augment existing datasets through
the generation of synthetic data. Future efforts should focus on all
four of these options: (i) collecting and curating more diverse
datasets, (ii) generating synthetic data, (iii) utilizing transfer learning
and (iv) incorporating domain knowledge in the design of the ML
model.
Another important concern is data quality, which is a common
challenge for ML models. Currently, there are no formal methods for
collecting and synthesizing datasets generated by the building SDPA
community. This lack of systematic curation procedures can lead to is-
sues such as the existence of outliers in the data, which can have an
adverse effect on the performance of ML models. This is especially
true for ML algorithms such as logistic regression, which are less ca-
pable of dealing with noise. There are anomaly detection procedures
such as DBSCAN (Ester et al., 1996) [101], K-Means clustering (Lloyd
and Stuart, 1982) [102] and Z-score (Rousseeuw and Hubert, 2011)
[103] that can be used to address outliers. Building SDPA domain-
specific procedures should also be implemented. In other words, the
universal data filtering procedure of ML models should be carefully
integrated with building SDPA domain knowledge. Ultimately, many
of the challenges related to the quality of building SDPA datasets can
be addressed if precise collection and processing techniques are estab-
lished and adopted.
Once the standardized benchmark datasets such as the ones sug-
gested in the previous paragraph have been created, a unified set of
performance measures and context-specific thresholds for determining
when models are deemed adequate, should be developed. This, along
with the creation of the ImageNet dataset [104], was a key factor in
the success of the field of computer vision.
5.2. Methods
A wide range of algorithms were used in the reviewed ML-SDPA
studies. Unfortunately, there is no consensus or general takeaway
about ML method-selection that could be inferred from the review. In
some studies, the author(s) chose to focus on a single method (e.g.
Morfidis and Kostinakis, 2017; Luo and Pall, 2018) [51,65]. However,
no clear compelling reason is ever provided for the selected method.
Other studies focused on comparing the performance of ML models
developed using different methods. However, the findings from these
comparative assessments are difficult to generalize because they are
11
Journal of Building Engineering 33 (2021) 101816
very much conditioned on the adopted dataset, and the model train-
ing (e.g. whether or not k-fold cross validation is adopted) and testing
(e.g. partition point for training-testing split, performance metric). Fu-
ture efforts should place a greater focus on analyzing domain specific
characteristics of the adopted datasets and applying knowledge-
informed strategies in selecting ML algorithms instead of using a
purely performance-driven search. For example, multi-output models
are especially useful for predicting backbone curves due to their capa-
bility in predicting multiple response variables. Addressing some of
the aforementioned challenges with creating systematic and well-
curated datasets would also help with the method-selection issue. The
advantage of having such benchmark datasets is that a standard
dataset will encourage focused attention on integrating domain
knowledge and the associated data patterns. Nevertheless, perfor-
mance-driven model selection is often the ideal solution when there is
no sense of how domain knowledge can be incorporated.
An immediate strategy that can be used to guide method-selection
is to begin by training and evaluating the performance of a linear (ba-
sis function) ML model (OLS, LASSO, ridge) for regression problems
and logistic regression for classification problems. With the exception
of very specific problems (e.g. computer vision or natural language
processing), linear models have been shown to perform reasonably
well, while being easy to implement (e.g. Burton et al., 2017; Man-
galathu and Jeon, 2018) [27,67] and more importantly, have high
model transparency and interpretability. In the event that the initial
linear models do not perform well, they should be further investigated
before moving on to more complex models. For example, the poor
performance could be because of simplicity of the linear ML model or
noisy data. The former situation requires exploring advanced (e.g.
non-parametric) models that can capture the complexity of data.
However, issues of noise in the data can be addressed by filtering.
5.3. Explainability and interpretability of machine learning models
One of the most significant challenges associated with ML-SDPA
models is explaining the feature effects and interpreting the physical
meaning of the model parameters. A commonly held view is that ML
models, especially the more advanced ones, are black boxes. In other
words, they are difficult to extract mechanics-based relationships be-
tween input (features) and output (response variables) parameters in
data-driven models. One approach to increasing model explainability
is to perform feature importance tests to understand their marginal ef-
fect on the response variable, which can then be benchmarked against
the fundamental principles that are known to govern the phenomena.
Statistical methods such as the F test (e.g. Sun et al., 2019) [55] and
analysis of variance (ANOVA) (e.g. Moradi and Burton, 2018) [53]
can be used to evaluate the relative strengths of association between
features and response variables. In addition, the partial dependence
plot (PD plot) and its variant, individual conditional expectation
curves (ICE) are also widely used [105,106]. Besides these general
measures of feature importance, model-specific techniques such as the
use of class activation mapping (CAM) to visualize focus areas on the
image of CNN models [107,108], have been developed. On the other
hand, some recent efforts on the interpretability of ML have demon-
strated the benefit of introducing domain knowledge into ML algo-
rithms by incorporating a physics-based loss function. A specific ex-
ample is to embed hard conditions with a Lagrange multiplier into the
loss function (e.g. Karpatne et al., 2017a; Muralidhar et al., 2018)
[109,110]. This approach provides a means to explain some of the ML
model by adding a physics-based law into the objective function. In
Karpatne et al. (2017b) [111], a spectrum of approaches is discussed,
whereby the wealth of domain knowledge is leveraged to improve the
performance of data-driven models. One recent article (Zhang and
Sun, 2020 [112]) in the SHM domain combines observed labeled field
data with unlabeled simulation data using a physics-guided neural
H. Sun et al.
network with a loss function that contains additional terms that re-
flect the discrepancy between observed and simulation output. Com-
bining ML and physics-based models remains a challenging problem
especially for the building SDPA community and will continue to be
explored in future research.
5.4. Overfitted ML models
Overfitted models, which result in inadequate performance outside
of the data used for training and/or testing, is a domain-agnostic chal-
lenge that is faced by the broader ML community. In Fig. 2, assuming
θ* presents the “true” model in the space Θ, overfitted models ( ) are
of high variance and do not generalize well while underfitted models
( ) have high bias and inferior predictive performance. In most cases
there is no so-called “true” model and the goal of ML is to find a bal-
anced model ( ). Standard ML procedures seek to address the overfit-
ting issue by utilizing training/testing split, k-fold cross validation,
bagging and bootstrapping, as well as other algorithm-specific ap-
proaches. For instance, the stochastic procedure used by RF to gener-
ate trees was intentionally developed to avoid the overfitting chal-
lenge associated with DT. It should be noted that overfitting is not
only associated with model training but also model selection. A so-
phisticated nonlinear model trained on a dataset with low-
dimensional (a small number of) features can also be overfitted. For
the building SDPA community, the application of domain knowledge
can also help with avoiding overfitting issues. The combination of a
data-driven procedure and domain knowledge, similar to the ap-
proach used to deal with data sparsity, may prove to be a powerful
combination. Although overfitting has been extensively studied in the
broader ML community, it could be more critical to building SDPA ap-
plications given the complexity of some of the mechanics-based rela-
tionships that data-driven models attempt to replicate. Consequently,
ML-SDPA models often require large amounts of data, better noise fil-
tering processes, and careful tuning to reduce the effects of overfit-
ting.
6. Conclusion
This paper provides a review of machine learning (ML) applica-
tions in building structural design and performance assessment
(SDPA). The vulnerability of aged structures under natural hazards
and the complexity of modern building systems call for efficient and
reliable frameworks for performance assessment, condition monitor-
ing and risk-informed decision making. The increase in computational
power in recent years has enhanced the capability of ML in complex
applications involving large-scale, high-dimensional nonlinear data.
With the advantages in pattern recognition and function approxima-
tion, ML offers a natural choice to help address the aforementioned
challenges in building SDPA.
In order to provide a good understanding of building SDPA prob-
lems that are suitable for ML applications and the available models
for solving specific problems, an overview of the ML methodology is
Fig. 2. Illustrating the tradeoff between bias and variance in machine learning
models.
12
Journal of Building Engineering 33 (2021) 101816
given, followed by a review of the supervised learning algorithms
most utilized in the building SDPA literature: Linear Regression, Ker-
nel Regression, Tree-Based Algorithms, Logistic Regression, Support
Vector Machine, K-Nearest Neighbors and Neural Networks. Next, the
ML applications in previous building SDPA studies are placed into the
following four categories and reviewed: (1) predicting structural re-
sponse and performance, (2) interpreting experimental data and for-
mulating models to predict component-level structural properties, (3)
information retrieval using images and written text and (4) recogniz-
ing patterns in structural health monitoring data. These successful ap-
plications have demonstrated the capability of ML in efficiently ex-
tracting information from multi-media building SDPA data and assess-
ing structural performance.
To bring ML into building SDPA practice, several key challenges
need to be addressed. First, adequate high-quality data sources essen-
tial for ML model development are currently unavailable within the
building SDPA community. Therefore, a unified effort is needed to
generate, collect and curate diverse datasets to an open-source reposi-
tory that can be populated by researchers and practitioners. This ef-
fort should also include the creation of benchmark datasets for spe-
cific SDPA sub-domains to align and focus research resources. Data
augmentation, transfer learning and reasonable design of ML algo-
rithms with domain knowledge can also help address the data spar-
sity. Second, previous studies did not establish general guidelines for
the selection of ML models. Future studies should incorporate more
knowledge-informed selection strategies. As a rule of thumb, initial
exploration should focus on simple linear models which are usually
easy to interpret and explain. The complexity of the data space can
also inform the model selection. Third, the results from ML models
are often difficult to interpret. This can be addressed by using impor-
tance testing to better understand the individual effects of features on
the response variable. The introduction of physics-based loss functions
can offer insight into ML model training and interpretation and can
potentially improve robustness. Lastly, overfitting is a significant issue
for ML models, especially when attempting to capture complex me-
chanics-based relationships in building SDPA problems. This issue can
be further studied by examining the SDPA data space and proposing
physics-based validation and evaluation techniques. Future research
should also focus on finding ways to combine data-driven procedures
with building SDPA domain knowledge, which will serve to boost per-
formance and provide model insights.
CRediT authorship contribution statement
Han Sun: Writing - original draft, Methodology, Writing - review
& editing, writing-original draft preparation and editing, literature re-
views, methodology presenting. Henry V. Burton: Conceptualization,
Writing - review & editing, Supervision. Honglan Huang: Writing -
review & editing.
Declaration of competing interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influ-
ence the work reported in this paper.
Acknowledgements
The research presented in this paper is supported by two National
Science Foundation CMMI research grants: No. 1538866 and No.
1554714.
H. Sun et al. Journal of Building Engineering 33 (2021) 101816

References [38] L. Breiman, Random forests, Mach. Learn. 45 (1) (2001) 5–32.
[39] C.M. Bishop, Pattern Recognition and Machine Learning, springer, 2006.
[40] L. Rosasco, E.D. Vito, A. Caponnetto, M. Piana, A. Verri, Are loss functions all
[1] K.P. Murphy, Machine Learning: a Probabilistic Perspective, MIT press, 2012.
the same?, Neural Comput. 16 (5) (2004) 1063–1076.
[2] J. Friedman, T. Hastie, R. Tibshirani, The Elements of Statistical Learning, vol.
[41] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (3) (1995) 273–
1, Springer series in statistics, New York, 2001.
297.
[3] H. Adeli, C. Yeh, Perceptron learning in engineering design, Comput. Aided Civ.
[42] Y.N. Wu, R. Gao, T. Han, S.C. Zhu, A tale of three probabilistic families:
Infrastruct. Eng. 4 (4) (1989) 247–256.
discriminative, descriptive, and generative models, Q. Appl. Math. 77 (2) (2019)
[4] J.J. Hopfield, Neural networks and physical systems with emergent collective
423–465.
computational abilities, Proc. Natl. Acad. Sci. Unit. States Am. 79 (8) (1982)
[43] R. Ghiasi, P. Torkzadeh, M. Noori, A machine-learning approach for structural
2554–2558.
damage detection using least square support vector machine based on a new
[5] R. Vanluchene, R. Sun, Neural networks in structural engineering, Comput.
combinational kernel function, Struct. Health Monit. 15 (3) (2016) 302–316.
Aided Civ. Infrastruct. Eng. 5 (3) (1990) 207–215.
[44] Y. Zhang, H.V. Burton, Pattern recognition approach to assess the residual
[6] D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-
structural capacity of damaged tall buildings, Struct. Saf. 78 (2019) 12–22.
propagating errors, Nature 323 (6088) (1986) 533–536.
[45] S. Mangalathu, H.V. Burton, Deep learning-based classification of earthquake-
[7] P. Hajela, L. Berke, Neurobiological computational models in structural analysis
impacted buildings using textual damage descriptions, International Journal of
and design, Comput. Struct. 41 (4) (1991) 657–667.
Disaster Risk Reduction 36 (2019) 101111.
[8] J. Ghaboussi, J. Garrett Jr., X. Wu, Knowledge-based modeling of material
[46] J. Seo, L. Dueñas-Osorio, J.I. Craig, B.J. Goodno, Metamodel-based regional
behavior with neural networks, J. Eng. Mech. 117 (1) (1991) 132–153.
vulnerability estimate of irregular steel moment-frame structures subjected to
[9] X. Wu, J. Ghaboussi, J. Garrett Jr., Use of neural networks in detection of
earthquake events, Eng. Struct. 45 (2012) 585–597.
structural damage, Comput. Struct. 42 (4) (1992) 649–659.
[47] E. Khojastehfar, S.B. Beheshti-Aval, M.R. Zolfaghari, K. Nasrollahzade, Collapse
[10] S. Masri, A. Chassiakos, T. Caughey, Identification of Nonlinear Dynamic
fragility curve development using Monte Carlo simulation and artificial neural
Systems Using Neural Networks, 1993.
network, Proc. Inst. Mech. Eng. O J. Risk Reliab. 228 (3) (2014) 301–312.
[11] H.T. Kang, C.J. Yoon, Neural network approaches to aid simple truss design
[48] F.K.G. Jough, S. Şensoy, Prediction of seismic collapse risk of steel moment
problems, Comput. Aided Civ. Infrastruct. Eng. 9 (3) (1994) 211–218.
frame mid-rise structures by meta-heuristic algorithms, Earthq. Eng. Eng. Vib. 15
[12] J.I. Messner, V.E. Sanvido, S.R. Kumara, StructNet: a neural network for
(4) (2016) 743–757.
structural system selection, Comput. Aided Civ. Infrastruct. Eng. 9 (2) (1994)
[49] J. Kiani, C. Camp, S. Pezeshk, On the application of machine learning techniques
109–118.
to derive seismic fragility curves, Comput. Struct. 218 (2019) 108–122.
[13] M. Elkordy, K. Chang, G. Lee, A structural damage neural network monitoring
[50] C.C. Mitropoulou, M. Papadrakakis, Developing fragility curves based on neural
system, Comput. Aided Civ. Infrastruct. Eng. 9 (2) (1994) 83–96.
network IDA predictions, Eng. Struct. 33 (12) (2011) 3409–3421.
[14] D. Gunaratnam, J. Gero, Effect of representation on the performance of neural
[51] K. Morfidis, K. Kostinakis, Seismic parameters’ combinations for the optimum
networks in structural engineering applications, Comput. Aided Civ. Infrastruct.
prediction of the damage state of R/C buildings using neural networks, Adv. Eng.
Eng. 9 (2) (1994) 97–108.
Software 106 (2017) 1–16.
[15] H. Adeli, H.S. Park, A neural dynamics model for structural optimization—
[52] Y. Zhang, H.V. Burton, H. Sun, M. Shokrabadi, A machine learning framework
theory, Comput. Struct. 57 (3) (1995) 383–390.
for assessing post-earthquake structural safety, Struct. Saf. 72 (2018) 1–16.
[16] Y. Reich, Machine learning techniques for civil engineering problems, Comput.
[53] S. Moradi, H.V. Burton, Response surface analysis and optimization of controlled
Aided Civ. Infrastruct. Eng. 12 (4) (1997) 295–310.
rocking steel braced frames, Bull. Earthq. Eng. 16 (10) (2018) 4861–4892.
[17] J.M. Buhmann, M. Held, Unsupervised learning without overfitting: empirical
[54] S. Moradi, H.V. Burton, I. Kumar, Parameterized fragility functions for
risk approximation as an induction principle for reliable clustering, International
controlled rocking steel braced frames, Eng. Struct. 176 (2018) 254–264.
Conference on Advances in Pattern Recognition, Springer, 1999.
[55] H. Sun, H. Burton, J. Wallace, Reconstructing seismic response demands across
[18] T.M. Huang, V. Kecman, I. Kopriva, Kernel Based Algorithms for Mining Huge
multiple tall buildings using kernel-based machine learning methods, Struct.
Data Sets, vol. 1, Springer, 2006.
Contr. Health Monit. 26 (7) (2019) e2359.
[19] P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple
[56] D. Hobbs, The compressive strength of concrete: a statistical approach to failure,
features, Proceedings of the 2001 IEEE Computer Society Conference on
Mag. Concr. Res. 24 (80) (1972) 127–138.
Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, IEEE, 2001.
[57] Z.P. Bažant, S. Zebich, Statistical linear regression analysis of prediction models
[20] R. Lienhart, J. Maydt, An extended set of haar-like features for rapid object
for creep and shrinkage, Cement Concr. Res. 13 (6) (1983) 869–876.
detection, Image Processing. 2002. Proceedings. 2002 International Conference
[58] Z.P. Bazant, J.C. Chern, Bayesian statistical prediction of concrete creep and
on, Vol. 1, IEEE, 2002.
shrinkage, ACJ Journal, Proceedings 81 (4) (1984) 319–330.
[21] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J.
[59] Z.P. Bažant, J.K. Kim, L. Panula, Improved prediction model for time-dependent
Comput. Vis. 60 (2) (2004) 91–110.
deformations of concrete: Part 1-Shrinkage, Mater. Struct. 24 (5) (1991) 327–
[22] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection,
345.
Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer
[60] Z.P. Bažant, J.K. Kim, Improved prediction model for time-dependent
Society Conference on, Vol. 1, IEEE, 2005.
deformations of concrete: Part 2—basic creep, Mater. Struct. 24 (6) (1991) 409.
[23] O. Bunke, B. Droge, Bootstrap and cross-validation estimates of the prediction
[61] A. Carpinteri, G. Ferro, S. Invernizzi, A truncated statistical model for analyzing
error for linear regression models, Ann. Stat. (1984) 1400–1424.
the size-effect on tensile strength of concrete structures, in: F.H. Wittmann,
[24] D.M. Powers, Evaluation: from Precision, Recall and F-Measure to ROC,
Zürich (Eds.), Proceedings of the 2 Nd International Conference on Fracture
Informedness, Markedness and Correlation, 2011.
Mechanics of Concrete Structures, Aedificatio, 1995.
[25] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep
[62] C.B. Haselton, A.B. Liel, S.C. Taylor-Lange, G.G. Deierlein, Calibration of model
convolutional neural networks, Adv. Neural Inf. Process. Syst. (2012) 1097–
to simulate response of reinforced concrete beam-columns to collapse, ACI
1105.
Struct. J. 113 (6) (2016).
[26] Y. Mack, T. Goel, W. Shyy, R. Haftka, Surrogate model-based optimization
[63] D.G. Lignos, H. Krawinkler, Deterioration modeling of steel components in
framework: a case study in aerospace design, Evolutionary Computation in
support of collapse prediction of steel moment frames under earthquake loading,
Dynamic and Uncertain Environments, Springer, 2007.
J. Struct. Eng. 137 (11) (2011) 1291–1302.
[27] H.V. Burton, S. Sreekumar, M. Sharma, H. Sun, Estimating aftershock collapse
[64] M. Naeej, M. Bali, M.R. Naeej, J.V. Amiri, Prediction of lateral confinement
vulnerability using mainshock intensity, structural response and physical damage
coefficient in reinforced concrete columns using M5′ machine learning method,
indicators, Struct. Saf. 68 (2017) 85–96.
KSCE Journal of Civil Engineering 17 (7) (2013) 1714–1719.
[28] R. Tibshirani, Regression shrinkage and selection via the lasso, J. Roy. Stat. Soc.
[65] H. Luo, S.G. Paal, Machine learning–based backbone curve model of reinforced
B 58 (1) (1996) 267–288.
concrete columns subjected to cyclic loading reversals, J. Comput. Civ. Eng. 32
[29] A.E. Hoerl, R.W. Kennard, Ridge regression: biased estimation for nonorthogonal
(5) (2018) 04018042.
problems, Technometrics 12 (1) (1970) 55–67.
[66] H. Luo, S.G. Paal, A locally weighted machine learning model for generalized
[30] H. Akaike, A new look at the statistical model identification, IEEE Trans.
prediction of drift capacity in seismic vulnerability assessments, Comput. Aided
Automat. Contr. 19 (6) (1974) 716–723.
Civ. Infrastruct. Eng. 34 (11) (2019) 935–950.
[31] G. Schwarz, others, Estimating the dimension of a model, Ann. Stat. 6 (2) (1978)
[67] S. Mangalathu, J.S. Jeon, Classification of failure mode and prediction of shear
461–464.
strength for reinforced concrete beam-column joints using machine learning
[32] J.H. Friedman, Multivariate adaptive regression splines, Ann. Stat. (1991) 1–67.
techniques, Eng. Struct. 160 (2018) 85–94.
[33] L. Breiman, J. Friedman, C.J. Stone, R.A. Olshen, Classification and Regression
[68] J.S. Jeon, A. Shafieezadeh, R. DesRoches, Statistical models for shear strength of
Trees, CRC press, 1984.
RC beam-column joints using machine-learning techniques, Earthq. Eng. Struct.
[34] T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning: Data
Dynam. 43 (14) (2014) 2075–2095.
Mining, Inference, and Prediction, Springer Science & Business Media, 2009.
[69] D.T. Vu, N.D. Hoang, Punching shear capacity estimation of FRP-reinforced
[35] Y. Freund, R.E. Schapire, A Desicion-Theoretic Generalization of On-Line
concrete slabs using a hybrid machine learning approach, Structure and
Learning and an Application to Boosting. European Conference on Computational
Infrastructure Engineering 12 (9) (2016) 1153–1161.
Learning Theory, Springer, 1995.
[70] H. Huang, H.V. Burton, Classification of in-plane failure modes for reinforced
[36] T. Hastie, S. Rosset, J. Zhu, H. Zou, Multi-class adaboost, Stat. Interface 2 (3)
concrete frames with infills using machine learning, Journal of Building
(2009) 349–360.
Engineering 25 (2019) 100767.
[37] Y.S. Abu-Mostafa, M. Magdon-Ismail, H.T. Lin, Learning from Data, vol. 4,
[71] S. Mangalathu, H. Jang, S.H. Hwang, J.S. Jeon, Data-driven machine-learning-
AMLBook, New York, NY, USA, 2012.
based seismic failure mode identification of reinforced concrete shear walls, Eng.

13
H. Sun et al.
Struct. 208 (2020) 110331.
[72] R. Szeliski, Computer Vision: Algorithms and Applications, Springer Science &
Business Media, 2010.
[73] Z. Zhu, S. German, I. Brilakis, Visual retrieval of concrete crack properties for
automated post-earthquake structural safety evaluation, Autom. ConStruct. 20
(7) (2011) 874–883.
[74] S. German, I. Brilakis, R. DesRoches, Rapid entropy-based detection and
properties measurement of concrete spalling with machine vision for post-
earthquake safety assessments, Adv. Eng. Inf. 26 (4) (2012) 846–858.
[75] S. German, J.S. Jeon, Z. Zhu, C. Bearman, I. Brilakis, R. DesRoches, et al.,
Machine vision-enhanced post earthquake inspection, J. Comput. Civ. Eng. 27 (6)
(2013) 622–634.
[76] C. Koch, S.G. Paal, A. Rashidi, Z. Zhu, M. König, I. Brilakis, Achievements and
challenges in machine vision-based inspection of large concrete structures, Adv.
Struct. Eng. 17 (3) (2014) 303–318.
[77] C. Koch, K. Georgieva, V. Kasireddy, B. Akinci, P. Fieguth, A review on
computer vision based defect detection and condition assessment of concrete and
asphalt civil infrastructure, Adv. Eng. Inf. 29 (2) (2015) 196–210.
[78] X. Kong, J. Li, Vision-based fatigue crack detection of steel structures using
video feature tracking, Comput. Aided Civ. Infrastruct. Eng. 33 (9) (2018) 783–
799.
[79] I. Brilakis, H. Fathi, A. Rashidi, Progressive 3D reconstruction of infrastructure
with videogrammetry, Autom. ConStruct. 20 (7) (2011) 884–895.
[80] S.G. Paal, J.S. Jeon, I. Brilakis, R. DesRoches, Automated damage index
estimation of reinforced concrete columns for post-earthquake evaluations, J.
Struct. Eng. 141 (9) (2015) 04014228.
[81] Y.J. Cha, W. Choi, G. Suh, S. Mahmoudkhani, O. Büyüköztürk, Autonomous
structural visual inspection using region-based deep learning for detecting
multiple damage types, Comput. Aided Civ. Infrastruct. Eng. 33 (9) (2018) 731–
747.
[82] F. Kucuksubasi, A. Sorguc, Transfer learning-based crack detection by
autonomous UAVs ArXiv Preprint ArXiv:180711785 2018 Proceedings of the
35th ISARC, Berlin, Germany, 2018, pp. 593–600 ISBN 978-3-00-060855-1.
[83] N.D. Hoang, Q.L. Nguyen, X.L. Tran, Automatic detection of concrete spalling
using piecewise linear stochastic gradient descent logistic regression and image
texture analysis, Complexity 2019 (2019).
[84] Y.J. Cha, K. You, W. Choi, Vision-based detection of loosened bolts using the
Hough transform and support vector machines, Autom. ConStruct. 71 (2016)
181–188.
[85] Y. Gao, K.M. Mosalam, Deep transfer learning for image-based structural
damage recognition, Comput. Aided Civ. Infrastruct. Eng. 33 (9) (2018) 748–
768.
[86] D. Gonzalez, D. Rueda-Plata, A.B. Acevedo, J.C. Duque, R. Ramos-Pollán, A.
Betancourt, et al., Automatic detection of building typology using deep learning
methods on street level images, Build. Environ. 177 (2020) 106805.
[87] S. Naito, H. Tomozawa, Y. Mori, T. Nagata, N. Monma, H. Nakamura, et al.,
Building-damage detection method based on machine learning utilizing aerial
photographs of the Kumamoto earthquake, Earthq. Spectra 36 (3) (2020) 1166–
1187 8755293019901309.
[88] A. Graves, J. Schmidhuber, Framewise phoneme classification with bidirectional
LSTM and other neural network architectures, Neural Network. 18 (5–6) (2005)
602–610.
[89] S. Hochreiter, J. Schmidhuber, LSTM can solve hard long time lag problems,
Advances in Neural Information Processing Systems, 1997.
[90] H. Sohn, C.R. Farrar, Damage diagnosis using time series analysis of vibration
signals, Smart Mater. Struct. 10 (3) (2001) 446.
[91] J.P. Lynch, Decentralization of Wireless Monitoring and Control Technologies
for Smart Civil Structures PhD Thesis Stanford University Stanford, CA, 2002.
[92] H. Young Noh, K. Krishnan Nair, D.G. Lignos, A.S. Kiremidjian, Use of wavelet-
based damage-sensitive features for structural damage diagnosis using strong
14
Journal of Building Engineering 33 (2021) 101816
motion data, J. Struct. Eng. 137 (10) (2011) 1215–1228.
[93] S.H. Hwang, D.G. Lignos, Assessment of structural damage detection methods
for steel structures using full-scale experimental data and nonlinear analysis,
Bull. Earthq. Eng. 16 (7) (2018) 2971–2999.
[94] E. Figueiredo, G. Park, C.R. Farrar, K. Worden, J. Figueiras, Machine learning
algorithms for damage detection under operational and environmental
variability, Struct. Health Monit. 10 (6) (2011) 559–572.
[95] M.H. Rafiei, H. Adeli, A novel machine learning-based algorithm to detect
damage in high-rise building structures, Struct. Des. Tall Special Build. 26 (18)
(2017) e1400.
[96] S. Mangalathu, H. Sun, C.C. Nweke, Z. Yi, H.V. Burton, Classifying earthquake
damage to buildings using machine learning, Earthq. Spectra 36 (1) (2020) 183–
208.
[97] E.M. Rathje, C. Dawson, J.E. Padgett, J.P. Pinelli, D. Stanzione, A. Adair, et al.,
DesignSafe: new cyberinfrastructure for natural hazards engineering, Nat.
Hazards Rev. 18 (3) (2017) 06017001.
[98] N.D. Hoang, X.L. Tran, H. Nguyen, Predicting ultimate bond strength of
corroded reinforcement and surrounding concrete using a metaheuristic
optimized least squares support vector regression model, Neural Comput. Appl.
(2019) 1–21.
[99] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et
al., Generative adversarial nets, Advances in Neural Information Processing
Systems, 2014.
[100] D.P. Kingma, M. Welling, Auto-encoding variational bayes, Proceedings of the
2nd International Conference on Learning Representations (ICLR), 2014 ArXiv
Preprint ArXiv:13126114 2013.
[101] M. Ester, H.P. Kriegel, J. Sander, X. Xu, others, A density-based algorithm for
discovering clusters in large spatial databases with noise, Kdd 96 (1996).
[102] S. Lloyd, Least squares quantization in PCM, IEEE Trans. Inf. Theor. 28 (2)
(1982) 129–137.
[103] P.J. Rousseeuw, M. Hubert, Robust statistics for outlier detection, Wiley
Interdisciplinary Reviews: Data Min. Knowl. Discov. 1 (1) (2011) 73–79.
[104] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, L. Fei-Fei, Imagenet: a large-scale
hierarchical image database. Computer Vision and Pattern Recognition, 2009.
CVPR 2009. IEEE Conference on, Ieee, 2009.
[105] J.H. Friedman, Greedy function approximation: a gradient boosting machine,
Ann. Stat. (2001) 1189–1232.
[106] A. Goldstein, A. Kapelner, J. Bleich, E. Pitkin, Peeking inside the black box:
visualizing statistical learning with plots of individual conditional expectation, J.
Comput. Graph Stat. 24 (1) (2015) 44–65.
[107] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features
for discriminative localization, Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2016.
[108] R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-
cam: visual explanations from deep networks via gradient-based localization,
Proceedings of the IEEE international conference on computer vision (2017)
618–626.
[109] A. Karpatne, G. Atluri, J.H. Faghmous, M. Steinbach, A. Banerjee, A. Ganguly, et
al., Theory-guided data science: a new paradigm for scientific discovery from
data, IEEE Trans. Knowl. Data Eng. 29 (10) (2017) 2318–2331.
[110] N. Muralidhar, M.R. Islam, M. Marwah, A. Karpatne, N. Ramakrishnan,
Incorporating prior domain knowledge into deep neural networks, 2018 IEEE
International Conference on Big Data (Big Data), IEEE, 2018.
[111] A. Karpatne, W. Watkins, J. Read, V. Kumar, Physics-guided Neural Networks
(PGNN): An Application in Lake Temperature Modeling, SIAM International
Conference on Data Mining (SDM), 2018.
[112] Z. Zhang, C. Sun, Structural damage identification via physics-guided machine
learning: a methodology integrating pattern recognition with finite element
model updating, Struct. Health Monit. (2020), https://doi.org/10.1177/
1475921720927488.