Mobile Hyperspectral Imaging for

Material Surface Damage Detection

Sameer Aryal 1; ZhiQiang Chen, M.ASCE 2; and Shimin Tang 3

Abstract: Many machine vision–based inspection methods aim to replace human-based inspection with an automated or highly efficient
procedure. However, these machine-vision systems have not been endorsed entirely by civil engineers for deployment in practice, partially
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

due to their poor performance in detecting damage amid other complex objects on material surfaces. This work developed a mobile hyper-
spectral imaging system which captures hundreds of spectral reflectance values in a pixel in the visible and near-infrared (VNIR) portion of
the electromagnetic spectrum. To prove its potential in discriminating complex objects, a machine learning methodology was developed with
classification models that are characterized by four different feature extraction processes. Experimental validation showed that hyperspectral
pixels, when used conjunctly with dimensionality reduction, possess outstanding potential for recognizing eight different surface objects
(e.g., with an F1 score of 0.962 for crack detection), and outperform gray-valued images with a much higher spatial resolution. The authors
envision the advent of computational hyperspectral imaging for automating damage inspection for structural materials, especially when
dealing with complex scenes found in built objects in service. DOI: 10.1061/(ASCE)CP.1943-5487.0000934. © 2020 American Society
of Civil Engineers.
Author keywords: Machine vision; Hyperspectral imaging; Damage detection; Machine learning; Dimensionality reduction.

Introduction inspection, including the Manual for Bridge Element Inspection

Civil engineering structures are vital for a society’s prosperity and
quality of life in general. To ensure the serviceability and safety of
structures, different technology-based inspection and monitoring
methods have been researched. These technologies generally can
be categorized into two types. The first is nondestructive evaluation
(NDE) through advanced sensing technologies (e.g., X-ray, micro-
wave, thermal, and ultrasonic imaging) (Cawley 2018), most of
which aim to detect subsurface anomalies or damage. The second
category includes various structural health monitoring (SHM)
methods (Sohn et al. 2003), which seek to monitor the dynamic
responses and identify the intrinsic parameters or changes in struc-
tures. Even with these technology-based inspection or monitoring
methods, the reality is that, at least for transportation structures that
are managed by DOT agencies in most states in the US, manual or
visual inspection is considered the mainstream approach (Olsen
et al. 2016). This mainly is because surface damage (e.g., cracks
and corrosion) on structural materials, compared with subsurface
damage, is more commonplace and can be viewed as a clue to in-
ternal damage or possible degrading of the system integrity. For
bridges, standard manuals and guidelines exist for practical
1
Graduate Student, Dept. of Civil and Mechanical Engineering, Univ. of
Missouri Kansas City, FH 5110 Rockhill Rd., Kansas City, MO 64110.
Email: sayd8@mail.umkc.edu
2
Associate Professor, School of Computing and Engineering, Univ. of
Missouri Kansas City, FH 5110 Rockhill Rd., Kansas City, MO 64110 (cor-
responding author). ORCID: https://orcid.org/0000-0002-0793-0089.
Email: chenzhiq@umkc.edu
3
Ph.D. Candidate, Dept. of Computer Science and Electrical Engineer-
ing, Univ. of Missouri Kansas City, FH 5110 Rockhill Rd., Kansas City,
MO 64110. Email: st78d@mail.umkc.edu
Note. This manuscript was submitted on March 30, 2020; approved on
July 15, 2020; published online on November 9, 2020. Discussion period
open until April 9, 2021; separate discussions must be submitted for indi-
vidual papers. This paper is part of the Journal of Computing in Civil En-
gineering, © ASCE, ISSN 0887-3801.
(AASHTO 2019) and the Guidelines for Inspecting Complex Com-
ponents of Bridges (Leshko 2002). Due to the low cost and ubiqui-
tous availability of optical sensors (i.e., digital cameras), it is no
surprise that optical imaging has become widely adopted for struc-
tural inspection, wherein digital images are recorded for records or
postinspection analysis (Olsen et al. 2016). Subsequently, this has
motivated the development of machine vision techniques to aid or
automate engineering inspection for civil structures.
Machine vision is a technical field that concerns the develop-
ment of digital imaging methods and the use of image processing
or computer vision algorithms for the extraction of useful informa-
tion from images (Morris 2004). With the advent of early digital
cameras, researchers in the 1980s and 1990s used simple digital
filters, including various edge-detection methods, for image-based
damage detection for structural materials (Cheng and Miyojim
1998; Ritchie 1987; Ritchie 1990). To further automate the process
of image capturing, researchers also strive to develop operationally
efficient methods that are expected to mitigate the human cost.
These methods include ground vehicle–based imaging, aerial ve-
hicle–based imaging, and crowdsourcing-based imaging (e.
g., Isawa et al. 2005; Kim et al. 2015; Lattanzi and Miller
2013; Ozer et al. 2015; Tung et al. 2002; Zhang and Elaksher
2012). For example, Ho et al. (2013) developed a system with
three cameras attached to a cable-climbing robot to detect surface
damage. Yeum and Dyke (2015) proposed the use of unmanned
aerial vehicles (UAVs) for remote imaging and damage detection.
Chen et al. (2015) proposed a mobile-cloud infrastructure enabled
approach that exploits collaborative mobile and cloud computing to
harness crowdsourcing for damage inspection.
With the abundance of these optical imaging platforms, one prom-
ising fact is the ease of obtaining imagery databases. Besides the ba-
sic image processing methods, these databases enable the adoption of
a machine learning (ML) paradigm. To this end, many machine learn-
ing methods feature the use of supervised or nonsupervised classi-
fiers (Chen et al. 2011; Gavilán et al. 2011; Kaseko and Ritchie
1993; Liu et al. 2002; Prasanna et al. 2014; Zakeri et al. 2017).

© ASCE 04020057-1 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 In recent years, coincident with the advances in artificial intelli-
gence (AI) and particularly the development of deep learning
(DL) techniques, many have heralded the era of AI-enabled damage
inspection. A simple search through Google Scholar, using the
combined keywords of crack detection, convolutional neural net-
work (CNN), and image returned more than 1,200 articles from
the period January 2016 to May 2020. Zhang et al. (2016) first used
a CNN model as a feature extractor, then fed the features into a
classification model for the detection of cracks in images. Such
a CNN-based machine learning approach then was adopted in
many other similar efforts (e.g., Alipour et al. 2019; Cha et al. 2017;
Ni et al. 2019).
One may expect that by duly considering these advances in
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
AI-enabled image-based damage detection and the lowering cost
of mobile or edge computing devices, autonomous damage inspec-
tion may become a reality. However, these machine-vision systems
have not been endorsed entirely by civil engineers for deploying
these techniques in practice. The authors argue that if the scene
complexity captured in digital images is not acknowledged, the
pace of automation ultimately would be hindered. In the case of
concrete structures in service, the scenes in images often are a mix-
ture of characteristic material texture, possible damage, and other
artifacts, such as artificial marking, vegetation, moisture, oil spill,
discoloring, and uneven illumination (Chen and Hutchinson 2010).
This implies that any image-based machine learning method or
end-to-end deep learning method may encounter the infamous
issue of generalization. In other words, if such an autonomous
image-based system is deployed in the field, its core detection
component (i.e., a classification model), even one trained based
on a relatively large data set with complex scenes, can overfit
the training data but cannot generalize to the multitude of different
and arbitrary scenes that are not in the prepared data sets for model
training.
To resolve this challenge, one obvious solution is to continue
developing much larger data sets, given the power of deep learning
with an architecture that potentially can accommodate any scale
of data sizes and any complexity in field scenes. However, this
inevitably triggers the issue of labeling big data (e.g., pixelwise
labeling of cracks and other artifacts), which is expensive and
time-consuming (Roh et al. 2019). Another approach is to resort
to transfer learning and use small data sets enhanced by effective
data augmentation techniques to obtain the notion of learning from
small data using DL models. Tang and Chen (2017) developed a
novel data augmentation technique for accommodating deep trans-
fer learning from small data. Regardless of the potential success of
these solutions, with the use of red, green, and blue (RGB) images,
these methods can only asymptotically match the intelligence of
trained inspectors (who provide labeled training data), although
possibly with much higher efficiency than human inspectors. In
other words, there is a performance ceiling that limits the capacity
of regular RGB images unless the machine intelligence supersedes
that of human beings.
An alternative solution is to break out the normal matching of
human vision. An emerging technology for nondestructive inspec-
tion is hyperspectral imaging (HSI). In a hyperspectral image,
a pixel contains tens to thousands of digital values in different
spectral bands in the visible and near-infrared (VNIR) portion of
the electromagnetic spectrum, at which each digital value represents
either the reflectance property or the transmittance property of a
material in one band. Such a high-dimensional spectral profile is
not directly visible to human eyes, which, roughly speaking, re-
spond to three discrete bands (RGB) (Kaiser and Boynton 1996).
Scientific knowledge of hyperspectral imaging and analysis is well
archived in the knowledge body of hyperspectral spectroscopy
© ASCE 04020057-2
J. Comput. Civ. Eng., 2021, 35(1): 04020057
(Siesler et al. 2008). In the context of image-based damage detec-
tion, HSI provides a significant possibility of detecting and iden-
tifying the presence of material damage amid complex scenes on
surfaces of civil structures.
The authors developed a mobile hyperspectral imaging system
for both ground and aerial vehicle–based remote sensing. With this
mobile HSI system and preliminary observation (e.g., by plotting
spectral profiles for different material surface objects), it is hypoth-
esized that material damage (e.g., cracks) and other complex arti-
facts on structural surfaces can be detected effectively with high
performance. The ensuing research problem is whether the resulting
machine-learning model based on hyperspectral pixels can outper-
form the same model trained based on regular images with a high
spatial resolution (i.e., those based on panchromatic or true-color
imaging). The essential contribution of this study is the proven ef-
fectiveness of HSI for material damage detection with complex
scenes, and its superb capacity compared with high-resolution pan-
chromatic (gray) images. Specifically, unlike any existing image-
based damage detection method, the proposed machine-learning
framework deals with the detection problem with many semanti-
cally rich structural-surface materials and objects, including con-
crete, asphalt, crack, dry vegetation, green vegetation, water, oil, and
artificial markings. Such a plentiful amount of surface object cat-
egories are not dealt with in the literature of image-based damage
detection, but are commonly found in civil structures in service. The
resulting inference model, although it falls in the paradigm of
traditional machine learning, can be implemented readily on an
onboard computer, hence providing the possibility of developing
a real-time mobile HSI system for both hyperspectral imaging and
damage detection. The significant contribution also lies in the se-
mantically labeled hyperspectral data set (Chen 2020), which pro-
vides an unprecedented basis for research in hyperspectral machine
vision and engineering inspection automation.
This paper first briefly introduces the concept of HSI, and
describes a mobile HSI system. Next, the machine learning meth-
odology is introduced with a focus on proving the concept of
HSI-based detection and its competitive performance. Performance
is evaluated and discussed in terms of four classification models,
and conclusions are presented.
Hyperspectral Imaging
Imaging Mechanism
The concept of HSI originated in the 1980s when Goetz and
colleagues at the Jet Propulsion Laboratory (JPL) began develop-
ing the seminal Airborne Visible/Infrared Imaging Spectrometer
(AVIRIS) (Green et al. 1998; Plaza et al. 2009). The AVIRIS sensor
measures the upwelling radiance with 224 contiguous spectral
bands at 10 nm intervals across the spectrum from 400 to 2,500 nm.
These radiance spectra are measured as push-broom scanning im-
ages with a swath width of 11 km and a flight length typically of
10–100 km (both at a spatial resolution of 20 m=pixel). Similar to
any common image, a hyperspectral image is formed by sampling
in a two-dimensional (2D) domain, generating a matrix of pixels.
Unlike gray-level or RGB images, a hyperspectral pixel consists of
a large number of intensity values sampled at various narrow spec-
tral bands that represent the contiguous spectral curve at the pixel.
A hyperspectral image is commonly called a hyperspectral cube,
because it is a three-dimensional (3D) matrix (two dimensions
in spatial directions and one in the spectral direction). Using an
AVIRIS image and taking a scanning distance of 50 km, one
can create a hyperspectral cube with a size of 550 × 2,500 × 224.
J. Comput. Civ. Eng.
 With advances in HSI and especially sensors that are not for
orbital or airborne platforms, the acquisition of hyperspectral cubes
can be realized with other mechanisms. In addition to the spatial
scanning or push-broom imaging mechanism, two other mecha-
nisms (spectral scanning and spatial-spectral scanning) were devel-
oped for HSI applications in medical and biological sciences (Lu
and Fei 2014). These three scanning techniques require complex
postprocessing steps to achieve the end product, a data cube. The
fourth mechanism is nonscanning, and is termed snapshot imaging
(Johnson et al. 2004). Snapshot imaging acquires spectral pixels
in a 2D field of view simultaneously, without the requirement of
trajectory flights or using any moving parts in the imager. There-
fore, this snapshot mechanism also is referred to as real-time HSI
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
(Bodkin et al. 2009), and can achieve much higher frame rates and
higher signal-to-noise ratios, and can provide hyperspectral cubes
immediately after every action of capturing, as in a regular digital
camera. Due to this property, real-time or snapshot HSI opens up
significant opportunities for its use in portable or low-altitude re-
mote sensing.
Hyperspectral Image Computing
A hyperspectral cube can be denoted hðx; y; sÞ, where ðx; yÞ are
the spatial coordinates (in terms of pixels) and s is the spectral
coordinate in terms of spectral wavelength—i.e., for visible bands,
s ∈ ½400,600 nm, and for visible to near-infrared bands, s ∈ ½400;
1,000 nm. At a select location of ðx; yÞ, therefore, hðx; y; sÞ rep-
resents a spectral profile when plotted against the spectral variable
s. In the remote-sensing context (not in a medical or biological
context), i.e., when the data cube is acquired in the air, the meas-
urement at the sensor is the upwelling radiance. In general, the re-
flectance property of a material at the ground nominally does not
vary with solar illumination or atmospheric disturbance. Therefore,
the reflectance profile of a pixel reflects the characteristic or sig-
nature of the material at that pixel. Therefore, a raw-radiance data
cube needs to be corrected to generate a reflectance cube, consid-
ering the environmental lighting and the atmospheric distortion.
This process is called atmospheric correction (Adler-Golden et al.
1998). For this, many physics-based algorithms exist, including the
widely used fast line-of-sight atmospheric analysis of spectral
hypercubes (FLAASH) (Cooley et al. 2002).
The most frequent use of hðx; y; sÞ as a reflectance cube is
material detection. Compared with single-band or three-band pixels
in common images, the distinct property of hyperspectral images
is the high dimensionality of the pixels. This high-dimensionality
is the foundation of spectroscopy for material identification. To
achieve such identification, various unmixing methods have been
developed to distinguish the statistically significant material types
and their abundance levels underlying the spectral profile of a pixel
(Bioucas-Dias et al. 2012; Heylen et al. 2014). If the concern is not
a specific material (that consists of single molecules or atoms), but
an object is of interest, extracting and selecting descriptive features
at hyperspectral pixels becomes the first stage. This stage, com-
monly called feature description, is as common as it is in a typical
machine-learning process. If pixel-based object detection is con-
cerned, the dimensionality of hyperspectral pixels poses the first
challenge. To avoid the so-called curse of dimensionality when
using machine learning methods, early efforts emphasized the
development of dimensionality reduction (DR) methods for select-
ing representative feature values across spectral bands. The most
commonly used DR methods are mathematically linear, including
principal component analysis (PCA) and linear discriminant analy-
sis (LDA) (e.g., Bandos et al. 2009; Chen and Qian 2008; Hang
et al. 2017; Palsson et al. 2015). Other advanced, and usually
© ASCE 04020057-3
J. Comput. Civ. Eng., 2021, 35(1): 04020057
nonlinear, methods, such as the orthogonal subspace projection
method (Harsanyi and Chang 1994) and the locally linear embedding
method (Chen and Qian 2007), are also found in the literature. For
semantic segmentation–based object detection, feature description
needs to consider the spatial distribution of hyperspectral pixels.
The most straightforward treatment is to extract feature vectors based
on the dimensionality-reduced hyperspectral pixels, as in extracting
features in a gray or a color image. Advanced methods in recent years
have concerned the extraction of spatial–spectral features in three di-
mensions (Fang et al. 2018; Hang et al. 2015; Zhang et al. 2014). For
the classification design, as the second stage of a conventional
machine-learning problem, the commonly used classical classifiers
include artificial neural network (ANN), support vector machine
(SVM), decision tree, k-nearest neighbor (KNN), and nonsupervised
clustering techniques such as k-means (Goel et al. 2003; Ji et al. 2014;
Marpu et al. 2009; Mercier and Lennon 2003).
As deep learning became a focus in the machine vision com-
munity, DL models have been developed in recent years which
potentially unify DR, feature description, and classification in
end-to-end architectures for hyperspectral image–based object de-
tection. Chen et al. (2014) integrated PCA, stacked autoencoder
(SAE), and logical regression into one DL architecture (Chen
et al. 2014). Convolutional neural networks (CNNs), as commonly
adopted in regular machine-vision problems, were introduced in
subsequent years (Chen et al. 2016; Hu and Jia 2015; Li et al.
2017). These advances imply the advent of computational hyper-
spectral imaging or hyperspectral machine vision, which may im-
pact many engineering fields, including damage inspection in civil
engineering.
Mobile Hyperspectral Imaging System
A mobile HSI system for ground-level and low-altitude remote
sensing was developed by the authors. The imaging system consists
of a Cubert S185 FireflEYE snapshot camera (Ulm, Germany) and
a mini-PC server for onboard computing and data communication
(Cubert 2018). For ground-based imaging, the system is mounted
to a DJI gimbal (Shenzhen, China) that provides two 15-W,
1580-mA·h batteries for powering both the imaging payload and
the operation of the gimbal. Fig. 1(a) shows the gimbaled imaging
system ready for handheld or other ground-based HSI. To enable
low-altitude remote sensing, an unmanned aerial vehicle is used,
and the gimbaled system can be installed easily on the UAV for
remote sensing [Fig. 1(b)].
The Cubert camera has a working wavelength range of 450–
950 nm covering most of the VNIR wavelengths, spanning 139
bands, with a spectral resolution of 8 nm in each band. The raw
sensor has a snapshot resolution of 50 × 50 spectral channels.
Natively, the imager captures radiance images; with a proper cal-
ibration procedure and the internal processing in the onboard com-
puter, the camera can output reflectance images directly. To do so, a
calibration process starts with the use of a standard white reference
board, achieving a data cube for the standard whiteboard (denoted
hW ). Then a perfect dark cube is obtained (by covering the lens
tightly with a black cap), denoted hB . The relative reflectance im-
age, h, is calculated given a radiance cube hR
hR − hw
h¼ ð1Þ
hw − hB
Following Eq. (1), a reflectance image can be produced by the
camera system directly or can be postprocessed from the generated
radiance image.
One unique feature of the Cubert HSI system is its dual acquis-
ition of hyperspectral cubes and a companion gray level–intensity
J. Comput. Civ. Eng.
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
Fig. 1. Mobile hyperspectral imaging system: (a) gimbaled setup for ground-based imaging; and (b) ready-to-fly setup with a DJI UAV.
(Images by ZhiQiang Chen.)
image. The gray-level image has an identical field of view as the
hyperspectral cube but has a much higher spatial resolution,
1,000 × 1,000. Denoting this gray image as gðu; vÞ, gðu; vÞ and
hðx; y; sÞ can be fused to achieve a hyperspectral cube with a
higher resolution; at its peak, one can obtain a cube with a size of
1,000 × 1,000 × 139. This process is called pansharpening; to
obtain smooth sharpening effects, many algorithms exist (Loncan
et al. 2015). However, pansharpening, which can provide visually
appealing hyperspectral images (if visualized in terms of pseudo-
color images) as a very effective data fusion technique, does not
offer information other than the original low-resolution hyperspec-
tral cube and the high-resolution gray image.
Machine Learning Design
Rationale and Overview of Methodology
A common machine-learning task, using either gray-level or RGB
color images, is extracting visual features based on the spatial
distribution of pixel intensities. When the features are extracted,
they are fed into a classifier for supervised or unsupervised clas-
sification. This two-stage process, a traditional machine learning
paradigm, can be replaced with an end-to-end methodology by
adopting the latest deep learning architecture, which possibly pro-
duces higher detection performance than the traditional process.
This was not adopted in this work for several reasons.
First, the high-capacity nature of a DL architecture lies in its
capacity to represent the intrinsic structure of a complex object via
its usually hundreds of networked layers of neural nets. To support a
converging fitting for such a complex model, a very large number of
labeled data points (or ground-truth data) is demanded for a regular
object detection task. In previous DL-based efforts for hyperspectral
data learning, such large-scale data sets have existed; for example,
the most used AVIRIS Indian Pines data (Baumgardner et al. 2015).
However, all of them are Earth observation (EO) images (Graña
et al. 2020). No data set was collected and labeled for damage
detection for structural materials using ground-based HSI equip-
ment. Centering around these EO images, DL methodologies only
recently have appeared for advanced spatial–spectral and semantic
segmentation. The conventional pixel-level techniques featuring
© ASCE 04020057-4
J. Comput. Civ. Eng., 2021, 35(1): 04020057
manually crafted feature description and classifier model selection,
including unsupervised DR, material and object detection, and seg-
mentation, are still considered as the trusted and robust approach.
Second, the authors argue that if a DL architecture is used, the feature
description and classification inevitably are coupled in an end-to-end
architecture, and the native power of hyperspectral pixels in the spec-
tral dimension and the gray-intensity pixels in the spatial dimensions
would not be revealed straightforwardly. This study adopted the tra-
ditional machine learning methodology; specifically, four different
feature description schemes were designed, and a standard classifi-
cation model was selected. This strategy properly facilitated proving
the primary hypothesis and the ensuing research problem of justify-
ing the competitiveness of hyperspectral features. Third, the resulting
methodology in this paper can be implemented readily as a robust
filtering mechanism for real-time imaging and processing. The future
development of DL and field implementation is discussed
subsequently.
To realize the primary goal of proving the hypothesis of this
study, four machine learning models were designed. The main
distinction between these four machine-learning models is their
feature description process. Fig. 2 presents the methodology com-
ponents for the four machine learning models. For the classifier,
the seminal support vector machine (SVM) was adopted, and is
reviewed subsequently.
These machine models are
1. Model 1 (M1): hyperspectral pixels with spectral values are used
directly as feature vectors; namely, hðx; y; sÞ at ðx; yÞ is used
directly as a feature vector, where s ∈ f1; 2; : : : ; 139g. This
model is denoted M1(HYP), where HYP represents the feature
extraction process.
2. Model 2 (M2): linear PCA is applied as an additional feature
selection step to reduce the dimensionality. The profile of
hðx; y; sÞ at ðx; yÞ is reduced from 139 to 6 dimensions and
becomes h 0 ðx; y; kÞ, where k ∈ f1; 2; 3; : : : ; 6g. This model is
denoted M2(HYP_PCA).
3. Model 3 (M3): feature vectors are extracted based on the
companion gray-level images, gðu; vÞ. As a result, a feature
vector at a 20 × 20 neighborhood in gðu; vÞ maps to the hyper-
spectral pixel at ðx; yÞ. To extract the gray-level features within a
sliding 20 × 20 neighborhood in gðu; vÞ, a variant of the widely
J. Comput. Civ. Eng.
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

Fig. 2. Methodology for feature description and different machine learning models.

used gradient-based feature extractor, the histogram of gradients
(HOG) is adopted. The resulting model is denoted
M3(GL_HOG).
4. Model 4 (M4): feature vectors are based on the combined use of
the feature vectors used in Model 2 and Model 3. By concat-
enating the two feature vectors, imagery information from both
the hyperspectral pixel–based spectrum and the gray value–
based spatial distribution is fused. This model is denoted
M4(GL_HOG+HYP_PCA), and GL_HOG+HYP_PCA repre-
sents the fourth feature extraction process in this paper.
With these models, the specific objectives of proving the hy-
potheses were
• Objective 1 was to evaluate the performance of the classification
model M1(HYP) and to conclude whether a hyperspectral pixel
is effective in recognizing the underlying object types, including
structural damage given a complex scene.
• Objective 2 was to evaluate and compare the performance of
M1(HYP) and M2(HYP_PCA) to conclude whether dimension-
ality reduction is effective in terms of improving the discrimi-
nation of different objects.
• Objective 3 was to evaluate and compare the performance of
M2(HYP_PCA) and M3(GL_HOG) to conclude whether hyper-
spectral pixels are more effective than high-resolution gray-level
images in identifying complex object types.
• Objective 4 was to evaluate the performance of the classification
model of M4(HYP_PCA, GL_HOG) to conclude whether,
through simple data fusion, the combined hyperspectral and gray-
level features provide more-competitive detection performance.
Feature Selection and Extraction Methods
The four models essentially differ in the use of the feature descrip-
tion processes based on the original hyperspectral data instance
(a data cube and a companion gray-level image). Two basic feature
description methods, the principal component analysis method as a
linear feature selection method and a variant of the histogram of
oriented gradients (HOG) method as a feature extraction method,
are involved in the four processes.
Principal Component Analysis
A hyperspectral profile at ðx; yÞ, or denoted as a set {hðx; y; sÞjs ∈
½1; 2; : : : ; 139}, if treated as a feature vector, gives rise to a
139 × 1 feature vector. As mentioned previously, such high-
dimensionality readily leads to poor performance when training a
classification model (particularly when the training data are few,
and the model itself cannot accommodate the high-dimensional
space). Principal component analysis is a commonly used method
for dimensionality reduction, which in many cases can greatly

© ASCE 04020057-5 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 improve classification performance. In general, PCA is a statistical
procedure that converts a set of data points with a high dimension
of possibly correlated variables into a set of new data points that
have linearly uncorrelated variables or are mutually orthogonal,
called principal components (PCs). The component scores associ-
ated with the orthogonal PCs construct the coordinates in a new
space spanned by the PCs. By selecting several dominant PCs (less
than the original dimension of the data point), the original data
points can be projected to a lower-dimension subspace. Therefore,
the select PC scores for an originally high-dimension feature point
can be used as a dimension-reduced feature vector. In this study, the
first six components were selected empirically after testing from the
first three to the first seven components. In addition, a normaliza-
tion step was conducted for all PCA features at all pixels. The PC
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
scores, when visualized, can be used as a straightforward approach
to inspecting the discrimination potential of the features, which oth-
erwise are not possible if the original high-dimension hyperspectral
profiles are plotted. These illustrations are described subsequently.
Histogram of Oriented Gradients
The histogram of oriented gradient feature descriptor characterizes
the contextual texture or shape of objects in images by counting the
occurrence of gradient orientations in a select block in an image or
in the whole image. It was first proven effective by Dalal and Triggs
(2005) in their seminal effort for pedestrian detection in images;
since then, HOG has been applied extensively for different objec-
tive detection tasks in the literature of machine vision. HOG differs
from other scale-invariant or histogram-based descriptors in that its
extraction is computed over a dense grid of uniformly spaced cells,
and it uses overlapping local contrast normalization for improved
performance. There are many variants of HOG descriptors for im-
proving the robustness and the accuracy; a commonly used variant
is HOG-UoCTTI (Felzenszwalb et al. 2010). The original HOG
extractor and its variants are formulated over a gray image, which
is convenient to the research in this study, in which a companion
gray-image exists for every hyperspectral cube.
Considering the resolution compatibility between the hyper-
spectral cube and the companion gray-level image, the feature ex-
traction was conducted in a 20 × 20 sliding neighborhood in a gray
image. Within this neighborhood, four histograms of undirected
gradients were averaged to obtain a no -dimensional histogram
(i.e., binned per their orientation into nine bins, no ¼ 9) and a sim-
ilar operation was performed for the directed gradient to obtain a
2no -dimensional histogram (i.e., binned according to gradient into
18 bins). Along with both directed and undirected gradient, the
HOG-UoCTTI also computed another four-dimensional texture-
energy feature. The final descriptor was obtained by stacking the
averaged directed histogram, averaged undirected histogram, and
four normalized factors of the undirected histogram. This led to
the final descriptor of size 4 þ 3 × no (i.e., a 31 × 1 feature vector).
Multiclass Support Vector Machine
In the traditional machine vision literature, the combined use of
SVM as a classifier and HOG as a feature descriptor is regarded
as a de facto approach for visual recognition tasks or a benchmark-
ing model if used to validate a novel model (Bilal and Hanif 2020;
Bristow and Lucey 2014). This fact partially supports the selection
of SVM (in lieu of many other classical models) in this effort.
A native SVM is a binary classifier that separates data into classes
by fitting an optimal hyperplane in the feature space separating all
training data points into two classes. As described by Foody and
Mathur (2006), the most optimal hyperplane is the one that max-
imizes the margin (i.e., from this hyperplane, the total distance to
the nearest data points of the two classes is maximized).
© ASCE 04020057-6
J. Comput. Civ. Eng., 2021, 35(1): 04020057
When the data do not have linear boundaries according to their
underlying class memberships, the finite feature-vector space can
be transformed into a functional space, which is termed the kernel
trick. The Gaussian kernel is widely used and was adopted in this
study, which is parameterized by a kernel parameter (γ). Another
critical parameter, often denoted C, is a regularization parameter
that controls the trade-off between achieving a low training error
and a low testing error or that controls the generalization capability
of the SVM classifier. Pertinent to a model selection problem, these
two hyperparameters can be determined through a cross-validation
process during the training. In the SVM literature, 10-fold cross-
validation often is used to determine the hyperparameters (Foody
and Mathur 2006). To accommodate multiclass classification based
on a binary SVM classifier, two well-known designs can be adopted:
(1) one-versus-all design; and (2) pairwise or one-versus-one design
(Duan et al. 2007). This study adopted the one-versus-one design.
MATLAB version R2018a and its machine-learning toolbox were
employed to implement the multiclass SVM–based learning, includ-
ing training and testing.
Data Collection and Preparation
With the mobile HSI system [Fig. 1(a)], a total of 68 instances of
hyperspectral images (and their companion gray-level images)
were captured in the field. Among these images, 43 images were
of concrete surfaces and 25 images were of asphalt surfaces. To
create scene complexity, artificial markings, oil, water, and green
and dry vegetation were added in 34 concrete images and 16 as-
phalt images that had hairline or apparent cracks. Of the remaining
18 images, 9 images each were were of concrete and asphalt pave-
ments without cracks. A proper calibration process was included
before the imaging; therefore, the obtained data all were reflectance
cubes. Due to the fixed exposure setting of the camera, the gray
images were captured physically as relatively dark pixels, and hence
had low visual contrast [Figs. 3(a and c)]. In Figs. 3(a and c), no
image enhancement (e.g., histogram equalization) is introduced;
nonetheless, the contrast-normalization step in a HOG descriptor
effectively can mitigate this issue before producing the gray value–
based feature vectors.
To create a supervised learning–ready data set, manual and se-
mantic labeling was carried out. Semantic labeling is the process of
labeling each pixel in an image with a corresponding class label.
This study used an image-segmentation (or image parsing) based
labeling approach in which clustered pixels belonging to the same
class were delineated in the image domain and rendered with a
select color. The labeling is based on the gray-level image that
accompanies a hyperspectral cube. In this work, this process was
conducted using an open-source image processing program, GIMP
version 2.10.12. During the labeling process, a total of six different
classes, including cracking, green vegetation, dry vegetation, water,
oil, and artificial marking, were assigned the colors black, green,
brown, blue, red, and yellow, respectively [Figs. 3(b and d)]. The
background materials (concrete and asphalt) were not classified in
these complex images or in the plain (concrete/asphalt) images.
To advance hyperspectral imaging–based machine learning and
damage detection research, the authors have shared this unique data
set publicly (Chen 2020).
Feature Calculation and Visualization
The colored mask images, denoted mðu; vÞ, share the same spatial
domain as the underlying gray image gðu; vÞ. For the sake of sim-
plicity, based on the color coding for the mask images, the value of
mð·Þ takes an integer value of 1, 2, : : : , 6 to indicate the underlying
J. Comput. Civ. Eng.
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

Fig. 3. (a) Gray image of concrete surface; (b) mask image for (a); (c) gray image of asphalt surface; and (d) mask image for (c). Colored mask images
in (b) and (d) appear in the online version.

six surface objects; in addition, the following notations are used to
describe the resulting data set:
D ¼ fhn ðx; y; sÞ; gn ðu; vÞ; mn ðu; vÞjn ¼ 1; 2; : : : ; 50g
To generate the machine-learning data for the models, the fol-
lowing process was developed:
1. Iterating with the location of (xi ; yj ), where i; j ¼ 1; 2; : : : ; 50,
the spectral profile is stored in the vector {hðxi ; yj ; sÞjs ∈
½1; 139}, and the gray values in the corresponding gray
image gðu; vÞ are confined in a neighborhood block of b ¼
fðu 0 ; v 0 Þju 0 ∈ ½ðxi − 1Þ × 20 þ 1; xi × 20; and v 0 ∈ ½ðyj − 1Þ ×
20 þ 1; yj × 20g. In this neighborhood of b, the gray-level im-
age patch and the corresponding mask patch corresponding to
the hyperspectral pixel at ðx; yÞ are denoted gðbÞ and mðbÞ,
respectively.
2. Given the mask patch mðbÞ, a simplified process is used to se-
lect the underlying class label for the hyperspectral pixel at
(xi ; yj ). By counting the number of pixels belong to different
object types within the neighborhood block b
a. if a dominant class label exists, i.e., the number of pixels that
belong to a class is greater than 50% of the total pixels in the
block (namely 200=400 pixels), this class label is assigned to
(xi ; yj ); and
b. if no dominant class label exists, this pixel (xi ; yj ) and the
corresponding neighborhood b is skipped.
3. At a pixel with a dominant class label, and per the feature ex-
traction method (HYP, HYP_PCA, and GL_HOG)
a. if HYP is used, fhðxi ; yj ; sÞjs ∈ ½1; 139g is directly used as
the feature vector with a dimension of 139 × 1;
b. If HYP_PCA is used, PCA is conducted over the vector
fhðxi ; yj ; sÞjs ∈ ½1; 139g, and the first 6 PC scores are used
to form a much lower-dimensional 6 × 1 feature vector;
c. if GL_HOG is used, the feature extraction is based on the
gray-level patch gðbÞ using the HOG-UoCTTI method, re-
sulting in a 31 × 1 feature vector; and
d. if HYP_PCA + GL_HOG is fused, the two corresponding
feature vectors simply are concatenated, resulting in a 37 × 1
feature vector.
4. By iterating this procedure over all the hyperspectral pixels for
all 50 instances of images, which include different types of
features in consideration, the following classification data set
is obtained for each of the feature extraction methods:
DFEA ¼ fðpk ; ck Þjk ¼ 1; 2; : : : ; Kg
where FEA represents one of the feature extraction processes,
HYP, HYP_PCA, GL_HOG, or HYP_PCA+GL_HOG. By
skipping many background pixels or pixels that do not have
ð2Þ dominant labels in Step 2b, the resulting number of meaningful
pixels (with dominant class labels) is 29,546, 8,132, 6,495,
6,273, and 5,312 features were obtained for the class labels (ck )
of water, oil, artificial marking, and green vegetation, respec-
tively. The number of features for cracks (concrete and asphalt
cracks) was 2,377. The dry vegetation features had the smallest
number of features, 957.
With the data cubes and gray images for the plain concrete
and asphalt surfaces (9 pairs each), 2,601 features arbitrarily were
obtained without using mask images individually for the concrete
and the asphalt labels. After adding these features into Eq. 3, the
number of labeled features used in this study [Eq. (3), K] was
34,748. As described previously, given the smallest (957) and the
largest (8,132) number of features, a moderate imbalance existed.
Based on the data in Eq. (3) and the original hyperspectral
pixels, three plots in Fig. 4 qualitatively illustrate the potential dis-
crimination power of hyperspectral pixels. Fig. 4(a) plots the origi-
nal spectral profiles of eight underlying objects (with only five
sample profiles for each object). Although the separability is evi-
dent, it is objectively hard to discern their potential capability of
prediction. Figs. 4(b and c) plot the first three primary PC scores
for obtained HYP_PCA features. These plotsshow the underlying
discrimination potential between all classes. HOG-based feature
vectors, which are in the form of histograms, also can be plotted;
however, it is challenging to discern their discrimination potential
visually; this is possible only through a classification model.
Experimental Tests and Performance Evaluation
Data Partition and Computational Cost
To proceed with the modeling and performance evaluation, a data
partition strategy is needed to split the data; one part is for the
training, and the other part is for model validation. In the literature,
the widely used scheme is to use 75% of the total data for training
and the remaining 25% for testing. Indeed, if there is a sufficient
amount of data, the data splitting ratio is flexible and is up to
the analyst. In this study, it was meaningful to examine whether
the amount of data size was sufficient, which can be determined
if the prediction performance increases with the amount of train-
ing data. Three data splitting schemes were considered for the
ð3Þ obtained feature data sets [Eq. (3)]. The first scheme, Test 1, after

© ASCE 04020057-7 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 12
Asphalt 5 Asphalt
Color Marking Concrete
Concrete 4 Crack
10 Crack Water

Second Principal Component

Dry Vegetation 3 Color Marking
Green Vegetation Dry Vegetation
8
Reflectance(103 )

Oil Green Vegetation

2
Water Oil

6 1

0
4
-1

2 -2
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

-3
0
0 20 40 60 80 100 120 140 -4 -2 0 2 4 6
(a) Spectral band (b) First Principal Component

Asphalt
2 Concrete
Crack
Water
1.5 Color Marking
Dry Vegetation
Third Principal Component

Green Vegetation
1 Oil

0.5

-0.5

-1

-5 -4 -3 -2 -1 0 1 2 3 4 5
(c) First Principal Component

Fig. 4. (a) Original spectral profiles as features for different classes; (b) PC scores as features from the first and second PCs; and (c) PC scores as
features from the first and third PCs.

randomly shuffling the data, considered 25% (8,147) of the data subsequently, is that M3 was the least accurate. In contrast, model
for training, and the remaining 75% (26,601) for testing purposes. M2 with the feature type HYP_PCA required the least training time
In Test 2, the total data set was divided equally (i.e., 17,374 (while achieving the best accuracy).
for training and 17,374 for testing). In Test 3, 75% (26,601) of the The average inference times of the resulting models were in-
data set was considered for training, and the remaining 25% spected, because the models potentially would be implemented
(8,147) for testing. With these three schemes, 12 different models as a predictive model. Fig. 5(b) reports the average inference times
were evaluated. of all models and test scenarios. As expected, the reported infer-
Training time and testing are important factors for assessing ence times did not vary much among the tests. The complexity did
the classification models as well as for the selection of the best not scale with the dimensionality of the feature vectors. Models
model for inference when implemented. The training times and the M2 (in which the feature HYP_PCA had a dimension of six)
inference times for prediction are presented in Fig. 5. In Fig. 5(a), and M4 (in which the joint feature had a dimension of 37) had
two trends are apparent. First, the model training scaled up with greater computational cost than the others (M1 and M3), although
the number of training data from Test 1 to Test 3. Second, the type M1’s feature vectors had a dimension of 139. As studied in the
of features significantly affects the training time; nonetheless, it literature, the computational complexity of a SVM model relies
does not scale with the dimensionality. As evident from Fig. 5(a), on the dimensionality of the input features and their realized struc-
M3 with the feature type GL_HOG required more training time tures of kernel functions, whichever dominates (Chapelle 2007).
than the other models in all tests, whereas the differences in train- Among the achieved models (M1–M4), it is arguable that the
ing of M1, M3, and M4 were less significant. The authors assert kernel structure determines the computational cost. Regardless,
that this was due to the underlying nonlinearity of the decision as elaborated subsequently, the model select for its accuracy was
boundaries defined in the feature space; hence the learning rate M2(HYP_PCA), which had an average inference time of 80 ms in
differed significantly. The corroborating evidence, as elaborated the MATLAB’s running environment.

© ASCE 04020057-8 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 (a) (b)

Fig. 5. Time cost for (a) training models; and (b) making an inference.
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

Performance Measures
The performance of the classification models was evaluated in
terms of quantitative measures. With a trained model iterating over
the testing data points, a confusion matrix can be generated; then,
several basic accuracy measures can be calculated, including the
overall accuracy, precision, and recall. However, it has been estab-
lished in the literature that the overall accuracy is not robust to im-
balanced data; in such a situation, misinterpretation may occur even
given a high accuracy measurement. Both precision and recall mea-
sures emphasize the outcomes of false predictions; therefore, in any
model, the precision competes against the recall, or vice versa. The
F1 score is defined as the harmonic mean of the precision and recall
values. Therefore, the F1 score tends to better gauge the classifi-
cation accuracy with an imbalanced data set, because only when
both precision and recall are high does the F1 score become high
(Goutte and Gaussier 2005).
To further evaluate the capacity and stability of the classifier, two
commonly used graphical analytics are found: receiver operating
characteristics (ROCs) curves and precision-recall (PR) curves. De-
tails of ROC and PR curves were given by Hossin and Sulaiman
(2015). In addition to the curves, two derived statistics as lumped
measures are used widely to indicate the classification capacity.
These are the area under the ROC curve (AU-ROC), and the area
under the precision-recall curve (AU-PR) for both the AU-ROC and
the AU-PR, a greater value (upper bound of 1) indicates a larger
capacity. One important consensus in the literature is that ROC
curves lack indicative power when the training data are imbalanced.
Nonetheless, because the AU-ROC is threshold-invariant, it is used
as the primary measure of classification capacity. The AU-PR also is
threshold-invariant; nonetheless, it may be statistically redundant if
the F1 score is used as well.
In summary, the performance evaluation in this paper used
the following performance measures: precision, recall, F1 score,
AU-ROC, and AU-PR. Among them, the precision and recall were
used to evaluate the consistency of the two measures, the F1
score was used as the primary accuracy measure, the AU-ROC
was treated as an important measure of model capacity, and the
AU-PR was treated as an auxiliary (possibly redundant) measure
to the F1 score and to the AU-ROC. Selectively, the ROC and
PR curves were adopted to observe and evaluate comprehensively
both the capacity and the stability of the underlying classification
models.
Test Results and Description
Test results from the four models (M1–M4) using three data split-
ting schemes (Tests 1–3) are reported herein. Tables 1–4 report
the performance results based on the testing data per the split-
ting schemes. Table 1 reports the results for M1(HYP), which
tested the hyperspectral feature vector type HYP. Tables 2–4
report results for models M2(HYP_PCA), M3(GL_HOG), and
M4(HYP_PCA+GL_HOG), respectively.
Some of the general trends in Tables 1–4 are summarized as
follows. In most of the testing, the prediction performance in-
creased as more data were used in training from Test 1 to Test 2
to Test 3 (Tables 1–4). In a few cases, the testing performance

Table 1. Performance measurements of M1(HYP)

Test Measure Concrete Asphalt Color marking Crack Dry vegetation Green vegetation Water Oil
Test 1 Precision 0.998 1.000 0.598 0.414 0.488 0.776 0.743 0.612
Recall 0.999 0.998 0.531 0.130 0.164 0.705 0.946 0.717
F1 0.999 0.999 0.563 0.198 0.246 0.739 0.832 0.661
AU-ROC 1.000 1.000 0.889 0.783 0.915 0.947 0.965 0.909
AU-PR 1.000 1.000 0.551 0.237 0.265 0.811 0.830 0.638
Test 2 Precision 1.000 1.000 0.611 0.567 0.558 0.818 0.787 0.628
Recall 0.999 1.000 0.655 0.167 0.192 0.704 0.957 0.702
F1 1.000 1.000 0.632 0.257 0.286 0.756 0.864 0.663
AU-ROC 1.000 1.000 0.906 0.808 0.921 0.952 0.971 0.912
AU-PR 0.999 0.999 0.592 0.312 0.290 0.800 0.845 0.642
Test 3 Precision 1.000 1.000 0.501 0.918 0.402 0.706 0.572 0.553
Recall 1.000 1.000 0.629 0.057 0.213 0.733 0.962 0.724
F1 1.000 1.000 0.558 0.107 0.278 0.719 0.717 0.627
AU-ROC 1.000 1.000 0.869 0.774 0.906 0.940 0.963 0.890
AU-PR 0.999 0.999 0.490 0.495 0.234 0.745 0.685 0.586

© ASCE 04020057-9 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 Table 2. Performance measurements of M2(HYP_PCA)
Test Measure Concrete Asphalt Color marking Crack Dry vegetation Green vegetation Water Oil
Test 1 Precision 0.930 0.901 0.957 0.914 0.878 0.986 0.973 0.955
Recall 0.886 0.910 0.964 0.921 0.664 0.979 0.992 0.974
F1 0.907 0.906 0.961 0.917 0.757 0.982 0.983 0.964
AU-ROC 0.993 0.995 0.997 0.995 0.985 0.999 0.999 0.998
AU-PRC 0.957 0.960 0.977 0.960 0.839 0.993 0.985 0.982
Test 2 Precision 0.970 0.936 0.973 0.973 0.900 0.990 0.994 0.979
Recall 0.959 0.945 0.989 0.952 0.808 0.994 0.995 0.981
F1 0.964 0.941 0.981 0.962 0.851 0.992 0.994 0.980
AU-ROC 0.998 0.998 0.999 0.997 0.993 1.000 1.000 0.998
AU-PRC 0.987 0.983 0.985 0.987 0.893 0.995 0.997 0.985
Test 3 Precision 0.987 0.942 0.984 0.971 0.908 0.989 0.988 0.992
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

Recall 0.955 0.954 0.991 0.950 0.867 0.997 0.996 0.995

F1 0.971 0.948 0.988 0.960 0.887 0.993 0.992 0.993
AU-ROC 0.998 0.997 0.999 0.997 0.994 1.000 1.000 1.000
AU-PRC 0.981 0.981 0.993 0.985 0.886 0.996 0.993 0.994

Table 3. Performance measurements of M3(GL_HOG)

Test Measure Concrete Asphalt Color marking Crack Dry vegetation Green vegetation Water Oil
Test 1 Precision 0.560 0.580 0.723 0.575 0.724 0.740 0.888 0.725
Recall 0.559 0.598 0.781 0.195 0.234 0.731 0.943 0.845
F1 0.560 0.589 0.751 0.292 0.354 0.736 0.915 0.780
AU-ROC 0.940 0.950 0.926 0.774 0.898 0.929 0.985 0.944
AU-PR 0.549 0.615 0.724 0.295 0.365 0.750 0.932 0.764
Test 2 Precision 0.638 0.690 0.752 0.676 0.827 0.786 0.929 0.785
Recall 0.669 0.637 0.827 0.273 0.290 0.801 0.964 0.903
F1 0.653 0.662 0.788 0.389 0.430 0.793 0.946 0.840
AU-ROC 0.963 0.966 0.948 0.833 0.931 0.958 0.991 0.962
AU-PR 0.647 0.701 0.756 0.399 0.466 0.796 0.954 0.831
Test 3 Precision 0.665 0.690 0.772 0.644 0.886 0.801 0.882 0.802
Recall 0.639 0.679 0.847 0.301 0.325 0.809 0.964 0.906
F1 0.652 0.684 0.808 0.410 0.476 0.805 0.921 0.851
AU-ROC 0.959 0.969 0.942 0.828 0.948 0.962 0.990 0.964
AU-PR 0.666 0.737 0.739 0.424 0.529 0.816 0.921 0.850

Table 4. Performance measurements of M4(HYP_PCA+GL_HOG)

Test Measure Concrete Asphalt Color marking Crack Dry vegetation Green vegetation Water Oil
Test 1 Precision 0.520 0.842 0.904 0.785 0.489 0.905 0.956 0.890
Recall 0.965 0.881 0.921 0.848 0.341 0.927 0.679 0.873
F1 0.676 0.861 0.913 0.815 0.402 0.916 0.794 0.881
AU-ROC 0.978 0.992 0.993 0.985 0.938 0.993 0.974 0.986
AU-PR 0.695 0.933 0.963 0.898 0.391 0.964 0.930 0.930
Test 2 Precision 0.450 0.829 0.922 0.841 0.670 0.875 0.930 0.903
Recall 0.963 0.888 0.931 0.876 0.436 0.910 0.567 0.902
F1 0.613 0.857 0.927 0.858 0.528 0.892 0.704 0.902
AU-ROC 0.969 0.993 0.994 0.990 0.957 0.990 0.961 0.988
AU-PR 0.592 0.928 0.966 0.931 0.583 0.947 0.888 0.938
Test 3 Precision 0.779 0.761 0.931 0.895 0.779 0.947 0.969 0.925
Recall 0.939 0.962 0.935 0.835 0.529 0.951 0.851 0.894
F1 0.852 0.849 0.933 0.864 0.630 0.949 0.906 0.909
AU-ROC 0.995 0.995 0.994 0.989 0.962 0.998 0.997 0.989
AU-PR 0.946 0.957 0.970 0.924 0.683 0.985 0.981 0.944

decreased in Test 3 (which used 25% of the data points for testing); positive effect on improving the accuracy of classification models.
it is speculated that random shuffling occasionally may bring in The prediction of cracks and dry vegetation was the most challeng-
outlier data that are not learned from the training data. This obser- ing task. This was expected, because cracks are treated as an object
vation indicates that the size of the training data does have a type that comes from two structural scenes (concrete and asphalt

© ASCE 04020057-10 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 pavements), the underlying materials are very diversified, and over-
all all the reflectance intensities are darker than other objects. The
dry vegetation type had the least number of instances in the data set,
which partially contributed to the relatively poor performance com-
pared with the prediction for the green vegetation. In addition, in all
tests of all models, the precision and recall measurements were rel-
atively close for all class labels except cracks and dry vegetation.
This was particularly significant for M1(HYP) and M3(GL_HOG),
for which the recall values tended to be much smaller than the pre-
cision values. This means that in the cases of these two objects, each
instance of prediction tended to be correct; however, the models
tended to miss many objects that were either cracks or dry vegetation
objects. Furthermore, this result implies that a trade-off measure,
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
The AU-ROC measure was adopted as a model-capacity measure
because it was invariant to classification thresholds and nonredun-
dant to the F1 score.
Performance Evaluation
The F1 score and AU-ROC were used as two primary performance
measures in this study. To meet the objectives outlined previously,
Figs. 7(a–d) illustrate the performance of all four models. Fig. 7(a)
shows the F1 scores for Test 1 data; Fig. 7(b) shows the F1 scores
for Test 3 data; Fig. 7(c) shows the AU-ROC for Test 1 data; and
Fig. 7(d) shows the AU-ROC for Test 3 data. The results from Test
2 are not included herein because the observed trends from Test 2

such as the F1 score, should be used as a better accuracy measure.
To determine the coherency and the distinction of these differ-
ent measurements, scatter plots of all measures are illustrated in
Figs. 6(a–d) for models M1–M4, respectively; for the sake of brev-
ity, only the Test 2 data scheme is included. Figs. 6(a–d) plot the F1
scores against themselves (as indicated by a 1:1 line), and all other
measurements are plotted against F1. The AU-PR as a model
capacity measure was statistically consistent with the F1 score, be-
cause it followed a linear trend in each model with a high R2 value
(R2 > 0.87). This implies that AU-PR largely was redundant as a
capacity measure. Therefore, the F1 score can replace it as both
a prediction accuracy measure and a model capacity measure.
results are between Test 1 and Test 2. The following evaluation
focuses on the four objectives of this study described previously.
The F1 scores of Model M1(HYP) [Figs. 7(a and b)] indicate
that the model successfully identifies (F1 > 0.7) the plain concrete,
asphalt, green vegetation, and water. For the ed marking and oil, the
F1 score was about 0.6. However, for cracks and dry vegetation, the
F1 measurements were less than 0.3, indicating little prediction ac-
curacy. First, this reflects the challenge in recognizing cracks, pri-
marily caused by the inherent spectral complexity. Second, it is
primarily due to the smaller number of dry-vegetation data points.
For both objects, this presumptively is attributed to the high dimen-
sionality of the feature vectors of HYP. Nonetheless, as expected

1
0.9

0.95 R2 = 0.9181
0.7
Measure value

Measure value

R2 = 0.9882 0.9
0.5

0.85
0.3

0.1 0.8
0.1 0.3 0.5 0.7 0.9 0.8 0.85 0.9 0.95 1
(a) F1 (b) F1

Preccision Recall F1 Preccision Recall F1

AU-ROC AU-PR Linear (F1) AU-ROC AU-PR Linear (F1)
Linear (AU-PR) Linear (AU-PR)

1 1

0.9
0.9
0.8
0.8
Measure value

Measure value

0.7
R2 = 0.8724
0.6 R2 = 0.9877 0.7

0.5
0.6
0.4
0.5
0.3
0.2 0.4
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.4 0.5 0.6 0.7 0.8 0.9 1
(c) F1 (d) F1
Preccision Recall F1 Preccision Recall F1
AU-ROC AU-PR Linear (F1) AU-ROC AU-PR Linear (F1)
Linear (AU-PR) Linear (AU-PR)

Fig. 6. Performance measurement scatter plots using Test 2 data: (a) M1; (b) M2; (c) M3; and (d) M4.

© ASCE 04020057-11 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
(a)
(c)
Fig. 7. Model performance: (a) F1 score with Test 1 data; (b) F1 score with Test 3 data; (c) AU-ROC with Test 1 data; and (d) AU-ROC with
Test 3 data.
from the AU-ROC measurements [Figs. 7(c and d)], there is no
doubt that Model M1(HYP) is highly effective in recognizing these
structural surface objects. For concrete, the average AU-ROC was
about 1. Therefore, hyperspectral pixels as feature vectors are ef-
fective in recognizing most of the structural surface objects, and
have relatively less accuracy only in the detection of cracks and
dry vegetation.
Comparing the accuracy of Model M2(HYP_PCA) with that of
M1(HYP) [Figs. 7(a and b)] showed that the detection accuracy sig-
nificantly increased. For the two weak prediction instances of cracks
and dry vegetation in M1(HYP), the F1 score increased from 0.198
to 0.917 in Test 1 and from 0.107 to 0.96 in Test 3 with Model M2
(HYP_PCA). For dry vegetation, the F1 scores changed from 0.246
to 0.757 in Test 1, and from 0.278 to 0.887 in Test 3. For all other
class labels, the classification accuracy still increased from their ini-
tially high F1 values from M1 to M2. When inspecting Fig. 7(c), the
AU-ROC measurements from M1 and M2 increased. Particularly,
the M2’s AU-ROC measurements were greater than 0.99 at predict-
ing all classes. This again signifies that Model M2(HYP_PCA) had
nearly perfect capacity to detect all structural surface objects. The
comparative tests herein provide the direct evidence that performing
dimensionality reduction over hyperspectral profiles (i.e., as HYP
feature vectors) substantially can unleash the embedded discrimina-
tion capacity of the data that otherwise is not exploitable.
A secondary yet important research problem in this work was
to prove the competitiveness of low spatial-resolution hyperspec-
tral data compared with high-resolution gray-intensity images in
detecting material surface objects. For this purpose, the results
© ASCE 04020057-12
J. Comput. Civ. Eng., 2021, 35(1): 04020057
(b)
(d)
of M2(HYP_PCA) and M3(GL_HOG) were evaluated and com-
pared. The F1 scores and the AU-ROC measurements shown in
Figs. 7(a–d), respectively, clearly indicate that in the prediction of
all class labels, M2(HYP_PCA) superseded M3(GL_HOG). Even
considering the fourth model, M4, to be evaluated, M2(HYP_PCA)
was the best model in Test 3. This model provided outstanding
performance; the smallest F1 score was 0.757, and the smallest
AU-ROC measurement was 0.985, both for predicting dry vegeta-
tion; for crack prediction, F1 ¼ 0.917 and AU-ROC ¼ 0.995. The
control model in this test, M3(GL_HOG), had F1 ¼ 0.354 and
AU-ROC ¼ 0.898 for dry vegetation, and AU-ROC ¼ 0.774, and
F1 ¼ 0.292 for cracks.
To further visualize the prediction capacity and the stability of
the two most important models in this work, M2(HYP_PCA) and
M3(GL_HOG), the ROC curves are illustrated in Figs. 8(a and b)
for the two models, respectively, based on Test 3 data. Similarly,
Figs. 9(a and b) report the PR curves. The two ROC and PR curves
show the different prediction capacity for each class label as the
underlying classification threshold changed. The ROC and PR
curves show that Model M2(HYP_PCA) had not only superb clas-
sification capability but also stronger stability, the latter of which is
indicated by the smoothness of the curves as the underlying thresh-
old varies. In the case of M3(GL_HOG), the classification capabil-
ity (and accuracy) overall was much moderate; the stability also
was worse.
For Objective 4, the authors evaluated a simple data fusion
technique, which joined the two feature types, HYP_PCA and
GL_HOG, and observed the possible performance gain or loss.
J. Comput. Civ. Eng.
 1 1

0.9 0.9

0.8 1 0.8

0.7 0.98 0.7

True positive rate

True positive rate

0.6 0.96 0.6
0.94
0.5 0.5
0.92
0.4 Concrete 0.4 Concrete
0.9 Asphalt Asphalt
0.3 Color Marking 0.3 Color Marking
0 0.005 0.01 0.015 0.02 0.025 0.03 Crack Crack
0.2 Green Vegetation 0.2 Green Vegetation
Water Water
0.1 0.1
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.

Oil Oil
Dry Vegetation Dry Vegetation
0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
(a) False positive rate (b) False positive rate

Fig. 8. ROC curves for (a) M2 (HYP_PCA); and (b) M3 (GL_HOG) using Test 3 data.

1 1
Concrete
Asphalt 0.9
Color Marking
0.8 Crack 0.8
Green Vegetation
0.7
Water
0.6 Oil 1 0.6
Precision

Precision

Dry Vegetation
0.95 0.5

0.4 0.4 Concrete

0.9 Asphalt
0.3 Color Marking
0.85 Crack
0.2 0.2 Green Vegetation
0.8 Water
0.1 Oil
0.9 0.95 1
Dry Vegetation
0 0
0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
(a) Recall (b) Recall

Fig. 9. Precision-recall curves for (a) M2 (HYP_PCA); and (b) M3 (GL_HOG) using Test 3 data.

The comparative evaluation was conducted between M4(HYP_ features corresponding to the hyperspectral pixel, even though the
PCA+GL_HOG) and M2, and between M4 and M3. In terms of gray images were captured at a much higher resolution (20 times
both F1 and AU-ROC measurements, the performance of M4 was higher). Although improved gray value–based feature extraction
worse than that of M2. This implies that the gray value–based techniques can be used, hyperspectral pixels (which are captured
features (GL_HOG) compromised the classification potential of in the visible to near-infrared spectral bands with high dimension-
HYP_PCA. When M4 and M3 were assessed, for the object types ality) have undoubtful promise in the detection of complex struc-
of concrete, asphalt, crack, and green vegetation, performance tural surface objects.
increased; however, for other objects (color marking, green vegeta- Furthermore, the methodology in this study was a traditional
tion, water, and oil), the performance decreased. machine learning paradigm. As reasoned previously, this method-
ology was proper considering the hypothesis-testing nature of this
effort. In addition, the hyperspectral data set presented in this paper
Discussion was unique, and the models developed herein form a baseline for
future studies (e.g., semantic object segmentation). Moreover, the
The observed superior performance of M2(HYP_PCA) compared selected Model M2(HYP_PCA) developed herein, which is much
with that of M3(GL_HOG) implies that a single hyperspectral pixel less complex than the latest DL models yet has very high accuracy
is much more effective than a 20 × 20 neighborhood of gray values at pixel-level prediction, can be implemented readily in an edge-
in discriminating the complex structural surface objects. Hyper- computing device (e.g., the onboard computer that was natively
spectral pixels with reflectance features at both visible and infrared designed for the Cubert camera in our imaging system). The re-
bands, once preprocessed (e.g., a PCA-based feature selection sulting UAV-based imaging system, therefore, can realize high-
step), outperformed the gray values or the embedded texture/shape throughput hyperspectral imaging and onboard processing, in which

© ASCE 04020057-13 J. Comput. Civ. Eng.

J. Comput. Civ. Eng., 2021, 35(1): 04020057

 the ML model developed in this study can serve as a rapid filtering
tool so that any images with cracks (or other anomalies) can be
detected quickly. Given the resulting data filtered by the model pro-
posed in this paper, a DL model, i.e., a semantic spatial-spectral
segmentation model, can be deployed to realize more detailed
and semantics-rich damage detection.
One specific future direction is to extract spectral–spatial fea-
tures using a more integral approach. As mentioned previously, a
pansharpening technique can be used to generate a high spatial-
resolution hyperspectral cube. Specifically, the hyperspectral cube
of 20 × 20 × 139 used in this study can be turned into a cube
with much higher spatial dimension, namely a 1,000 × 1,000 ×
139 cube. With this 3D cube, and a mask image of the same size,
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
advanced spatial-spectral features may be extracted (Fang et al.
2018; Hang et al. 2015; Zhang et al. 2014), and then a classifier
(e.g., a kernel SVM as used in this study) can be adopted. This
pansharpening may act as an effective data augmentation tech-
nique, such that it can support a deep learning architecture. This
promising direction is beyond the scope of this work.
Conclusion
A mobile hyperspectral imaging system was developed for ground-
based and aerial data collection. One of its primary applications
is to inspect surfaces of structural materials, such as concrete and
asphalt materials, which commonly are used in transportation struc-
tures. The innovation lies in its ability to detect surface damage
and other artifacts at the material levels due to its high-dimensional
pixels with reflectance in both visible and near-infrared bands. The
effectiveness of the technique was proven in composition with
regular gray-level images that are very high-resolution and com-
monly used in practice. Four machine-learning models character-
ized by different feature description processes were trained and
tested. With a total of 34,748 labeled features of different types,
three data-splitting schemes were used to evaluate the effects of
data size. A multiclass support vector machine with a Gaussian
kernel was adopted in all models. State-of-the-art measures were
adopted to test the models, and the issue of data imbalance was
considered. From a comprehensive evaluation, two major conclu-
sions were formulated:
1. The use of hyperspectral reflectance values at pixels as feature
vectors was effective in recognizing all eight surface objects.
However, dimensionality reduction is essential for this effective-
ness. The linear PCA approach was adopted; using only the first
six component scores, the resulting classification models were
highly accurate, high-capacity, and stable.
2. Compared with the model based on gray-level features (GL_
HOG), based on a popular variant of the HOG descriptor, the
PCA-adapted hyperspectral features had much better detection
performance. A single hyperspectral pixel with high-dimension
spectral reflectance evidentially is competitive compared with
the corresponding high-resolution gray intensities that express
the shape and texture of the underlying objects. The data fusion
technique in this study had a less desirable performance, which
may be due to the simple concatenation technique used to
combine the GL_HOG and HYP_PCA features.
With the experimental and machine learning–based evaluation
results in this paper, the authors further envision the dawn of com-
putational hyperspectral imaging or hyperspectral machine vision
for material damage detection in civil engineering and their promise
in dealing with complex scenes that are often found in civil struc-
tures in service.
© ASCE 04020057-14
J. Comput. Civ. Eng., 2021, 35(1): 04020057
Data Availability Statement
Some or all data, models, or code generated or used during the
study are available from the corresponding author by request.
The training and testing data set is archived publicly (Chen 2020).
Acknowledgments
This material is based partially upon work supported by the National
Science Foundation (NSF) under Award number No. IIA-1355406
and work supported by the United States Department of Agricul-
ture’s National Institute of Food and Agriculture (USDA-NIFA)
under Award No. 2015-68007-23214. Any opinions, findings, and
conclusions or recommendations expressed in this material are those
of the author(s) and do not necessarily reflect the views of NSF or
USDA-NIFA.
References
AASHTO. 2019. Manual for bridge element inspection. Washington, DC:
AASHTO.
Adler-Golden, S., A. Berk, L. Bernstein, S. Richtsmeier, P. Acharya, M.
Matthew, G. Anderson, C. Allred, L. Jeong, and J. Chetwynd. 1998.
“FLAASH, a MODTRAN4 atmospheric correction package for hyper-
spectral data retrievals and simulations.” In Proc., 7th JPL Airborne
Earth Science Workshop, 9–14. Pasadena, CA: California Institute
of Technology.
Alipour, M., D. K. Harris, and G. R. Miller. 2019. “Robust pixel-level crack
detection using deep fully convolutional neural networks.” J. Comput.
Civ. Eng. 33 (6): 04019040. https://doi.org/10.1061/(ASCE)CP.1943
-5487.0000854.
Bandos, T. V., L. Bruzzone, and G. Camps-Valls. 2009. “Classification of
hyperspectral images with regularized linear discriminant analysis.”
IEEE Trans. Geosci. Remote Sens. 47 (3): 862–873. https://doi.org/10
.1109/TGRS.2008.2005729.
Baumgardner, M. F., L. L. Biehl, and D. A. Landgrebe. 2015. 220 band
AVIRIS hyperspectral image data set: June 12, 1992 Indian Pine Test
Site 3. West Lafayette, IN: Purdue Univ. Research Repository. https://
doi.org/10.4231/R7RX991C.
Bilal, M., and M. S. Hanif. 2020. “Benchmark revision for HOG-SVM
pedestrian detector through reinvigorated training and evaluation meth-
odologies.” IEEE Trans. Intell. Transp. Syst. 21 (3): 1277–1287. https://
doi.org/10.1109/TITS.2019.2906132.
Bioucas-Dias, J. M., A. Plaza, N. Dobigeon, M. Parente, Q. Du, P. Gader,
and J. Chanussot. 2012. “Hyperspectral unmixing overview: Geomet-
rical, statistical, and sparse regression-based approaches.” IEEE J. Sel.
Top. Appl. Earth Obs. Remote Sens. 5 (2): 354–379. https://doi.org/10
.1109/JSTARS.2012.2194696.
Bodkin, A., A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer.
2009. “Snapshot hyperspectral imaging: The hyperpixel array camera.”
In Proc., Algorithms and Technologies for Multispectral, Hyperspec-
tral, and Ultraspectral Imagery XV, 73340H. Bellingham, WA:
International Society for Optics and Photonics.
Bristow, H., and S. Lucey. 2014. Why do linear SVMs trained on HOG
features perform so well? Ithaca, NY: Cornell Univ.
Cawley, P. 2018. “Structural health monitoring: Closing the gap between
research and industrial deployment.” Struct. Health Monit. 17 (5):
1225–1244. https://doi.org/10.1177/1475921717750047.
Cha, Y.-J., W. Choi, and O. Büyüköztürk. 2017. “Deep learning-based crack
damage detection using convolutional neural networks.” Comput.-Aided
Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice
.12263.
Chapelle, O. 2007. “Training a support vector machine in the primal.”
Neural Comput. 19 (5): 1155–1178. https://doi.org/10.1162/neco.2007
.19.5.1155.
J. Comput. Civ. Eng.
 Chen, G., and S.-E. Qian. 2007. “Dimensionality reduction of hyperspectral
imagery.” J. Appl. Remote Sens. 1 (1): 013509. https://doi.org/10.1117
/1.2723663.
Chen, G., and S.-E. Qian. 2008. “Simultaneous dimensionality reduction
and denoising of hyperspectral imagery using bivariate wavelet shrink-
ing and principal component analysis.” Can. J. Remote Sens. 34 (5):
447–454. https://doi.org/10.5589/m08-058.
Chen, Y., H. Jiang, C. Li, X. Jia, and P. Ghamisi. 2016. “Deep feature ex-
traction and classification of hyperspectral images based on convolu-
tional neural networks.” IEEE Trans. Geosci. Remote Sens. 54 (10):
6232–6251. https://doi.org/10.1109/TGRS.2016.2584107.
Chen, Y., Z. Lin, X. Zhao, G. Wang, and Y. Gu. 2014. “Deep learning-
based classification of hyperspectral data.” IEEE J. Sel. Top. Appl.
Earth Obs. Remote Sens. 7 (6): 2094–2107. https://doi.org/10.1109
/JSTARS.2014.2329330.
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
Chen, Z. 2020. “Hyperspectral imagery for material surface damage.” Fig-
share. https://doi.org/10.6084/m9.figshare.12385994.v1.
Chen, Z., J. Chen, F. Shen, and Y. Lee. 2015. “Collaborative mobile-cloud
computing for civil infrastructure condition inspection.” J. Comput. Civ.
Eng. 29 (5): 04014066. https://doi.org/10.1061/(ASCE)CP.1943-5487
.0000377.
Chen, Z., R. Derakhshani, C. Halmen, and J. T. Kevern. 2011. “A texture-
based method for classifying cracked concrete surfaces from digital im-
ages using neural networks.” In Proc., 2011 Int. Joint Conf. on Neural
Networks, 2632–2637. New York: IEEE.
Chen, Z., and T. C. Hutchinson. 2010. “Image-based framework for
concrete surface crack monitoring and quantification.” Adv. Civ. Eng.
2010: 215295. https://doi.org/10.1155/2010/215295.
Cheng, H. D., and M. Miyojim. 1998. “Automatic pavement distress
detection system.” Inf. Sci. 108 (1): 219–240. https://doi.org/10.1016
/S0020-0255(97)10062-7.
Cooley, T., et al. 2002.“FLAASH, a MODTRAN4-based atmospheric
correction algorithm, its application and validation.” In Proc., IEEE
Int. Geoscience and Remote Sensing Symp., 1414–1418. New York:
IEEE.
Cubert. 2018. “Hyperspectral VNIR Camera FireflEYE 185.” Accessed
January 1, 2020. https://cubert-gmbh.com/product/firefleye-185
-hyperspectral/.
Dalal, N., and B. Triggs. 2005.“Histograms of oriented gradients for human
detection.” In Vol. 881 of Proc., IEEE Computer Society Conf. on
Computer Vision and Pattern Recognition (CVPR’05), 886–893.
New York: IEEE.
Duan, K.-B., J. C. Rajapakse, and M. N. Nguyen. 2007. “One-versus-one
and one-versus-all multiclass SVM-RFE for gene selection in cancer
classification.” In Proc., Evolutionary Computation, Machine Learning
and Data Mining in Bioinformatics, 47–56. Berlin: Springer.
Fang, L., N. He, S. Li, A. J. Plaza, and J. Plaza. 2018. “A new spatial–
spectral feature extraction method for hyperspectral images using local
covariance matrix representation.” IEEE Trans. Geosci. Remote Sens.
56 (6): 3534–3546. https://doi.org/10.1109/TGRS.2018.2801387.
Felzenszwalb, P. F., R. B. Girshick, D. McAllester, and D. Ramanan. 2010.
“Object detection with discriminatively trained part-based models.”
IEEE Trans. Pattern Anal. Mach. Intell. 32 (9): 1627–1645. https://doi
.org/10.1109/TPAMI.2009.167.
Foody, G. M., and A. Mathur. 2006. “The use of small training sets con-
taining mixed pixels for accurate hard image classification: Training on
mixed spectral responses for classification by a SVM.” Remote Sens.
Environ. 103 (2): 179–189. https://doi.org/10.1016/j.rse.2006.04.001.
Gavilán, M., D. Balcones, O. Marcos, D. F. Llorca, M. A. Sotelo, I. Parra,
M. Ocaña, P. Aliseda, P. Yarza, and A. Amírola. 2011. “Adaptive road
crack detection system by pavement classification.” Sensors 11 (10):
9628–9657. https://doi.org/10.3390/s111009628.
Goel, P. K., S. O. Prasher, R. M. Patel, J. A. Landry, R. B. Bonnell, and
A. A. Viau. 2003. “Classification of hyperspectral data by decision trees
and artificial neural networks to identify weed stress and nitrogen status
of corn.” Comput. Electron. Agric. 39 (2): 67–93. https://doi.org/10
.1016/S0168-1699(03)00020-6.
Goutte, C., and E. Gaussier. 2005. “A probabilistic interpretation of
precision, recall and F-score, with implication for evaluation.” In Proc.,
© ASCE 04020057-15
J. Comput. Civ. Eng., 2021, 35(1): 04020057
European Conf. on Information Retrieval, 345–359. New York:
Springer.
Graña, M., M. Veganzons, and B. Ayerdi. 2020. “Hyperspectral remote
sensing scenes.” Accessed May 1, 2020. http://www.ehu.eus/ccwintco
/index.php/Hyperspectral_Remote_Sensing_Scenes.
Green, R. O., et al. 1998. “Imaging spectroscopy and the airborne visible/
infrared imaging spectrometer (AVIRIS).” Remote Sens. Environ.
65 (3): 227–248. https://doi.org/10.1016/S0034-4257(98)00064-9.
Hang, R., Q. Liu, H. Song, and Y. Sun. 2015. “Matrix-based discriminant
subspace ensemble for hyperspectral image spatial–spectral feature fu-
sion.” IEEE Trans. Geosci. Remote Sens. 54 (2): 783–794. https://doi
.org/10.1109/TGRS.2015.2465899.
Hang, R., Q. Liu, Y. Sun, X. Yuan, H. Pei, J. Plaza, and A. Plaza. 2017.
“Robust matrix discriminative analysis for feature extraction from hy-
perspectral images.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
10 (5): 2002–2011. https://doi.org/10.1109/JSTARS.2017.2658948.
Harsanyi, J. C., and C. Chang. 1994. “Hyperspectral image classification
and dimensionality reduction: an orthogonal subspace projection ap-
proach.” IEEE Trans. Geosci. Remote Sens. 32 (4): 779–785. https://doi
.org/10.1109/36.298007.
Heylen, R., M. Parente, and P. Gader. 2014. “A review of nonlinear
hyperspectral unmixing methods.” IEEE J. Sel. Top. Appl. Earth
Obs. Remote Sens. 7 (6): 1844–1868. https://doi.org/10.1109/JSTARS
.2014.2320576.
Ho, H.-N., K.-D. Kim, Y.-S. Park, and J.-J. Lee. 2013. “An efficient image-
based damage detection for cable surface in cable-stayed bridges.” NDT
E Int. 58 (Sep): 18–23. https://doi.org/10.1016/j.ndteint.2013.04.006.
Hossin, M., and M. N. Sulaiman. 2015. “A review on evaluation metrics
for data classification evaluations.” Int. J. Data Min. Knowl. Manage.
Process 5 (2): 1. https://doi.org/10.5121/ijdkp.2015.5201.
Hu, W., and C. Jia. 2015. “A bootstrapping approach to entity linkage on
the semantic web.” J. Web Semant. 34 (Oct): 1–12. https://doi.org/10
.1016/j.websem.2015.07.003.
Isawa, K., S. Nakayama, M. Ikeda, S. Takagi, S. Tosaka, and N. Kasai.
2005. “Robotic 3D SQUID imaging system for practical nondestructive
evaluation applications.” Physica C 432 (3–4): 182–192. https://doi.org
/10.1016/j.physc.2005.08.008.
Ji, R., Y. Gao, R. Hong, Q. Liu, D. Tao, and X. Li. 2014. “Spectral-spatial
constraint hyperspectral image classification.” IEEE Trans. Geosci.
Remote Sens. 52 (3): 1811–1824. https://doi.org/10.1109/TGRS.2013
.2255297.
Johnson, W. R., D. W. Wilson, G. H. Bearman, and J. Backlund. 2004. “An
all-reflective computed tomography imaging spectrometer.” In Proc.,
Instruments, Science, and Methods for Geospace and Planetary Remote
Sensing, 88–97. Bellingham, WA: International Society for Optics and
Photonics.
Kaiser, P. K., and R. M. Boynton. 1996. Human color vision. Washington,
DC: Optical Society of America.
Kaseko, M. S., and S. G. Ritchie. 1993. “A neural network-based method-
ology for pavement crack detection and classification.” Transp. Res.
Part C: Emerging Technol. 1 (4): 275–291. https://doi.org/10.1016
/0968-090X(93)90002-W.
Kim, H., S. Sim, and S. Cho. 2015. “Unmanned aerial vehicle (UAV)-
powered concrete crack detection based on digital image processing.”
In Proc., Int. Conf. on Advances in Experimental Structural Engineer-
ing. Urbana-Champaign, IL: Univ. of Illinois.
Lattanzi, D. A., and G. Miller. 2013. “A prototype imaging and visualiza-
tion system for robotic infrastructure inspection.” In Proc., Structures
Congress 2013: Bridging Your Passion with Your Profession, 410–421.
Reston, VA: ASCE.
Leshko, B. J. 2002. Guidelines for inspecting complex components of
bridges. Washington, DC: AASHTO.
Li, W., G. Wu, F. Zhang, and Q. Du. 2017. “Hyperspectral image classi-
fication using deep pixel-pair features.” IEEE Trans. Geosci. Remote
Sens. 55 (2): 844–853. https://doi.org/10.1109/TGRS.2016.2616355.
Liu, Z., S. A. Suandi, T. Ohashi, and T. Ejima. 2002.“Tunnel crack detec-
tion and classification system based on image processing.” In Proc.,
Machine Vision Applications in Industrial Inspection X, 145–152. Bel-
lingham, WA: International Society for Optics and Photonics.
J. Comput. Civ. Eng.
 Loncan, L., L. B. De Almeida, J. M. Bioucas-Dias, X. Briottet,
J. Chanussot, N. Dobigeon, S. Fabre, W. Liao, G. A. Licciardi, and
M. Simoes. 2015. “Hyperspectral pansharpening: A review.” IEEE
Geosci. Remote Sens. Mag. 3 (3): 27–46. https://doi.org/10.1109/MGRS
.2015.2440094.
Lu, G., and B. Fei. 2014. “Medical hyperspectral imaging: A review.”
J. Biomed. Opt. 19 (1): 010901. https://doi.org/10.1117/1.JBO.19.1
.010901.
Marpu, P. R., P. Gamba, and I. Niemeyer. 2009. “Hyperspectral data
classification using an ensemble of class-dependent neural networks.”
In Proc., 2009 1st Workshop on Hyperspectral Image and Signal
Processing: Evolution in Remote Sensing, 1–4. New York: IEEE.
Mercier, G., and M. Lennon. 2003. “Support vector machines for hyper-
spectral image classification with spectral-based kernels.” In Vol. 281 of
Proc., IGARSS 2003. 2003 IEEE Int. Geoscience and Remote Sensing
Downloaded from ascelibrary.org by University of Liverpool on 11/14/20. Copyright ASCE. For personal use only; all rights reserved.
Symp. (IEEE Cat. No.03CH37477), 288–290. New York: IEEE.
Morris, T. 2004. Computer vision and image processing. Basingstoke, UK:
Palgrave Macmillan.
Ni, F., J. Zhang, and Z. Chen. 2019. “Pixel-level crack delineation in
images with convolutional feature fusion.” Struct. Control Health
Monit. 26 (1): e2286. https://doi.org/10.1002/stc.2286.
Olsen, M. J., A. Barbosa, P. Burns, A. Kashani, H. Wang, M. Veletzos,
Z. Chen, G. Roe, and K. Tabrizi. 2016. Assessing, coding, and marking
of highway structures in emergency situations, Volume 1: Research
overview. Washington, DC: Transportation Research Board.
Ozer, E., M. Q. Feng, and D. Feng. 2015. “Citizen sensors for SHM:
Towards a crowdsourcing platform.” Sensors 15 (6): 14591–14614.
https://doi.org/10.3390/s150614591.
Palsson, F., J. R. Sveinsson, M. O. Ulfarsson, and J. A. Benediktsson. 2015.
“Model-based fusion of multi- and hyperspectral images using PCA and
wavelets.” IEEE Trans. Geosci. Remote Sens. 53 (5): 2652–2663.
https://doi.org/10.1109/TGRS.2014.2363477.
Plaza, A., et al. 2009. “Recent advances in techniques for hyperspectral
image processing.” Remote Sens. Environ. Supplement, 113 (S1):
S110–S122. https://doi.org/10.1016/j.rse.2007.07.028.
Prasanna, P., K. J. Dana, N. Gucunski, B. B. Basily, H. M. La, R. S. Lim,
and H. Parvardeh. 2014. “Automated crack detection on concrete
bridges.” IEEE Trans. Autom. Sci. Eng. 13 (2): 591–599. https://doi
.org/10.1109/TASE.2014.2354314.
Ritchie, S. G. 1987. “Expert systems in pavement management.” Transp.
Res. Part A: General 21 (2): 145–152. https://doi.org/10.1016/0191
-2607(87)90007-0.
© ASCE 04020057-16
J. Comput. Civ. Eng., 2021, 35(1): 04020057
Ritchie, S. G. 1990. “Digital imaging concepts and applications in pave-
ment management.” J. Transp. Eng. 116 (3): 287–298. https://doi.org
/10.1061/(ASCE)0733-947X(1990)116:3(287).
Roh, Y., G. Heo, and S. E. Whang. 2019. “A survey on data collection for
machine learning: A big data—AI integration perspective.” In Proc.,
IEEE Transactions on Knowledge and Data Engineering. New York:
IEEE.
Siesler, H. W., Y. Ozaki, S. Kawata, and H. M. Heise. 2008. Near-infrared
spectroscopy: Principles, instruments, applications. New York: Wiley.
Sohn, H., C. R. Farrar, F. M. Hemez, D. D. Shunk, D. W. Stinemates, B. R.
Nadler, and J. J. Czarnecki. 2003. A review of structural health mon-
itoring literature: 1996–2001. Los Alamos, NM: Los Alamos National
Laboratory.
Tang S., and Z. Chen. 2017. “Detection of complex concrete damage—A
deep learning framework and performance evaluation.” In Int. Work-
shop on Computing for Civil Engineering (IWCCE). Seattle, WA:
ASCE.
Tung, P.-C., Y.-R. Hwang, and M.-C. Wu. 2002. “The development of
a mobile manipulator imaging system for bridge crack inspection.”
Autom. Constr. 11 (6): 717–729. https://doi.org/10.1016/S0926-5805
(02)00012-2.
Yeum, C. M., and S. J. Dyke. 2015. “Vision-based automated crack detec-
tion for bridge inspection.” Comput.-Aided Civ. Infrastruct. Eng.
30 (10): 759–770. https://doi.org/10.1111/mice.12141.
Zakeri, H., F. M. Nejad, and A. Fahimifar. 2017. “Image based techniques
for crack detection, classification and quantification in asphalt pave-
ment: A review.” Arch. Comput. Methods Eng. 24 (4): 935–977. https://
doi.org/10.1007/s11831-016-9194-z.
Zhang, C., and A. Elaksher. 2012. “An unmanned aerial vehicle-based im-
aging system for 3D measurement of unpaved road surface distresses.”
Comput.-Aided Civ. Infrastruct. Eng. 27 (2): 118–129. https://doi.org
/10.1111/j.1467-8667.2011.00727.x.
Zhang, L., F. Yang, Y. D. Zhang, and Y. J. Zhu. 2016. “Road crack
detection using deep convolutional neural network.” In Proc., 2016
IEEE Int. Conf. on Image Processing (ICIP), 3708–3712. New York:
IEEE.
Zhang, Q., Y. Tian, Y. Yang, and C. Pan. 2014. “Automatic spatial–spectral
feature selection for hyperspectral image via discriminative sparse mul-
timodal learning.” IEEE Trans. Geosci. Remote Sens. 53 (1): 261–279.
https://doi.org/10.1109/TGRS.2014.2321405.
J. Comput. Civ. Eng.