Automation in Construction 132 (2021) 103936

Contents lists available at ScienceDirect

Automation in Construction
journal homepage: www.elsevier.com/locate/autcon

Machine-learning based vulnerability analysis of existing buildings

Sergio Ruggieri *, Angelo Cardellicchio, Valeria Leggieri, Giuseppina Uva
DICATECH Department, Polytechnic University of Bari, Bari, Italy

A R T I C L E I N F O A B S T R A C T

Keywords: The paper presents a machine-learning based framework, named VULMA (VULnerability analysis using MAchine-
Vulnerability analysis learning), for vulnerability analysis of existing buildings. The underlying idea is to provide an indication of the
Existing buildings seismic vulnerability by exploiting available photographs, which can be properly processed to provide some
Machine-learning
input data for empirical vulnerability algorithms. To this scope, a complete processing pipeline has been defined,
which consists in four consecutive modules offering different and specific services. The first module, Street
VULMA, performs the image gathering starting from the raw data; the second module, Data VULMA, provides a
mean for the data labelling and storage; the third module, Bi VULMA, uses the collected data to train several
machine-learning models for image classification; the fourth module, In VULMA, performs a ranking of the
images, their analysis and consequently assigns the vulnerability index. The proposed procedure has been
employed on the existing building portfolio in an extended area of the municipality of Bisceglie, Puglia, Southern
Italy, for which all the modules have been tested and, above all, the machine-learning models of Bi VULMA have
been trained. After, in order to test the efficiency and the reliability of the proposed tools, the entire procedure
has been applied on five case study buildings. The results in terms of vulnerability index have been compared
with the manual computations performed by the authors applying the same algorithm. Despite the proposed tool
could be improved or modified in some of its modules, the obtained results show a good effectiveness of VULMA,
which opens new scenarios in the field of vulnerability assessment procedures and risk mitigation strategies.

1. Introduction of a building taxonomy (characterized by few features such as year of

The development of fast procedures of vulnerability analysis for
seismic actions represents one of the main challenges faced by scientific
community and public institutions in the last years, with the aim to
support requalification strategies of the existing building stock by
developing risk mitigation plans to prioritize the economic resources. To
this end, especially in seismic-prone areas, several approaches for
quantifying the seismic safety of existing building portfolios have been
developed, mainly oriented at a large-scale screening on the main
vulnerable parts of the building stock. As recently summarized in Silva
et al. [1], among the wide range of proposals dealing with vulnerability
models at class-level in which the fragility/vulnerability assessment is
performed through a direct derivation of the relationship between
seismic intensity and damage states/losses, the two fairly popular
methods are empirical and mechanical ones. Empirical methods (also
called observational) consist in the derivation of the vulnerability
function from the damage observed after the earthquakes occurred in
the focused area. The damaged buildings are firstly classified by means
construction, construction materials, number of storeys, etc.), and data
about damages are then statistically processed to calibrate a class-
specific function, able to provide information about seismic vulnera­
bility. Several examples of empirical methods are provided by the sci­
entific literature, with regard both to reinforced concrete (RC) [2,3] and
masonry buildings [4,5]. Methods of this type are also often applied on
complex buildings, such as masonry churches [6,7], for which the
detailed identification of the behaviour under earthquakes becomes
hardly feasible. Mechanical methods (also called analytical) consist in
the derivation of the vulnerability functions from the analytical estimate
of the seismic behaviour of a building class, by means of a numerical
model (or a sample of numerical models) representative of the focused
typology. Again, there are myriads of scientific works with regard to RC
[8–10] and masonry buildings [11], of which only a few will be here
mentioned. Still, other methodologies are available, such as hybrids
approaches or rapid visual screening methods, which are often used for
strategic and socially important buildings, such as schools [12] and
hospitals [13].

* Corresponding author.
E-mail address: sergio.ruggieri@poliba.it (S. Ruggieri).

https://doi.org/10.1016/j.autcon.2021.103936
Received 31 May 2021; Received in revised form 2 August 2021; Accepted 31 August 2021
Available online 6 September 2021
0926-5805/© 2021 Elsevier B.V. All rights reserved.
S. Ruggieri et al.
From this brief introduction, it is quite clear that the options for
vulnerability analysis are manifold, with a continuous growth and
refinement of the methodologies, which from one side improve the
quality of the results and, on the other side, strongly reduce the
computational overburden related to the statistical and numerical ana­
lyses. Despite these beneficial aspects, however, it is worth reminding
the crucial problem in this kind of analyses: the availability of data. As a
matter of fact, for any mechanical or empirical method, the analyst
needs to dispose of a large amount of data to process. In this framework,
the phase of inventory of the building portfolio is fundamental for the
evaluation of different typologies of impact scenarios at a large-scale
and, it represents the basis for identifying the distribution of different
vulnerability classes in a geographic area that presents buildings having
similar features, given a certain taxonomy. Obviously, different degrees
of detail can be achieved, depending on the size of the building stock, the
distribution of buildings on the territory and the adopted assessment
model [14]. Additionally, perhaps the most important problem in the
data gathering is the possible occurrence of human errors, considering
that the input of fragility/vulnerability methods is provided by engi­
neering judgments about buildings and their elements. In the end, the
availability, accessibility and reliability of data sources for vulnerability
analysis at large-scale is yet an open issue, despite a consistent state of
the art about data collection procedures (see Section 2) and several
freely accessible web platforms are available (e.g., [15]).
To face this problem and to improve the data reliability, in this study
we are proposing a new machine-learning based method for identifying
the key features of buildings in the existing stock, simply starting from
the image of a building. As originally conceived, the tool can be used for
several scopes, from the definition of the fast vulnerability response on
the individual buildings to the seismic vulnerability mapping to be
employed in large-scale risk assessment procedures. Herein, we are
going to train the entire methodology on the existing building portfolio
in a specific area and after, we test the results for five different case study
buildings but the tool could be directly used for an urban scale appli­
cation. The main advantage of the procedure is to provide a supervised
classification of images for the identification of a simple vulnerability
index, by reducing the bias introduced in the phase of subjective eval­
uation of the building features from the judgmental assignments of
domain experts. The proposed tool, which has been named VULMA
(VULnerability analysis using MAchine-learning) is composed by four
consequent modules, which will be defined in the following Sections.
Even if VULMA is still a proof-of-concept, an initial processing pipeline
has been established; however, despite in the proposed tool we adopt
existing scientific-based methods, such as transfer learning in Bi VULMA
(see Section 3.3) and the vulnerability index proposed by [65] in In
VULMA (see Section 3.4), each of the proposed modules may be subject
to future modifications in order to improve its overall efficiency and
reliability.
2. State of the art about data collection of building portfolios
2.1. Existing approaches and applications of seismic vulnerability
assessment for class-level purposes
The main aim of data collection for seismic vulnerability purposes is
the definition of prioritization ranks for seismic risk mitigation strate­
gies. The most common methodologies of class-level investigation are
based on the classification into groups of buildings having common
morpho-typological features and that could show overall comparable
performances [16]. The development of schemes of building classifica­
tion can be improved through the preliminary identification of all
recurrent geometrical and mechanical features of the buildings
belonging to a specific area or region and the consequence definition of a
certain number of classes. At this point, it is possible to proceed with the
assignment of a class to each building, aiming to achieve a good
compromise between the accuracy of results and the management of
2
Automation in Construction 132 (2021) 103936
data. With regard to the accuracy, the availability of several large-scale
methods developed in the last five decades [17] allows to combine
several kinds of datasets characterized by different levels of detail.
Concerning to the management aspects, it is paramount to collect a huge
amount of data, but these have to be elaborated with the aim to define
specific classes of buildings under common taxonomies, in order to
provide more efficient studies of the seismic vulnerability [18].
In most approaches currently used in many countries, seismic risk
assessments employ classification schemes based on early scientific
works. An example is the Medvedev-Sponheuer-Karnik scale (MSK–64)
[19], in which buildings were classified into three distinct vulnerability
classes according to their construction material and type of load-bearing
system. Later, this classification scheme evolved into the European
Macro-seismic Scale – EMS98 [20] that included three additional clas­
ses, in order to account further structural typologies (e.g. structural
steel). Another classification example is provided in HAZUS [21], in
which the building stock is represented by typical buildings designed or
not according to the seismic standards (High-Code, Moderate-Code or
Low-Code) and the main parameters detected are the structural system
and the usage or occupancy class. In HAZUS, the structural system is
assumed to be a key factor in the assessment of the overall building
performance, loss function and fatalities, and 36 model building types
are proposed and used to classify buildings within the general categories
of wood, steel, concrete, masonry building systems. Another example is
provided by the RISK-UE projects [22], in which the vulnerability
analysis requires only the age of construction and general information
about the building typology. In this framework, the analyses of the most
recurrent different building types in all European and Mediterranean
countries were performed, proposing a matrix of 23 building typologies
having similar performance under seismic events. Recently, the GEM
Building Taxonomy [19] has described and classified buildings in a
uniform framework for achieving a reliable assessment under seismic
actions. The criteria for developing the taxonomy were based on the
seismic performance of different construction typologies.
Despite the existing several classification schemes, the most critical
issue remains the one of data retrieval. Polese et al. [14] have proposed
an exhaustive list of the techniques available in the recent literature,
starting from those applicable to a wide scale [19,23] to those looking at
a more specific level [24]. Accordingly, the data collection phase will
take into account the different levels of insight required by the different
application scales. Obviously, the applicability scale is strictly related to
the typology of data, the available source and the scopes of the evalu­
ation. With regard to the methods for data collection highlighted in [14],
four different sources for extracting information about the existing
building stock are available: (i) Census data; (ii) interview-based sur­
veys; (iii) Geographic Information Systems (GIS) and remote sensing
methods (high resolution – HR, or very high resolution – VHR satellite
imagery); (iv) building by building.
Census data are perhaps the simplest source to use, considering that
each Country periodically performs building Census, and provide basilar
information about buildings (e.g., storey number, age, construction of
material, state of preservation). Following a similar paradigm, but in a
more refined way, interview-based surveys characterize the primary
methods adopted for collecting data. These procedures require expert
judgments by specialised technicians, who perform rapid in-situ surveys
and evaluate the properties and the actual state of the observed build­
ings. In this framework, the main Italian method is represented by the
GNDT form [25] developed by the National Group for Defence against
Earthquake (subsequently adopted and extended by several researches,
such as Uva et al. [26] or Cara et al. [27]), which provides two different
levels of detail. The first one requires data about location, geometry,
usage class, year of construction, structural typology, maintenance and
damage state, while the second one consists in the collection of 11 pa­
rameters for masonry buildings and 4 parameters for RC buildings [28].
Eventually, the parameters are used to extract a vulnerability index. A
recent advancement towards the compilation of regional scale
S. Ruggieri et al.
inventories is provided by Cartis approach [29], implemented in Italy by
Civil Protection Department in ReLUIS project. The Cartis survey form
has a different approach form the GNDT form, aiming to collect data
about the recurrent features of some building classes within homoge­
neous urban sectors (USs). These units are geographic zones character­
ized by a homogenous building stock in terms of year of construction,
construction material and structural typology. The procedure is based on
the gathering of data through a form organized in 4 sections, and the
subsequent definition of the recurrent features:
• Section 0: delimitation of urban sectors;
• Section 1: identification of the most common buildings typology in
each urban sector;
• Section 2: identification of general features of each building typology
class;
• Section 3: characterization of the structural elements of all building
typology classes.
The Cartis survey form (one form for each town, comprising one or
more USs) is compiled by interviewing one or more technicians (e.g.
engineers or architects) that are local experts with a deep knowledge of
the construction features in the area. If properly disaggregated and
processed, the information collected on building typologies through the
Cartis form is potentially useful for applying more refined vulnerability
models than the ones employable through the simple Census or GNDT
data.
Turning to remote sensing and GIS-based approaches, the most
innovative techniques rely on image processing, taken as a source for
building inventory to rapidly gather widespread geo-referenced infor­
mation about the building stock. Building features can be objectively
evaluated from high resolution (HR) satellite imagery, through the
identification of footprints and size, number of floors, height of floors
etc. However, some building features that are crucial to vulnerability
assessment, such as construction material or age, cannot be easily
established by these methods and, for this reason, they should be com­
bined with other sources of information. On the other hand, the power of
GIS platforms is the possibility to easily provide access to information,
which can be transferred, transformed, overlaid, processed and dis­
played using numerous software applications and a wide variety of
formats. In addition, GIS databases are flexible, freely available, open-
source and capable to contain multi-hazard information. Several ex­
amples are available in the literature and can be mentioned [30–32],
which show the use of remote sensing and GIS for collecting big amounts
of data with a low burden and allowing a large evaluation of the building
typologies to employ in assessment procedures.
Finally, building-by-building surveys represent the most detailed
source of data for individual buildings in a specific area and are aimed to
perform an accurate vulnerability assessment of the investigated unit. Of
course, detailed investigations at building-level, as required by national
and international building codes, are of difficult application for large-
scale purposes, considering the requirement of time, computational ef­
forts and, above all, the necessity of economic resources. For these
reasons, this kind of detailed survey is generally applied only during
post-earthquake vulnerability and damage survey campaigns [33,34], to
assess buildings that can be yet used or as a benchmark information for
data mining approaches [35,36].
In this broad framework of methods aimed to the collection of
building data for large-scale analyses, it is possible to explore new
methods based on more recent concepts that in the last years are gaining
ground, such as machine-learning.
2.2. Use of machine learning approaches: recent applications and new
perspectives for seismic vulnerability assessment at class-level purposes
The use of the machine learning (ML) in civil engineering application
is gaining momentum, especially in the research fields related to the
3
Automation in Construction 132 (2021) 103936
definition of buildings performance. ML can be defined as a branch of
Artificial Intelligence (AI), which collects several widely used
methodologies.
In general, ML approaches can be divided in two main categories: (a)
supervised learning; (b) unsupervised learning. The main difference
between them lies in the data used to feed the algorithm, as the first type
of algorithms uses labelled data, while the latter does not. The concept of
labelled data is mainly related to domain expertise: the intrinsic
meaning of labelled data is determined beforehand by a domain expert.
As such, supervised algorithms try to mimic domain expertise by finding
relationships which bind an input (i.e., a new data sample) to an output,
which can be either a class (for classification problems) or a numeric
value (for regression problems). As for unsupervised algorithms, they do
not rely on previous labelling, therefore trying to gather insight directly
from the inner structure of data. A typical example of unsupervised
learning are clustering algorithms, where data are divided according to
their relative distance in a n-dimensional space, where n is the number
of features available for each sample.
Here, it is necessary to mention that several applications have been
developed in the scientific community in the last years with regard to
structural engineering, and this will be done by referring to some review
papers (and references therein) recently released. A comprehensive re­
view is proposed by Xie et al. [37], who have explored the possible
application of ML to earthquake engineering, accounting for the first
two of four traits of interest in the field, as well as ML methodologies,
topic area, data resource and scale of analysis. More in detail, they
assessed seven classes of ML approaches applied on four areas of
application in which ML approaches are used to support seismic hazard
analysis, structural identification and damage detection, seismic
fragility analysis and structural control. The scientific literature includes
other review papers about ML methods in structural design and assess­
ment of structure and infrastructure, as recently released by Sun et al.
[38] and Falcone et al. [39]. In Sun et al. [38] the authors have
formalized all the techniques of supervised and unsupervised learning
and explored the application in building structural design and perfor­
mance assessment by identifying four classes of application: (a) struc­
tural response and performance prediction, (b) use and interpretation of
experimental data to predict the structural properties of components, (c)
use of the images to gather information; (d) application in structural
health monitoring.
Herein, for the scopes of this paper, the problem of data collection,
which can represent the limit in the application of any kind of vulner­
ability analysis method, is of interest. In particular, in order to have
efficient predictive models, it become essential to dispose of sufficient
and high-quality data that can be investigated. Furthermore, when using
ML approaches, data should be carefully sampled, mainly to avoid
negative effects, such as imbalanced data [40] or overfitting, which is a
common problem in large ML architectures [41]. In addition, one of the
main problems is due to the availability of the data for training the ML
algorithms, which for any kind of purpose should be freely accessible
and subjected to quality control measures. Some solutions to this issue
can be mentioned, where the most significant is represented by the ac­
tion of the expert domain, which uses the proper knowledge to reduce
the amount of data for achieving good solutions. Still, techniques of
transfer learning or generative models (e.g. Monte Carlo simulation,
often employed in structural engineering problem) could be used to
increase the starting dataset. Additionally, another concern is repre­
sented by the data quality, which can heavily affect the results, caused
by the presence of outliers in the dataset.
In the end, considering this brief overview (which is not compre­
hensive of the myriads of applications proposed by the scientific liter­
ature), our aim is to employ ML approaches to develop seismic
vulnerability assessment for class-level purposes. For large scale anal­
ysis, few applications are available in the scientific literature. For
example, Harirchian et al. [42] proposed a prototype of a smartphone
app aimed to collect data and to assess vulnerability on the base of the
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

combination among the geometrical features that affect seismic per­

formance of RC buildings. Another example of ML approach is provided
in the work by Mangalathu et al. [43], which adopted ML approaches for
predicting damages on buildings provided by earthquakes on the base of
a dataset of about 2000 buildings and accounting for features as spectral
acceleration, soil category, year of construction, number of storeys, base
area and presence of irregularities. Again, Mangalathu et al. [44] pro­
posed a ML based method for the assessment of damages caused by
earthquakes on bridge portfolios.
Anyway, as mentioned in Section 2.1, the actual practice for large-
scale assessment is based on frameworks that employ large efforts of
technicians and researchers in the definition of the main parameters to
be considered in an analytical/empirical vulnerability model. For these
reasons, VULMA represents a real possibility for overcoming some of the
main issues intrinsically contained in the methods shown in Section 2.1.
Especially for the most employed methodologies (that use Census data
and interview-based surveys), a ML approach could overcome the
subjectivity of the surveyor judgments and, above all, could strongly
reduce the time related to surveys and interviews. Also for the GIS
methodologies, a ML approach could provide data that can be easily
integrated with the existing databases, to complete the set of existing
data to use for a large-scale analysis.

2.3. VULMA: tool definition and organization of modules

The proposed tool is named VULMA, which stands for VULnerability

analysis using MAchine-learning. The aim of VULMA is to provide a
pipeline framework for automatically extracting a vulnerability index
from raw data associated to a building. It is composed by several mod­
ules, each one offering specific features. Each module can be used as an
individual tool for handling a specific use case. Let us assume an overall
flow, concerning each module, where the rationale behind VULMA can
be summarized by the simple flow charts shown in Fig. 1. The relation
between the two flow charts is established according to the adopted
colours, where to each VULMA module (right) corresponds one or more
specific tasks (left). Still, in yellow are highlighted the roles of domain
experts, as following specified.
The first step is raw data provisioning. Data usually concern a
building stock within a certain geographical area and are provided by
freely available georeferenced databases (e.g., Technic Regional Car­
tographies, CTRs) that contain georeferenced polygons encompassing
buildings in the area of interest. These data have a great utility, espe­
cially for the historical centres, where the localization of buildings is
provided for masonry aggregate blocks that cannot be considered as
single unit. VULMA accepts data in a standardized GeoJSON format,
meaning that it can be adapted to (and used into) several application
scenarios.
In the second step, raw data are sent to a data gathering module,
named Street VULMA, which processes them to extract images taken
from online services, such as Google StreetView. Obviously, images
gathered during this step do not present any kind of labelling. Therefore,
the subsequent phase requires a collective effort from domain experts,
who should identify the main features of interest for each gathered
image beforehand. To support the time – intensive labelling task,
domain experts can rely on another module, which allows for data
labelling and storage, named Data VULMA. Once labelled data have
been stored, they can be used as a base for training a ML model (i.e., a
neural network) for image classification. This is possible using another
module provided by VULMA, which is named Bi VULMA.
Finally, processed data can be associated with a vulnerability index,
which is provided by the last module included in the VULMA toolset,
named In VULMA.
This brief introduction allows to understand the logical reasoning
behind our proposal, which is summarized in the sequence diagram
shown in Fig. 2. In the next sections, each module will be described in
detail. In the end, it is worth highlight the role of VULMA as a real
Fig. 1. Abstract flow charts of the interactions among VULMA modules. The
colours associate the modules of VULMA (right) and the tasks of each module
(left). In addition, domain experts’ roles are highlighted in yellow. (For inter­
pretation of the references to colour in this figure legend, the reader is referred
to the web version of this article.)
Decision Support System. As a matter of fact, VULMA does not neces­
sarily take autonomous decisions on the status of a specific building, but
instead provides automatic support to domain experts (especially for
large-scale purposes), whose role will be to validate the suggestions
provided by VULMA.
2.4. Street VULMA
Street VULMA is the first module of VULMA and it is able to gather
data concerning buildings from online services, such as Google Street
View, starting from raw data of one/a set of buildings. It is written in
Python [45], and it offers a simplified interface, with two main
submodules:
• The Fetch submodule, which has the task to take a GeoJSON file as
input and to fetch images of buildings included within the borders of
provided data (given such information are available within Street
View).
• The Clean submodule, which is responsible to remove duplicates
from data fetched by the Fetch submodule.
As for the Fetch module, it acquires images by fixing three different
parameters: 1) pitch (i.e., the vertical angle of the camera); 2) field of
view (i.e., the horizontal angle of the camera, which can be adjusted to
provide a certain zoom effect); 3) heading (i.e., the angle where the

4
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

Fig. 2. Sequence diagram of the operation of the VULMA toolset.

camera is headed). Data are acquired considering a spatial granularity where images fetched from Internet will be downloaded; (iii) using the
(e.g., the distance between two adjacent geographical positions) of 5 m. button Ok, to download photos; (iv) to remove the possible duplicates
After the acquisition of the images, they are automatically converted through the activation of the Clean module. Once that the photos have
in ASCII string format according to a radix-64 representation (Base64). been downloaded, they are stored in a proper database, in order to be
This allows to evaluate the SHA-512 hash function of the Base64 version elaborated in the next module.
of the original image, computing a fixed-length code, which is univo­
cally related to the original image. Hence, by comparing the hashed
2.5. Data VULMA
representation of all images pairs, the Clean submodule is able to
identify, and therefore discard, duplicate images. It is worth noting that
Once that the photos have been stored, it is necessary to switch on
the noise characterizing some images has been considered as an
the second module of VULMA, named Data VULMA. Here, it is required a
“advantage” for the proposed procedure, considering noise as a non-
labelling step, performed by domain experts. This step is particularly
systematic bias for a subset of the whole image dataset (images are
important, because labelled data will be used to build a supervised
supposedly acquired by the same camera, i.e., the one used by the
model for image classification. Obviously, an open issue is the lack of
Google Car, and are mainly related to changes in lighting and weather
objectivity caused by experts while selecting labels [46], which may
conditions). This introduces a variability component in the dataset,
result in biases inserted in the overall model. To overcome it, data will
which leads to a model constituted by better generalization capabilities.
be labelled twice by two different independent domain experts, making
In the end, noisy images are commonly retained in object recognition
the labels less prone to subjective biases. For the case at hand, to assign a
approaches through deep neural networks and data augmentation ap­
unique label to each image, authors used a weighted consensus procedure.
proaches, such as the one used in this work and accounting for the
Namely, after an internal training, two of the authors independently
addition of random noise to original images. An example of images
performed the first labelling (splitting the total amount of photos in two
gathered by Street VULMA is shown in Fig. 3 (a – d). The interface
parts). Afterwards, another author checked the outcomes from the
provided by Street VULMA is shown in Fig. 4, where it is possible to
previous step and, as his/her labelling was assumed to have a higher
upload the GeoJSON file obtained by user from other sources. Once the
weight than the ones from the first domain experts, it was more relevant
GeoJSON is available, the following operations must be performed: (i)
for the final decision. In addition, for the scopes of our application,
using the button Browse, to select the GeoJSON location from the user’s
which aim to define a vulnerability index that usually depends on the
local machine and upload the file into the tool; (ii) to report the folder
subjectivity of surveyors (see Section 2.1), the proposed double data

5
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

Fig. 3. (a – d) – Example of images gathered through Street VULMA.

Table 1
Labels and type of assignment to photos by domain experts (*Typology in­
dicates an established number of options, selectable from a drop-down menu;
**Number indicates an integer number to assign; *** Yes-No is a Boolean
indication to indicate the presence or not of the observed parameter).
Labels Type of assignment

Structural typology Typology*

Number of units Number**
Number of storeys Number
Presence of pilotis floor Yes-No***
Presence of basement floor Yes-No
Presence of superelevation floor Yes-No
Total number of openings Number
Fig. 4. Graphical interface of Street VULMA
Type of roof floor Typology
Presence of vaults Yes-No
labelling could help to improve the key aspect about objectivity, char­ Presence of visible seismic details Yes-No
acterizing this practice. Basing on the abovementioned concepts, Presence of higher ground floor Yes-No
Presence of overhangs Yes-No
domain experts are called upon to elaborate images (one at time) and to
Regularity in-plan Yes-No
assign the proposed set of labels, only basing on the figure observation Regularity in-height Yes-No
and avoiding any kind of mental processing. Firstly, images are cropped
to highlight the relevant content (e.g., the building itself) and reduce the
influence of occlusions. Afterwards, each image is labelled according to the image, avoiding making hypotheses and ensuring to cover all
the criteria defined in Table 1 and according to the proposed types of possible detectable features from the most common typologies of
assignment. It is worth noticing that if two images contain the same buildings. If the domain expert is not sure or does not see the detail, it is
building but displayed from different positions, they are treated as possible avoid assigning the feature. This latter possibility will not in­
different buildings (perhaps with the same labels value). fluence the following phases (see Section 3.3 and subsections), where we
More in detail, all the required labels are defined only by observing can anticipate that the involved ML procedures will be independently

6
S. Ruggieri et al.
applied on each label. This means that, for each image, if the value of a
specific label is not assigned, that picture will not be considered for
training that specific classifier. Still, the same image will be used to train
other classifiers, for which a valid label has been assigned by domain
experts. Regarding to labels, the structural typology and the type of roof
floor can be defined by observing, respectively, the structural material
(RC, masonry, steel) and the kind of cover on the floor (dome, pitches,
flat). Still, number of units, storeys and openings can be defined ac­
cording to the count of the features on the photo. Obviously, if the im­
ages have been cropped, number of units is equal to 1 while, for
aggregate buildings, the number of units increases. Some properties are
indicated with a Boolean indication (Yes–No). For example, the pres­
ence of basement floor can be inserted if some openings are visible at the
base of the building; the presence of superelevation floor can be inserted
if the picture shows that the colour and size of the last floor are different
from the rest of building; vaults and seismic details can be inserted with
reference to masonry buildings if there are evident features (e.g. capo­
chiave belonging to steel ties); the presence of a higher ground floor can
be inserted if a shop is visible in the image; regularities and irregularities
(in-plan and in-elevation) can be inserted if the picture shows evident
recesses along the elevation of the focused building.
It is worth mentioning that the proposed features have been estab­
lished on the base of the authors’ opinion, with the aim to satisfy the
following requirements: (1) possible information that can be objectively
observed in a photo of a building (or of a group of buildings); (2) con­
struction typology of the observed building (or the observed group of
buildings); (3) possible features representing vulnerability sources for a
building (or for a group of buildings); (4) possible information to define
a simple vulnerability index based on the morpho-typological features of
a building (or of a group of buildings). Matching the answers provided to
the above guidelines, authors selected the labels reported in Table 1. Not
all the detected features will be used but only few of them will be
implemented in this work. At the same time, the set of labels could be
increased in the future and used even for making a mechanical model.
Nevertheless, in this work we evaluated all the declared features for each
photo with an appropriate model.
In order to allow the storage of data labelled by domain experts, Data
VULMA has been structured for providing a specific web service having a
web architecture as shown in Fig. 5.
To ensure future enhancements, Data VULMA provides full support
to two important features:
• geospatial databases, made by means of GeoDjango and PostGIS;
• containerization through Docker.
As for usability, Data VULMA provides a user-friendly interface for
adding new data, which can be easily used by domain experts who may
not have a strong ICT background, as shown in Fig. 6. The graphical
interface contains all labels in Table 1 that can be set by domain expert,
by means of drop-down menu for assigning a typology, checkboxes for
assigning a Boolean value and textboxes for assigning a number. On the
base of the structural typology, the system allows or not to add some
information (e.g., presence of vaults cannot be set in the case of RC
building). In addition, Data VULMA allows to import images and to
assess the entire database.
Fig. 5. Architecture of Data VULMA.
7
Automation in Construction 132 (2021) 103936
2.6. Bi-VULMA
The third module is Bi VULMA, which is used to perform both the
training and classification steps shown in Fig. 1. More in detail, Bi
VULMA allows to train a (deep) Convolutional Neural Network (CNN)
for image classification. Following, two Sections are provided, with a
general introduction to CNN algorithms and the principles at the base of
Bi VULMA.
2.6.1. Convolutional neural network and deep learning
CNNs have significantly advanced the state of the art in computer
vision, mainly after the seminal contribution given by Krizhevsky et al.
with AlexNet [47], which achieved outstanding results in image classi­
fication on the extremely challenging ImageNet dataset [48].
Since then, the architecture of CNNs has continuously evolved,
embedding a growing representational power through new outbreaking
concepts and intuitions. In the following, we will briefly introduce some
of the most relevant among these advances, on which Bi VULMA has
been developed.
2.6.1.1. Base layer of a CNN. A typical layer in a CNN is composed by
three stages, which are represented by a set of convolutions, a nonlinear
activation function and a pooling step. Convolutions [49] are the main
building block on which CNNs are built. For a two-dimensional signal,
such as an image I(m,n) where m and n are the number of picture ele­
ments (pixels) for the width and height of I, convolution is expressed as:
∑∑
(i, j) = (I*K)(i, j) = I(m, n)K(i − m, j − n) (1)
m n
where K is a two-dimensional kernel. Convolution layers enable
three important properties of a CNN:
• Sparse connectivity: in traditional neural networks, interactions be­
tween layers are dense, meaning that every neuron at layer w in­
teracts with each neuron at layer w-1, for every w. In CNNs, however,
this is usually not true and is accomplished using a kernel K whose
dimensions are lower with respect to the sizes of I. This reduces the
number of parameters used by the CNNs, allowing for an improved
efficiency in terms of memory requirements and statistical
significance;
• Parameter sharing means the value of a certain input xi applied to a
specific output si is tied to the value of the input xj applied to the
output sj. Intuitively, this is achieved by CNNs, as the same kernel is
applied to each pixel of I. Parameter sharing therefore allows for
memory efficiency, as fewer parameters are learned and stored;
• Equivariance to translations: this is directly derived from parameter
sharing and causes a shift to the output which is directly related to a
shift applied to the input. That is, if g is a function which shifts the
input of the image, and f(xi) is the output of the convolution applied
to the input xi, then f(g(xi)) = g(f(xi)).
After the convolution step, a nonlinear activation function is applied.
This is mainly done to insert nonlinearities within the CNN, which ac­
count for real-world effects. Historically, sigmoid functions have been
used [50], however, starting from AlexNet, rectified linear units (ReLU)
have been preferred, due to their capability to achieve same results in a
more efficient way. A ReLU is expressed as follows:
{
xi if xi ≥ 0
f (xi ) = (2)
0 if xi < 0
which means every positive value will be propagated as is to the next
layer, while negative values will be set to zero. The last step is pooling,
which further modifies the output of the layer, by subsampling the input
signal replacing each subset of its pixels with a fixed representation.
S. Ruggieri et al.
Fig. 6. Graphical interface of Data VULMA.
2.6.1.2. Dropout. Dropout layers have achieved increased popularity
after their introduction by Srivastava et al. in [41], and are now
considered as one of the main regularization layers within CNNs. Let us
assume the standard feed forward operation for a CNN (and, more
generally, for deep neural networks) can be described as follows:
(3)
(l+1) (l+1) l (l+1)
si = wi yi + bi
( )
y(l+1)
i = f s(l+1)
i (4)
where l ∈ {1, …,L} is the lth hidden layer within the network, sl the inputs
into layer l,yl the outputs from layer l, the pair (Wl,bl) weights and biases
learned by the network at layer l, and f the nonlinearities applied to sl to
get yl. Dropout regularization weights each output value for a vector of
independent Bernoulli random variables rl, each one with a probability p
to be equals to 1. Obviously, due to the nature of the Bernoulli distri­
bution, a reduced output vector ỹ(l) will be produced. This has the effect
to randomly drop nodes during each epoch, therefore allowing the
network to assume different points of view throughout the whole
training, causing a mitigation of overfitting phenomena.
2.6.1.3. Residual blocks. The first generation of CNNs tried to stack as
8
Automation in Construction 132 (2021) 103936
many convolution layers as possible on top of each other, assuming that
a more complex representation would lead to improved classification
results. However, over a certain limit, the accuracy of the network in
terms of classifying out-of-sample data saturates and, eventually, de­
grades. This effect, commonly known as the degradation problem, has
been shown by He et al. in [51]. To overcome this problem, He et al.
introduced the concept of residual blocks.
The rationale behind residual block is to allow deep networks to
match performance of shallower networks; this is achieved by skipping
layers which cause the degradation problem. Let us note that the term
residual derives from what the network is trying to learn. As a matter of
fact, denoting with H(x) the real output of the network, the residual R(x)
is given by:
R(x) = H(x) − x (5)
which implies:
H(x) = R(x) + x (6)
As an intuitive consequence, we can state that the residuals R(x) are
what is actually being learned by the inner layers of the network.
2.6.1.4. Inception layers. Inception modules [52] have been introduced
S. Ruggieri et al.
as an evolution of classic convolution layers. First, let us expand the
concept of image, moving from a m x n image to a m x n x 3 tensor, as
information for each colour channel (red, green and blue) are stored in a
three-dimensional representation.
As previously stated, a single convolution kernel achieves sparse
representation and parameter sharing. This implies that correlations
between each pixel within a single channel (i.e., spatial correlations)
and between pixels belonging to different channels (i.e., cross-channel
correlations) are learned jointly. The intuition that motivates the
inception module is that such correlations are decoupled. Therefore, an
inception module tries to independently learn spatial correlations and
cross-channel correlations by using a stacked set of convolution kernels.
Specifically, 1 × 1 convolutions are first used to look at cross-channel
correlations; then, the input of such convolutions is mapped to a small
set of separate spaces (usually three to four), which are independently
explored via regular 3 × 3 or 5 × 5 convolution kernels.
2.6.1.5. GPGPU and ImageNet. The advancement highlighted in previ­
ous Sections are only a small part of the overall research which has been
conducted on CNNs over the last decade. Still, they clearly show how
wide has been the adoption of CNNs (and, more generally, deep
learning), even though these ideas can be traced back to the 1960s, with
the work made by Hubel and Wiesel on visual cortex [53]. However,
until the end of the 2000s, CNNs were not practically feasible due to
severe limitations in both computational capacity and available data­
sets. This is directly related to the number of parameters which must be
learned in a deep CNN, which is often in the magnitude of millions.
Therefore, the most significant advances have been the introduction of
ImageNet [47], a large dataset composed by several millions labelled
images, and general-purpose computing on graphics processing units
(GPGPU), which provided enough computational power to train com­
plex deep network architectures. Nowadays, GPGPU is easily accessible,
due to the availability of cheap, commercial-grade GPUs able to train
complex networks from scratch. However, gathering an adequate
amount of data for specific use cases is still a significant issue. Hence,
data augmentation techniques have been developed [54]; however,
these may not always prevent overfitting and, consequently, poor clas­
sification performance. Therefore, the use of CNNs in real scenarios with
limited datasets is often led by two approaches, named transfer learning
and fine tuning, which can be combined to achieve state-of-the-art
performance.
2.6.1.6. Transfer learning and fine tuning. The idea behind transfer
learning is simple and exploits the internal structure of a deep CNN. As
the work made by Yosinski [55], CNNs extract different representation
of the input image at the top of each one of their internal layers. Spe­
cifically, low level layers (i.e., the ones which are adjacent to the input)
extract generic features, such as edges and basic shapes, while high level
layers (i.e., the ones next to the classification layer) extract specific
features to the input image. This suggests that training only the pa­
rameters of the high-level layers on a dataset relative to a specific
problem may lead to optimal results even with a relatively small amount
of data, as the number of trainable parameters is significantly lower than
the one from the original architecture. Therefore, the idea is to use a
network which has been already trained on generic purpose data (often
ImageNet), and fix its parameter on a smaller, use-case specific dataset.
Optionally, transfer learning can be followed by fine tuning, which
consists in performing a round of training using a significantly low
learning rate, hence tuning all the parameters of the network on the
small use-case dataset.
2.6.2. Principle of operation and tool description of Bi-VULMA
Bi VULMA is written in Python 3 [45]. While most of its function­
alities depend on the state of the art machine learning libraries (spe­
cifically, TensorFlow [56], Keras [57] and Scikit-Learn [58]), its output
9
Automation in Construction 132 (2021) 103936
is mainly provided in a visual form. To this end, a frontend is provided,
implemented using PySimpleGUI, a wrapper on tkinker, one of the most
well-known graphic libraries for Python. Bi VULMA has two main
operating modes: the first is training, which allows to train a neural
network architecture either from scratch (assuming enough data is
provided) or exploiting transfer learning and fine tuning; the second is
inference, which allows to load a previously trained model to classify an
input image. The graphical interface of BI VULMA, presented while in
training mode, is shown in Fig. 7.
The most interesting section of this graphical interface is probably
the middle part, which allows users to define several parameters, as
listed below:
• Batch size, which is the number of samples (that is, images) using per
step during each training epoch.
• Image height and width, which is automatically adapted to selected
neural network architecture (however, the user is still able to edit
that parameter);
• Training epochs, which provides the maximum number of “times”
that the weights of the neural network will be refined during
training;
• Optimizer, which is used for optimizing network weights during
training. Currently, two algorithms are available, that is, Stochastic
Gradient Descent (SGD) [59] and ADAM [60];
• Loss function and accuracy, which is used to evaluate the perfor­
mance of the network in terms of correct evaluation of the class
assigned to each image. Currently, two types of losses and accuracies
are supported, specifically categorical and binary crossentropy/loss.
It is important to underline that the number of classes is automati­
cally inferred by the structure of the dataset itself, and both loss func­
tions and accuracy automatically adjust accordingly. Furthermore, there
are currently two callbacks which can be provided to the trainer: (i)
selectable using the Save logs checkbox, which can be enabled to sup­
port for visualizing the results achieved by the network during training;
(ii) use early stopping, which automatically stops network training when
no any improvement is achieved over a certain number of epochs, given
by the value associated to the patience parameter. It is important to
underline that early stopping can be useful to prevent overfitting [61].
Finally, Bi VULMA gives the option to choose from six base models to
perform transfer learning, that are MobileNetV2 [62], Xception [63],
ResNet152v2, InceptionResNetV2, InceptionV3 [51,52] and NasNet
[64]. Fig. 8 shows the graphical interface that appears to the user in the
case of inference mode. The inference operating mode allows to load a
pre-trained neural network model, which will be used to classify a given
image, selected on the left-side menu.
2.7. In VULMA
The last module allows to compute the vulnerability index for
buildings on the base of the previous steps. Several methodologies could
be employed in this step, considering the broad variety of methodologies
available in the scientific literature (see Section 2.1). For now, In VULMA
has been developed on the base of the simple approach proposed by
Frassine and Giovinazzi [65] to define a vulnerability index. The reason
of this choice is mainly due to the necessity to test In VULMA with a
methodology already available, which is simple and that allows us to
assess the efficiency of the entire proposed procedure. Of course, in the
future developments new methodologies will be proposed and, as above-
mentioned, the estimated parameters could be also used to define a
proper mechanical method for more accurately evaluating the seismic
vulnerability.
Coming back to the methodology in [65], it consists in the applica­
tion of the following formulation:
∑
ṼI = VIb + ΔV m (7)
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

Fig. 7. Graphical interface of Bi-VULMA.

Fig. 8. Inference mode in Bi VULMA.

where ṼI is the vulnerability index (ranging from 0 to 1, with higher or RC) and the year of construction, while the values of ΔVm are estab­
values that imply a more vulnerable building), computed by summing a lished on the base of some parameters influencing the seismic vulnera­
base vulnerability index (VIb) to some modification coefficients (ΔVm). bility quantification. For our scopes, not all the ΔVm parameters can be
Namely, VIb is defined on the basis of the structural typology (masonry considered in the evaluation and, for this reason, the corresponding

10
S. Ruggieri et al.
coefficients will be accounted as null values, as allowed by the original
method. Tables 2 and 3 report the sets of parameters and the related
values associated, respectively, to VIb and ΔVm.
It is worth noting that masonry buildings are classified according to a
range of the year of construction (according to the release of the Italian
building technical standards in the last century), while RC buildings are
identified through the level of seismic design (absent, low, medium). For
these latter, the concept is similar to the one adopted for masonry
buildings, considering that a medium level of seismic design can be
attributed to buildings constructed after the 1971, while absent and low
levels of seismic design could be identified, respectively, for buildings
dating back from 1919 to 1945 and from 1946 to 1970.
In order to apply this methodology to In VULMA, it is necessary to
firstly know the year of construction for the focused building. Consid­
ering that the Bi-VULMA is not able to provide this information, this
issue can be solved with a prior knowledge about the period in which the
focused area (from which the building image is taken) has been built.
This can represent a limit of VULMA and, more in general, of a ML based
vulnerability assessment at large-scale. On the other hand, this kind of
information is easily available (for example from the Census data) and it
can be automatically integrated in the architecture of the system, thanks
to the availability of the geographical coordinates of the focused
buildings. In addition, for a simple vulnerability index, the year of
construction is provided in a range form and hence it has not to be
extremely accurate.
Once that the value of VIb has been defined (based on the structural
typology), the values of ΔVm can be easily assigned, according to the
work made in Bi VULMA. More in detail, the modification factors
associated to number of floors, plan and vertical irregularities, super­
elevation floor, aseismic devices and structural element details
(considering the presence or not of the pilotis floor) can be evaluated as
defined in Table 3. Instead, the modification factors associated to state of
preservation, structural system, retrofit intervention (for masonry
buildings), and foundation cannot be accounted for. Finally, the modi­
fication factors associated to the aggregation of buildings (position and
elevation) are considered only in the worst condition (assumed equal to
+0.04) if the number of units in the image is greater than one. From the
base set of labelled parameters shown in Table 1, the number of open­
ings, the presence of overhangs and higher floor have not been used.
3. Application and training of VULMA: the case study of
Bisceglie, Puglia, Southern Italy
In order to apply the concept and the modules of VULMA to a real
case study, the municipality of Bisceglie, Puglia, Southern Italy has been
considered. Data of this municipality have been extracted and elabo­
rated with the aim to train VULMA to predict the vulnerability index.
Still, we can anticipate that the validation of the effectiveness of VULMA
has been carried out on different buildings out of the selected area. The
reasons for which we selected this municipality are manifold: (a) it is the
hometown of the first author; (b) all authors have a deep knowledge of
the municipality and its building stock; (c) the municipality is charac­
terized by RC and masonry buildings, homogenously built during the
time and distributed in the area.
As shown in Fig. 9, Bisceglie is located on the Adriatic coast and it
characterized by an urban centre (blue zone) surrounded by a large
agricultural area (dark grey). For the scopes of this paper, only a part of
Table 2
Set of parameters and related values associated to VIb [65].
Typology Masonry
Class I II III
Year of construction <1919 19–45 46–70
Design level – – – –
VIb 0.704 0.689 0.669
11
Automation in Construction 132 (2021) 103936
the inhabited centre has been considered. Fr this entire municipality we
dispose of a wide database from which the raw data to employ in Street
VULMA can be extracted:
1. Technic Regional Cartographies (CTRs);
2. Census Data, subdivided in Census sections.
A CTR is a vector file freely online available, shown in Fig. 10 for the
municipality of Bisceglie and implemented through a GIS environment.
A CTR contains several polygons representing all the urban blocks of
buildings belonging to a municipality and including a reduced set of
attributes, such as the use destination of the units enclosed, area and
height of each block.
Regarding to Census Data, a vector file for the desired municipality is
freely online available and contains several Census sections, each one
composed by few buildings grouped in a limited portion of the munic­
ipality and for which some specific data are provided. For each Census
section, information about structural typology, state of preservation and
number of storeys is available, even if it should be observed that the
buildings inside a Census Section could not have similar features.
In order to train VULMA, we have selected a “complete” portion of
the municipality characterized by a similar percentage of RC and ma­
sonry buildings. The building evolution over time is described in discrete
terms according to the year of construction steps shown in Table 2 (for
RC buildings we considered the design levels “absent”, “low” and “me­
dium”, respectively in these time range, 1919–1945, 1945–1970,
>1970). Fig. 11 shows the selected area (red boundary) and the Census
sections, subdivided per year of construction, while Table 4 reports the
total number of buildings, the percentages of buildings for construction
materials and for the year of construction. The total number of building
into the selected area is equal to 817.
Basing on this quite accurate characterization of information, raw
data (GeoJSON format) have been extracted and transferred as an input
for Street VULMA. According to the concepts shown in Section 3, about
20,000 photos have been downloaded and elaborated (one at time) by
domain experts, who performed the cropping of images, the required
labelling, according to the features in Table 1. After this step, all photos
have been stored through the Data VULMA service. The entire ensemble
amounted to about 2500 labelled photos, which represented the input of
Bi VULMA. As expected, for some labels, photos for RC and masonry
buildings present a usable classification, such as occurs for the number
of storeys. For other labels, only some of the photos displaying RC or
masonry buildings have provided a certain value for that feature, such as
occurs for pilotis floor (only for some RC buildings) or steel ties (only for
some masonry buildings). The consequence of this occurrence is that for
some labels the training dataset was unbalanced (the number of photos
without a certain feature is strongly greater than the number of photos
with that feature). Hence, before to employing the CNN of Bi VULMA,
the authors have balanced the number of photos in the dataset for each
label. For this scope, some photos were automatically downloaded and
manually selected from Google Images.
Training has been performed using transfer learning (see Section
3.3.1.6) mainly due to constrains related to the size of the available
dataset, which does not allow to train a neural network model from
scratch. For this study, MobileNetV2 [66] has been chosen as the base
network, with base parameters weighted on the ImageNet dataset. As for
the training algorithm, cross-entropy (either binary or categorical,
Reinforced Concrete
IV V VI VII
>1971 – – –
Absent Low Medium
0.667 0.519 0.434 0.364
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

Table 3
Set of parameters and related values associated to ΔVm [65].
Typology Masonry Reinforced Concrete

Class Description I - IV Description V VI VII

State of preservation Good − 0.04 Good – – –
Bad +0.04 Bad +0.04 +0.02 0
Number of floors − 0.08 Low (1–3) − 0.02 − 0.02 − 0.02
Low (1–2)
Medium (3–5) 0 Medium (4–7) 0 0 0
High (>5) High (>7) +0.08 +0.06 +0.04
Structural system Wall thickness
Wall distance +/− 0.04 -
Wall connection
Plan Irregularity Geometry +0.04 Geometry +0.04 +0.02 0
Mass distribution Mass distribution +0.02 +0.01 0
Vertical Irregularity Geometry +0.04 Geometry +0.04 +0.02 0
Mass distribution Mass distribution
Superelevation – +0.04 –
Floor
Roof details Weight, thrust connections +0.04 –
Retrofit intervention ±0.08 –
Aseismic Barbican, foil arches, buttresses − 0.04 –
Devices
Aggregate Building: position Middle − 0.04 0
Corner +0.04 Insufficient aseismic joints +0.04 0
Header +0.06
Aggregate Building: elevation
Staggered floors + 0.04 -
+/− 0.04
Buildings with different height
Foundation Beams − 0.04 0 0
Different level foundations +0.04 Connected beams 0 0 0
Isolated footing +0.04 0 0
Structural elements details Captive column +0.02 +0.01 0
Bow windows +0.04 +0.02 0

Fig. 9. Boundaries of the municipality of Bisceglie, Puglia, Southern Italy.

12
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

Fig. 10. CTR for the municipality of Bisceglie, Puglia, Southern Italy.

Fig. 11. Selected part of the municipality of Bisceglie and indication of Census sections, according to the year of construction.

13
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

Table 4 has been used, preventing the overall accuracy of the training to drop for
Number of buildings and percentages for the buildings of the selected area, more than a patience interval of three consecutive epochs. Finally, the
according to the parameters of the year of construction and construction training/test split of the dataset is in a standard 70/30 percentages.
materials. Once that the parameter is actually present in the picture of the
Parameter Number of Percentage of buildings building, it is identified by a single network, which has been specifically
buildings [%] trained to identify that parameter among the ones shown in Table 3. This
Construction typology RC 435 50.00 results in a set of 15 networks, which can be used as a cascade of models
Construction typology 351 43.00 to determine the overall characteristics of each building. The specific
Masonry choice of this setting (using a cascade of models instead of a single,
Construction typology Other 85 7.00
Year of construction <1919 228 27.91
comprehensive model able to determine each possible combination of
Year of construction 83 10.15 parameters) is mainly related to the small size of the dataset, which
1919–1945 would likely result in an inadequate number of samples for each class if
Year of construction 212 25.95 parameters were to be combined. The results in terms of accuracy on the
1945–1970
test set for some of the trained models are shown in Fig. 12 (indicated
Year of construction >1970 294 35.99
from a) to e) for the labels about structural typology, number of storeys,
irregularity in-plan, irregularity in-height and superelevation floor).
according to the specific number of classes involved in the sub-problem) As it can be seen, even with a small dataset, optimal results can be
has been used as the loss function, while ADAM [60] has been selected as easily achieved using transfer learning, as the accuracy is over 97% for
the optimization algorithm with a learning rate of 0.01. Early stopping each trained model.

Fig. 12. Accuracy after 10 epochs for (a) Structural typology; (b) number of storeys; (c) irregularity in-plan; (d) irregularity in-height; (e) superelevation floor.

14
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

In the end, we avoided applying the last step of VULMA to the same
photos used for the training of Bi VULMA, with the aim to validate the
procedure and to compute the index on different cases studies. Obvi­
ously, once that the ML algorithms are trained, it is possible to compute
for each photo the vulnerability index (according to In VULMA), which
can be employed in a fast vulnerability analysis at large-scale.
4. VULMA Validation: evaluation of the vulnerability index
Once that the ML models have been trained in Bi VULMA, a valida­
tion process has been developed. The aim of this step is twofold: (a) to
assess if Bi VULMA is able to recognize the right values of the labels
shown in Table 1 on a photo about a general building; (b) to evaluate if

Fig. 13. Case study buildings (B1 – B5) for VULMA validation.

15
S. Ruggieri et al. Automation in Construction 132 (2021) 103936

the vulnerability index given by In VULMA is comparable with the same Table 6
one manually computed by authors. To this aim scopes, 5 buildings have Estimated values of VIb, ΔVm and ṼI from the manual computation and In VULMA
been selected and subjected to Bi VULMA. These buildings, named B1- and absolute percentage differences for RC buildings B3, B4 and B5 (the symbol
B5, have been selected from the municipality of Panni, Puglia, South­ “/” indicates that VULMA does not provide that value).
ern Italy, 120 km far from the municipality of Bisceglie. This choice can Label B3 B4 B5
be motivated by the necessity to apply our tool to buildings character­ Manual In Manual In Manual In
ized by features different from the ones selected for the training. In VULMA VULMA VULMA
addition, for the selected buildings, the final value of the vulnerability VIb 0.434 0.434 0.434 0.434 0.434 0.434
index ṼI was already available and could be directly compared with the ΔVm - State of 0.00 / 0.00 / 0.00 /
one provided by In VULMA. preservation
Fig. 13 shows the photos of the 5 buildings B1-B5, where B1 and B2 ΔVm - Number − 0.02 − 0.02 − 0.02 − 0.02 − 0.02 − 0.02
of floors
are masonry building whereas B3-B4 and B5 are RC buildings. All the
ΔVm - 0.00 / 0.00 / 0.00 /
photos have been manually extracted by Google Street View and crop­ Structural
ped in order to be directly subjected to ML algorithms. system
More in detail, B1 is a masonry one-storey building, characterized by ΔVm - 0.00 0.00 0.00 0.00 0.00 0.00
a flat roof with no evident vaults or visible seismic details and dating Irregularity
in-plan
back before the 1919. Also B2 was built before 1919 and is a masonry ΔVm - 0.02 0.00 0.00 0.00 0.00 0.00
building presenting 2 storeys, vaults as detectable over the openings, a Irregularity
pitched roof and no seismic detail. B3 was built in the 60’s and it is the in-height
more peculiar building, since it is a 3-storey RC building with pitched ΔVm - 0.00 0.00 0.00 0.00 0.00 0.00
Aggregate
roof, a lateral part of different constitution (similar to a superelevation
buildings
floor but at an intermediate level), classifiable as irregular in-elevation ΔVm - 0.00 / 0.00 / 0.00 /
and located on a sloping basement. B4 and B5 are simple RC buildings Foundation
made in the 80’s and they present respectively 3 and 2 storeys, pitched ΔVm – 0.00 / 0.00 / 0.00 /
roof and structural regularity. In none of the case study buildings is Structural
elements
present a higher ground floor, while in the RC buildings are always details
visible overhangs. ṼI 0.434 0.414 0.414 0.414 0.414 0.414
From the application of Bi VULMA and In VULMA the vulnerability Absolute 4.60% 0.00% 0.00%
index ṼI has been computed and the results are reported in Table 5 and percentage
differences
Table 6, respectively for masonry (B1 and B2) and RC buildings (B3, B4
[%]
and B5), with the specification of the considered VIb and ΔVm and the
percentage differences (expressed in absolute value) between the
manual and the automatic procedures. From the obtained results it is the irregularity in height (in the higher part of the photo there is an out-
possible to make some observations about the ṼI computed by In of-focus zone).
VULMA. For the masonry buildings B1 and B2, the vulnerability index In addition, in order to apprise the effectiveness of the tool, it is
automatically obtained presents differences of, respectively, of about 10 possible to estimate, for each building, the deviation from the set of
and 20% with respect to those manually computed. These differences parameters (VIb and ΔVm) constituting the automatically computed ṼI
can be addressed to some parameters that have been not considered in (assumed as the predicted values) to the set of parameters (VIb and ΔVm)
the automatic index, such as the state of preservation and the informa­ constituting the manually computed ṼI (assumed as the observed
tion about structural system (e.g., wall thickness, wall distance and wall values). To this scope, a common metric, such as the root mean square
connection), which actually can be detected only through an on-site error (RMSE), can be employed and the results are following reported:
survey. Regarding to RC buildings B3, B4 and B5, the values of the 0.0163 for B1 and B2, 0.0067 for B3 and 0 for B4 and B5. These values
vulnerability indexes are almost completely the same, with a unique suggest a certain degree of effectiveness for the tool and, in this case, a
difference of about 5% for B3, which is due to the wrong identification of reliable prediction of the desired values by Bi VULMA and In VULMA
modules.
Another aspect to clarify is the principle of operation of In VULMA.
Table 5 To compute the final index, the tool considers all the accounted features
Estimated values of VIb, ΔVm and ṼI from the manual computation and In VULMA in Bi VULMA and then, in the automatic computation of ṼI a specific
and absolute percentage differences for masonry buildings B1 and B2 (the
feature cannot be currently excluded as desired by user. This is due to
symbol “/” indicates that VULMA does not provide that value).
the functioning principle of VULMA, where classifiers are based on a
Label B1 B2 cascade of neural networks, each one trained for a certain feature in the
Manual In Manual In current version. Then, using the trained classifiers, the value of each
VULMA VULMA label is automatically assigned by the network itself and the index will
VIb 0.689 0.689 0.704 0.704 always take into account the considered features. This could be a
ΔVm - State of preservation +0.04 / +0.04 / disadvantage for In VULMA in the case of ambiguous values of some
ΔVm - Number of floors − 0.04 − 0.04 − 0.04 − 0.04 labels. On the other hand, considering that the validation accuracy on
ΔVm - Structural system / /
each trained neural network has been estimated to be above 97%, the
+0.04 +0.04
ΔVm - Irregularity in-plan 0.00 0.00 0.00 0.00
ΔVm - Irregularity in-height 0.00 0.00 0.00 0.00 effect on the result could be quantified as a slight variation of the final
ΔVm - Superelevation floor 0.00 0.00 0.00 0.00 value of ṼI, never provoking strongly incorrect estimates.
ΔVm - Roof floor 0.00 0.00 0.00 0.00 The obtained results suggest that the proposed tool is extremely
ΔVm - Retrofit intervention 0.00 / 0.00 /
efficient for RC buildings, while some additional operations are neces­
ΔVm - Aseismic details +0.08 +0.08 +0.08 +0.08
ΔVm - Aggregate buildings 0.00 0.00 0.00 0.00 sary for masonry buildings. Also for building B3 some differences occur,
ΔVm - Foundation 0.00 / 0.00 / considering that the bad quality of the image and the complex evolution
ṼI 0.809 0.729 0.824 0.664 of the building are still open issues for Bi VULMA. Nevertheless, as just
Absolute percentage differences 9.89% 19.42% mentioned above, this is an unborn version of In VULMA, which will be
[%]
subjected to future developments, while Bi VULMA is just highly

16
S. Ruggieri et al.
efficient. Lastly, as for the indexes computed by In VULMA, one should
consider that this approach is currently limited by visually available
information and details such as foundation or wall thickness have not
been considered in this application.
5. Conclusions and future developments
This study presents a new machine-learning based tool, named
VULMA (VULnerability analysis using MAchine-learning), able to capture
the key features of buildings in the existing stock starting from a simple
photograph of a building. The general framework, presented according
to a processing pipeline, consists in four independent modules, which
can be separately used for different aims. In particular, VULMA is
composed by: (1) Street VULMA, which processes raw data to extract
photos of buildings from a desired area; (2) Data VULMA, which allows
domain expert to store photos of buildings once that they have been
labelled; (3) Bi VULMA, which is composed by a set of machine-learning
algorithms (based on convolution neural networks) that, once trained,
are able to capture the labelled features of buildings and (4) In VULMA,
which provides a simple vulnerability index for a single building.
After a detailed description of each phase, the steps of the general
tool have been applied on the municipality of Bisceglie, Southern Italy,
to provide a trained version of Bi VULMA and In VULMA that in turn
have been validated on 5 case study buildings belonging to another
municipality. The results of the application show that, despite the
vulnerability index employed is very simple, the tool provide reliable
estimates.
For concluding, we wish to highlight the new scenarios that a tool
like VULMA can provide in the existing fast vulnerability assessment
procedures to develop for risk mitigation strategies:
1. VULMA provides a method that reduces the time effort related to
surveys and interviews, usually involved in the fast vulnerability
assessment procedures (e.g., Cartis). As a matter of fact, it allows to
define a vulnerability index only by identifying buildings in an area
and by using their photos;
2. VULMA provides a supervised classification of images and, if
continuously upgraded, can reduce the usual bias introduced in the
phase of subjective evaluation of the building features by the judg­
mental assignments of domain experts;
3. VULMA can be easily integrated with other data, freely online
available, such as regional technical maps and Census Data. The
processes of data integration can improve the actual state of the tool
and can represent the base for further improvements in the analytical
vulnerability estimates;
4. VULMA structure is conceived in order to be continuously evolved in
each of the proposed steps. In particular, especially for In VULMA,
the identify parameters can be processed in analytical/mechanical
methods, with the aim of computing fragility and vulnerability
curves for the single photos or for the entire class;
5. VULMA can assume the role of Decision Support System, as an
automatic support to domain experts in large-scale vulnerability
analysis.
Obviously, some improvements are required for each module.
Among the possible developments for In VULMA, we have to consider
that the actual state of the involved labelling, even with the double-
check procedure described in Section 3, could present a subjective
bias that may be still an open issue. In addition, also the photos may be
automatically elaborated, avoiding the direct operations made by
domain experts. In the end, VULMA can be also shared with the scientific
community, in order to increase the actual datasets and improve the
power of the tool.
The training procedure itself could be greatly improved by focusing
on mainly three aspects: first, improving the overall data quantity, by
gathering and labelling an adequate and balanced amount of samples for
17
Automation in Construction 132 (2021) 103936
each parameter. Second, training can be evaluated using also other
metrics, and both fine tuning and hyper-parameter optimization can be
used to improve classification accuracy. Finally, index computation by
In VULMA could be further refined and extended by embedding
contextual information on the building, which cannot be easily extrac­
ted by a visual inspection (such as information on foundation).
Declaration of Competing Interest
The authors whose names are listed immediately below certify that
they have NO affiliations with or involvement in any organization or
entity with any financial interest (such as honoraria; educational grants;
participation in speakers’ bureaus; membership, employment, consul­
tancies, stock ownership, or other equity interest; and expert testimony
or patent-licensing arrangements), or non-financial interest (such as
personal or professional relationships, affiliations, knowledge or beliefs)
in the subject matter or materials discussed in this manuscript.
References
[1] V. Silva, S. Akkar, J. Baker, P. Bazzurro, J.M. Castro, H. Crowley, M. Dolsek,
C. Galasso, S. Lagomarsino, R. Monteiro, D. Perrone, K. Pitilakis, D. Vamvatsikos,
Current challenges and future trends in analytical fragility and vulnerability
modeling, Earthquake Spectra 35 (4) (2019) 1927–1952, https://doi.org/10.1193/
042418EQS101O.
[2] C. Del Gaudio, M. Di Ludovico, M. Polese, G. Manfredi, A. Prota, P. Ricci, G.
M. Verderame, Seismic fragility for Italian RC buildings based on damage data of
the last 50 years, Bull. Earthq. Eng. 18 (5) (2020) 2023–2059, https://doi.org/
10.1007/s10518-019-00762-6.
[3] M. Dolce, A. Goretti, Building damage assessment after the 2009 Abruzzi
earthquake, Bull. Earthq. Eng. 13 (8) (2015) 2241–2264, https://doi.org/10.1007/
s10518-015-9723-4.
[4] G. Zuccaro, F.L. Perelli, D. De Gregorio, F. Cacace, Empirical vulnerability curves
for Italian mansory buildings: evolution of vulnerability model from the DPM to
curves as a function of acceleration, Bull. Earthq. Eng. 19 (8) (2021) 3077–3097,
https://doi.org/10.1007/s10518-020-00954-5.
[5] A. Rosti, C. Del Gaudio, M. Rota, P. Ricci, M. Di Ludovico, A. Penna, G.
M. Verderame, Empirical fragility curves for Italian residential RC buildings, Bull.
Earthq. Eng. 19 (8) (2021) 3165–3183, https://doi.org/10.1007/s10518-020-
00971-4.
[6] S. Ruggieri, C. Tosto, G. Rosati, G. Uva, G.A. Ferro, Seismic Vulnerability Analysis
of Masonry Churches in Piemonte after 2003 Valle Scrivia earthquake: post-event
screening and situation 17 years later, Int. J. Archit. Heritage 8 (2020), https://doi.
org/10.1080/15583058.2020.1841366.
[7] G. De Matteis, G. Brando, V. Corlito, Predictive model for seismic vulnerability
assessment of churches based on the 2009 L’Aquila earthquake, Bull. Earthq. Eng.
17 (9) (2019) 4909–4936, https://doi.org/10.1007/s10518-019-00656-7.
[8] V. Silva, H. Crowley, H. Varum, R. Pinho, L. Sousa, Investigation of the
characteristics of Portuguese regular moment-frame RC buildings and development
of a vulnerability model, Bull. Earthq. Eng. 13 (5) (2015) 1455–1490, https://doi.
org/10.1007/s10518-014-9669-y.
[9] B. Borzi, R. Pinho, H. Crowley, Simplified pushover-based vulnerability analysis for
large-scale assessment of RC buildings, Eng. Struct. 30 (3) (2008) 804–820,
https://doi.org/10.1016/j.engstruct.2007.05.021.
[10] S. Ruggieri, F. Porco, G. Uva, D. Vamvatsikos, Two frugal options to assess class
fragility and seismic safety for low-rise reinforced concrete school buildings in
southern Italy, Bull. Earthq. Eng. 19 (2021) 1415–1439, https://doi.org/10.1007/
s10518-020-01033-5.
[11] B. Borzi, H. Crowley, R. Pinho, Simplified pushover-based earthquake loss
assessment (SP-BELA) method for masonry buildings, Int. J. Archit. Heritage 2 (4)
(2008) 353–376, https://doi.org/10.1080/15583050701828178.
[12] S. Ruggieri, D. Perrone, M. Leone, G. Uva, M.A. Aiello, A prioritization RVS
methodology for the seismic risk assessment of RC school buildings, Int. J. Dis. Risk
Reduct. 51 (2020) 101807, https://doi.org/10.1016/j.ijdrr.2020.101807.
[13] D. Perrone, M.A. Aiello, M. Pecce, F. Rossi, Rapid visual screening for seismic
evaluation of RC hospital buildings, Structures 3 (2015) 57–70, https://doi.org/
10.1016/j.istruc.2015.03.002.
[14] M. Polese, M.G. d’Aragona, A. Prota, Simplified approach for building inventory
and seismic damage assessment at the territorial scale: an application for a town in
southern Italy, Soil Dyn. Earthq. Eng. 121 (2019) 405–420, https://doi.org/
10.1016/j.soildyn.2019.03.028.
[15] M. Dolce, E. Speranza, F. Giordano, B. Borzi, F. Bocchi, C. Conte, A. Di Meo,
M. Faravelli, V. Pascale, Observed damage database of past Italian earthquakes: the
Da DO WebGIS, Boll. Geofis. Teor. Appl. 60 (2) (2019) 141–164, https://doi.org/
10.4430/bgta0254.
[16] D.H. Lang, A. Kumar, S. Sulaymanov, A. Meslem, Building typology classification
and earthquake vulnerability scale of central and south Asian building stock,
J. Build. Eng. 15 (2018) 261–277, https://doi.org/10.1016/j.jobe.2017.11.022.
[17] G.M. Calvi, R. Pinho, G. Magenes, J.J. Bommer, L.F. Restrepo-Velez, H. Crowley,
Development of seismic vulnerability assessment methodologies over the past 30
S. Ruggieri et al.
years, ISET J. Earthq. Technol. 43 (3) (2006) 75–104. https://pdfs.semanticsch
olar.org/ff47/e72c838a6cba9227fa54d2b622da160b295e.pdf.
[18] S. Brzev, C. Scawthorn, A.W. Charleson, L. Allen, M. Greene, K. Jaiswal, V. Silva,
GEM Building Taxonomy Version 2.0, Technical Report, 2013, pp. 1–188. https://
issuu.com/gem_wrld/docs/exp-mod-gem-building-taxonomy-20130 (accessed
March 15, 2021).
[19] S.W. Medvedev, W. Sponheuer, V. Karnik, Seismic Intensity Scale Version MSK
1964, United Nation Educational, Scientific and Cultural Organization, Paris, 1965,
p. 7, in: https://academic.microsoft.com/journal/93825082/publication/search?
q=Proceedings%20of%20the%20USSR%20Academy%20of%20Sciences&qe=And
(Composite(J.JId%253D93825082)%252CTy%253D%270%27)&f=&orderBy=0.
[20] G. Grünthal, European Macroseismic Scale, Luxembourg: Cahiers du Center
Europèen de Gèodynamique et de Seismologie, Conseil de l’Europe 1998 (1998)
99. http://media.gfz-potsdam.de/gfz/sec26/resources/documents/PDF
/EMS-98_Original_englisch.pdf.
[21] FEMA, HAZUS99 Technical Manual. Service Release 2, Federal Emergency
Management Agency, Washington, D.C., USA, 2001, p. 42. https://nehrpsearch.
nist.gov/static/files/FEMA/PB2008108263.pdf.
[22] P. Mouroux, E. Bertrand, M. Bour, B. Le Brun, S. Depinois, P. Masure, The European
RISK-UE project: an advanced approach to earthquake risk scenarios, in:
Proceedings of the 13th World Conference on Earthquake Engineering, 2004,
p. 3329. https://www.iitk.ac.in/nicee/wcee/article/13_3329.pdf.
[23] S. Lagomarsino, S. Giovinazzi, Macroseismic and mechanical models for the
vulnerability and damage assessment of current buildings, Bull. Earthq. Eng. 4 (4)
(2006) 415–443, https://doi.org/10.1007/s10518-006-9024-z.
[24] K. Pitilakis, P. Franchin, B. Khazai, H. Wenzel, SYNER-G: Systemic Seismic
Vulnerability and Risk Assessment of Complex Urban, Utility, Lifeline Systems and
Critical Facilities: Methodology and Applications, 2014, p. 184. https://www.sprin
ger.com/gp/book/9789401788342 (978–94–017–8835-9).
[25] GNDT, Gruppo Nazionale Per la Difesa dai Terremoti, Rischio sismico di edifici
pubblici, 1993, pp. 1–134. https://emidius.mi.ingv.it/GNDT2/Pubblicazioni/Bib
lioteca/Risk_ed_pubbl/rischio_sismico_di_edifici_pubblici_parteI.pdf (accessed April
10, 2021).
[26] G. Uva, C.A. Sanjust, S. Casolo, M. Mezzina, ANTAEUS project for the regional
vulnerability assessment of the current building stock in historical centers, Int. J.
Archit. Heritage 10 (1) (2016) 20–43, https://doi.org/10.1080/
15583058.2014.935983.
[27] S. Cara, A. Aprile, L. Pelà, P. Roca, Seismic Risk Assessment and Mitigation at
Emergency Limit Condition of Historical Buildings along Strategic Urban
Roadways. Application to the “Antiga Esquerra de L’Eixample” Neighborhood of
Barcelona, Int. J. Archit. Heritage 12 (7–8) (2018) 1055–1075, https://doi.org/
10.1080/15583058.2018.1503376.
[28] D. Benedetti, V. Petrini, On the Seismic Vulnerability of Masonry Buildings: A
Method of Evaluation, L’industria delle Costruzioni, 1984, pp. 66–74 (In Italian),
http://www.lindustriadellecostruzioni.it/.
[29] G. Zuccaro, M. Dolce, D. De Gregorio, E. Speranza, C. Moroni, La Scheda Cartis Per
La Caratterizzazione Tipologico - Strutturale Dei Comparti Urbani Costituiti Da
Edifici Ordinari. Valutazione dell’esposizione in analisi di rischio sismico, 34
Convegno Nazionale GNGTS, Trieste, 2015, pp. 281–287. http://www3.ogs.trieste.
it/gngts/files/2015/S23/Riassunti/Zuccaro.pdf.
[30] H. Miura, S. Midorikawa, Updating GIS building inventory data using high-
resolution satellite images for earthquake damage assessment: application to metro
Manila, Philippines, Earthquake Spectra 22 (1) (2006) 151–168, https://doi.org/
10.1193/1.2162940.
[31] M. Indirli, Organization of a geographic information system (GIS) database on
natural hazards and structural vulnerability for the historic center of san Giuliano
di Puglia (Italy) and the city of Valparaiso (Chile), Int. J. Archit. Heritage 3 (4)
(2009) 276–315, https://doi.org/10.1080/15583050902803780.
[32] E.J. Kim, G. Plessis, J.L. Hubert, J.J. Roux, Urban energy simulation: simplification
and reduction of building envelope models, Ener. Build. 84 (2014) 193–202,
https://doi.org/10.1016/j.enbuild.2014.07.066.
[33] F. Braga, M. Dolce, D. Liberatore, A Statistical Study on Damaged Buildings and an
Ensuing Review of the MSK-76 Scale, Seventh European Conference on Earthquake
Engineering, Athens, Greece, 1982, pp. 431–450. http://hdl.handle.net/11573/45
9303.
[34] M. Dolce, A. Masi, M. Marino, M. Vona, Earthquake damage scenarios of the
building stock of Potenza (southern Italy) including site effects, Bull. Earthq. Eng. 1
(1) (2003) 115–140, https://doi.org/10.1023/A:1024809511362.
[35] I. Riedel, P. Guéguen, M. Dalla Mura, E. Pathier, T. Leduc, J. Chanussot, Seismic
vulnerability assessment of urban environments in moderate-to-low seismic hazard
regions using association rule learning and support vector machine methods, Nat.
Hazards 76 (2) (2015) 1111–1141, https://doi.org/10.1007/s11069-014-1538-0.
[36] I. Riedel, P. Gueguen, F. Dunand, S. Cottaz, Macroscale vulnerability assessment of
cities using association rule learning, Seismol. Res. Lett. 85 (2) (2014) 295–305,
https://doi.org/10.1785/0220130148.
[37] Y. Xie, M. Ebad Sichani, J.E. Padgett, R. DesRoches, The promise of implementing
machine learning in earthquake engineering: a state-of-the-art review, Earthquake
Spectra 36 (4) (2020) 1769–1801, https://doi.org/10.1177/8755293020919419.
[38] H. Sun, H.V. Burton, H. Huang, Machine learning applications for building
structural design and performance assessment: state-of-the-art review, J. Build.
Eng. 33 (2020) 101816, https://doi.org/10.1016/j.jobe.2020.101816.
[39] R. Falcone, C. Lima, E. Martinelli, Soft computing techniques in structural and
earthquake engineering: a literature review, Eng. Struct. 207 (2020) 110269,
https://doi.org/10.1016/j.engstruct.2020.110269.
18
Automation in Construction 132 (2021) 103936
[40] S. Visa, A. Ralescu, Issues in Mining Imbalanced Data Sets-a Review Paper, Sixteen
Midwest Artificial Intelligence and Cognitive Science Conference 2005, 2005,
pp. 67–73. https://eecs.ceas.uc.edu/~ralescal/PAPERS/VRMaics2005.pdf.
[41] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a
simple way to prevent neural networks from overfitting, J. Mach. Learning Res. 15
(1) (2014) 1929–1958. http://jmlr.org/papers/v15/srivastava14a.html.
[42] E. Harirchian, K. Jadhav, V. Kumari, T. Lahmer, ML-EHSAPP: a prototype for
machine learning-based earthquake hazard safety assessment of structures by using
a smartphone app, Eur. J. Environ. Civ. Eng. (2021) 1–21, https://doi.org/
10.1080/19648189.2021.1892829.
[43] S. Mangalathu, H. Sun, C.C. Nweke, Z. Yi, H.V. Burton, Classifying earthquake
damage to buildings using machine learning, Earthquake Spectra 36 (1) (2020)
183–208, https://doi.org/10.1177/8755293019878137.
[44] S. Mangalathu, S.H. Hwang, E. Choi, J.S. Jeon, Rapid seismic damage evaluation of
bridge portfolios using machine learning techniques, Eng. Struct. 201 (2019)
109785, https://doi.org/10.1016/j.engstruct.2019.109785.
[45] Python Software Foundation, Python: A Dynamic, Open Source Programming
Language, Python Software Foundation, 2015. https://www.python.org/psf/.
[46] H. Jiang, O. Nachum, Identifying and correcting label bias in machine learning, in:
International Conference on Artificial Intelligence and Statistics, Proceedings of the
Twenty Third International Conference on Artificial Intelligence and Statistics,
PMLR 108, 2020, pp. 702–712, in: http://proceedings.mlr.press/v108/jiang20a.
html.
[47] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep
convolutional neural networks, Adv. Neural Inf. Proces. Syst. 25 (2012)
1097–1105, in: https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6
b76c8436e924a68c45b-Paper.pdf (ISSN: 10495258).
[48] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale
hierarchical image database, in: 2009 IEEE Conference on Computer Vision and
Pattern Recognition, 2009, pp. 248–255, https://doi.org/10.1109/
CVPR.2009.5206848.
[49] I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning Vol. 1, MIT press,
Cambridge, 2016, p. 800. No. 2, https://books.google.it/books?hl=it&lr
=&id=omivDQAAQBAJ&oi=fnd&pg=PR5&dq=Goodfellow,++Deep+learning+
isbn&ots=MNO4iskBOZ&sig=VQxkT4Fwg069MBKQyB0hGB_FLlw#v=onepage
&q=Goodfellow%2C%20%20Deep%20learning%20isbn&f=false (0262035618).
[50] J. Han, C. Moraga, The influence of the sigmoid function parameters on the speed
of backpropagation learning, in: International Workshop on Artificial Neural
Networks, 1995, pp. 195–201, https://doi.org/10.1007/3–540-59497-3_175.
[51] K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual networks, in:
European Conference on Computer Vision, 2016, pp. 630–645. https://arxiv.
org/abs/1603.05027v3.
[52] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception
architecture for computer vision, in: IEEE Conference on Computer Vision and
Pattern Recognition, 2016, pp. 2818–2826. https://arxiv.org/abs/1512.00567.
[53] D.H. Hubel, T.N. Wiesel, Receptive fields, binocular interaction and functional
architecture in the cat’s visual cortex, J. Physiol. 160 (1) (1962) 106–154, https://
doi.org/10.1113/jphysiol.1962.sp006837.
[54] C. Shorten, T.M. Khoshgoftaar, A survey on image data augmentation for deep
learning, J. Big Data 6 (1) (2019) 1–48, https://doi.org/10.1186/s40537-019-
0197-0.
[55] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, H. Lipson, Understanding neural
networks through deep visualization, in: ICML Deep Learning Workshop, 2015,
p. 12. https://arxiv.org/abs/1506.06579.
[56] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, X. Zheng, Tensorflow: A
system for large-scale machine learning, in: 12th Symposium on Operating Systems
Design and Implementation (OSDI 16), 2016, pp. 265–283. https://arxiv.
org/abs/1605.08695.
[57] F. Chollet, Keras, GitHub. https://github.com/fchollet/keras, 2015.
[58] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,
M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos,
D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: machine
learning in Python, J. Mach. Learning Res. 12 (2011) 2825–2830. https://www.
jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf?source=post_page.
[59] S.I. Amari, Backpropagation and stochastic gradient descent method,
Neurocomputing 5 (4–5) (1993) 185–196, https://doi.org/10.1016/0925-2312
(93)90006-O.
[60] D.P. Kingma, J. Ba, Adam: A Method for Stochastic Optimization, 3rd International
Conference for Learning Representations, San Diego. https://arxiv.org/abs/
1412.6980, 2015.
[61] W.S. Sarle, Stopped training and other remedies for overfitting, in: Proceedings of
the 27th Symposium on the Interface of Computing Science and Statistics, 1995,
pp. 1–10. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.3920.
[62] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.C. Chen, Mobilenetv2: Inverted
residuals and linear bottlenecks, in: IEEE Conference on Computer Vision and
Pattern Recognition, 2018, pp. 4510–4520. https://openaccess.thecvf.com/cont
ent_cvpr_2018/html/Sandler_MobileNetV2_Inverted_Residuals_CVPR_2018_paper.
html.
[63] F. Chollet, Xception: Deep learning with depthwise separable convolutions, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
2017, pp. 1251–1258. https://arxiv.org/abs/1610.02357.
[64] B. Zoph, V. Vasudevan, J. Shlens, Q.V. Le, Learning transferable architectures for
scalable image recognition, in: Proceedings of the IEEE Conference on Computer
S. Ruggieri et al.
Vision and Pattern Recognition, 2018, pp. 8697–8710. https://arxiv.org/abs/1
707.07012.
[65] L. Frassine, S. Giovinazzi, Basi di dati a confronto nell’analisi di vulnerabilità
sismica dell’edilizia residenziale: un’applicazione per la città di Catania, XI
Congresso Nazionale “L’ingegneria Sismica in Italia”, Genova, 2004, p. 16 (in
Italian), https://www.anidis.it/convegni-anidis/2004-genova/.
19
Automation in Construction 132 (2021) 103936
[66] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.C. Chen, Mobilenetv2: Inverted
residuals and linear bottlenecks, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2018, pp. 4510–4520. https://arxiv.
org/abs/1801.04381.