International Journal of Architectural Heritage

Conservation, Analysis, and Restoration

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/uarc20

Creating Historical Building Models by Deep

Fusion of Multi-Source Heterogeneous Data Using
Residual 3D Convolutional Neural Network

Wenfa Hu & Ruiqi Hu

To cite this article: Wenfa Hu & Ruiqi Hu (26 Jun 2023): Creating Historical Building Models
by Deep Fusion of Multi-Source Heterogeneous Data Using Residual 3D Convolutional Neural
Network, International Journal of Architectural Heritage, DOI: 10.1080/15583058.2023.2229253

To link to this article: https://doi.org/10.1080/15583058.2023.2229253

Published online: 26 Jun 2023.

Submit your article to this journal

Article views: 113

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=uarc20
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE
https://doi.org/10.1080/15583058.2023.2229253

Creating Historical Building Models by Deep Fusion of Multi-Source

Heterogeneous Data Using Residual 3D Convolutional Neural Network
Wenfa Hua and Ruiqi Hub
a
School of Economics and Management, Tongji University, Shanghai, China; bManning College of Information and Computer Sciences,
University of Massachusetts Amherst, Amherst, USA

ABSTRACT ARTICLE HISTORY

There are a large number of historical buildings in the world, but many of them are seriously Received 5 December 2021
damaged. A fundamental task to revitalize them is to record their details and redraw them, but it is Accepted 20 June 2023
a highly inefficient task. This paper aimed to create three-dimensional (3D) models automatically KEYWORDS
for those damaged historical buildings while respecting the 1964 Venice Charter, the ICOMOS Convolutional neural
recommendations, and local ordinances. After reviewing the advantages and disadvantages of network; deep fusion;
various measurement tools, we implemented a machine learning algorithm, the residual 3D heterogeneous data;
convolutional neural network (CNN) method, to combine heterogeneous data from multiple historical building; historical
sources including a 3D laser scanner, an unmanned aerial vehicle, a total station, and a camera, building information model
in a historical building information model (HBIM) to increase its accuracy. Those data, preprocessed
by four steps including data registration, reducing noises, filtering data, and data integration, were
trained and fused in a 3D CNN model, which consisted of one input layer, five 3D convolutional and
pooling layers, and two connected layers. A Shanghai historical building conservation project was
taken as a pilot case to verify its productivity and accuracy in developing the HBIM, on which the
architectural features were thoroughly recorded, and the damaged details were recovered.

1. Introduction traditional reverse engineering method for historical

buildings, e.g. building information modeling (BIM)
Historic buildings, built decades or hundreds of years
(Bastem and Cekmis 2022), depends on a large number
ago, are important assets of their cities and states,
of field investigations, measurements, and surveys, and
attracting tourists and adding cultural perspectives and
this method is time-consuming (Clarke and Laefer
other qualities to their communities. They are perma­
2014), expensive (Milic et al. 2019), inaccurate (Wang
nent symbols of their history and culture, providing
et al. 2020), and inefficient.
tangible reminders of the past (Walter 2014). But
Modern technology provides us with some solutions
many historical buildings are at risk of collapse because
to those problems. Three-dimensional (3D) laser scan­
of inevitable corrosion and inadequate maintenance,
ning technology, as a new surveying and mapping tech­
their structural materials are being broken (Ozmen,
nology (Jaafar et al. 2017), is widely used in creating 3D
Akan, and Unay 2011; Wang et al. 2019), and their
models of historical buildings because of its advantages
facilities are worn out (Chen, Chiu, and Tsai 2018).
of high efficiency, precision, and integration. However,
The 1964 Venice Charter, the ICOMOS recommenda­
practices indicate some shortcomings, e.g. the poor
tions, and local ordinances require that their original
quality of point cloud in the upper part of a high-rise
and historical features would be well-conserved and
building due to scanning distance or scanning angle, or
restored. Therefore, their details should be carefully
blind areas or blocks of other objects when scanning the
recorded before the construction and the damaged fea­
building top. An unmanned aerial vehicle (UAV) can fly
tures would be completely restored after the construc­
high to avoid those blocks, carrying multi-lens sensors
tion (Collepardi 1999). Since local ordinances require all
(Templin and Popielarczyk 2020). With the assistance of
rebuilding proposals to follow the latest building codes
tilt photogrammetry technology, a UAV can survey and
which will potentially ruin the conservation of the his­
map the building continuously from various angles so
torical features, their construction proposals must be
that the point cloud data with the spatial and color
carefully evaluated before the construction. One obsta­
information in building appearances are collected
cle to restoring historic buildings is the lack of architec­
through a large number of image-matching calculations.
tural documents and drawings (Karimi et al. 2022). The

CONTACT Wenfa Hu wenfahu@tongji.edu.cn

© 2023 Taylor & Francis
2 W. HU AND R. HU
Then, a 3D model can be created and the model can be
refined with more detailed information if the UAV takes
more images from the top of the building (Lim et al.
2022), which is a fast reverse high-precision modeling
technology for historical buildings. Due to the flight
altitude control of a commercial UAV, the angle limita­
tion of the airborne camera lens on the UAV, and the
blocks of ground trees or other buildings, there are still
many errors or mistakes in the 3D models for
a historical building, e.g. high quality in the top of the
building model but poor quality in its bottom when
a tree sticks to the building bottom or flight paths are
blocked by other objects. As an electronic theodolite, the
total station is generally used in surveying and building
construction for measuring angles and distances of con­
trol points quickly and accurately (Alizadeh-Khameneh
et al. 2018), but a normal total station can not record the
rich details of a historical building. Though an advanced
total station can depict a pictorial view of the historical
building from the ground standpoint, it is more expen­
sive and hardly takes detailed pictures of the building’s
upper parts.
None of those modern technologies is perfect to
survey or measure a historical building. Since any of
them has advantages over others, they are complemen­
tary to the measurements of historical buildings by
enhancing the surveying accuracy, complementing the
missed data, and recovering the damaged details. But
how to create a 3D building model by fusion of those
heterogeneous data is a technical challenge (Patrucco
et al. 2022). This paper presents a deep fusion frame­
work based on a residual 3D convolutional neural net­
work (CNN) algorithm (Zou et al. 2019) to develop a 3D
building model, and then a historical building informa­
tion modeling (HBIM) platform can be further intro­
duced to refine 3D building models.
The remainder of this paper is organized as follows:
Section 2 is a review of the literature on the reconstruction
of a 3D building model, including HBIM, the fusion of
multiple data, and the development of a 3D CNN algo­
rithm. Section 3 presents a framework for creating 3D
building models and the main data sources. The process,
algorithm, and formation of multiple data fusions based
on 3D CNN are presented in Section 4. Section 5 illus­
trates a pilot project of historical buildings in Shanghai,
and Section 6 is the conclusion and future studies.
2. Literature review
2.1. 3D modeling methods for historical buildings
The HBIM, as an extension of BIM, is a new method to
utilize remote sensing data to reconstruct 3D models of
historical buildings (Bruno and Roncella 2019; Bruno,
De Fino, and Fatiguso 2018). Sztwiertnia et al. (2021)
claimed that HBIM was a system combining BIM and
other modeling technology like laser scanning and
photogrammetry. Nieto-Julian, Anton, and Jose
Moyano (2020) illustrated how the HBIM process was
applied in a reverse-engineering solution. One key pro­
blem in developing HBIM is the mapping of parameter­
ized objects of building elements onto laser scanning or
photogrammetric data (Jordan-Palomar et al. 2018).
Yang et al. (2020) proposed that there were two main
techniques of photogrammetry and laser-scanning to
measure memorial buildings, but they were difficult to
be measured because of their valuable characteristics. Pu
and Vosselman (2009) also mentioned that ground laser
scanning was an important data source for the auto­
matic reconstruction of building models. Lee et al.
(2019) proposed a building modeling method based on
the information extracted from light detection and ran­
ging (LiDAR), and Wu et al. (2018) proposed to do it on
aerial images. Yang, Sheng, and Wang (2016) proposed
to create a 3D building facade with fused data of terres­
trial LiDAR data and optical images.
To create building models efficiently, Yang et al.
(2019) developed an HBIM system to reconstruct his­
torical building models with data from a terrestrial laser
scanner and digital cameras. Cornelis et al. (2008) cre­
ated 3D solid models and 3D/2D surface models for an
existing object after concerning its methods of structure
and construction in an HBIM system, presenting
a procedure to automatically generate 3D models of
historical buildings with multi-spectral texture informa­
tion by a 2D range camera. Leonov et al. (2015) pro­
posed to combine the two-dimensional geographic
information system (GIS) and a three-dimensional
model in HBIM for the preservation of architectural
heritage. Jo and Hong (2019) demonstrated that a three-
dimensional solution, based on a simplified parametric
model, was suitable for industrial elements and build­
ings and could be used for the conservation and man­
agement of heritage documents.
Previous literature reveals that the methods of creat­
ing 3D historical building models are time-consuming
and inefficient, therefore an automatic 3D modeling
approach would be more productive for repairing and
upgrading a historical building.
2.2. Heterogeneous data from multi-sources in
surveying historical buildings
Although 3D laser scanners have many advantages,
Cheng et al. (2013) claimed that they might not be
suitable for rebuilding an entire castle since they were

not portable or fast enough and the vast amounts of data
they generated were difficult to be recorded and visua­
lized. To avoid those problems, Pesci, Teza, and Bonali
(2011) combined aerial and ground images to build
a building model based on multi-source data using
existing floor plans, assisted by limited measurement
data. Franceschi et al. (2015) proposed to combine
LiDAR and photogrammetric data to investigate the
extensional synsedimentary structures of the Early
Jurassic in the Southern Alps of Italy, where airborne
LiDAR data helped to obtain geological information of
vegetation-covered areas and photogrammetry was
applied to generate high-resolution 3D texture models
caused by inaccessible parts of airborne LiDAR. Cheng
et al. (2011) developed a parametric object prototype
library based on historical building data in a cross-
platform. Andriasyan et al. (2020) developed an auto­
matic modeling system for historical buildings using
point clouds and image measurement data. The HBIM
process starts with the remote collection of measure­
ment data using a ground laser scanner combined with
digital photos and then designs and constructs an object
parameterization library.
Martinez-Carricondo et al. (2021) addressed that the
combination of HBIM and UAV photogrammetry was
more efficient for modeling and documentation of for­
gotten heritage than traditional drawing methods. As
a new innovative technology, UAV provides an eco­
nomic, reliable, and direct method to capture building
data effectively and sustainably (Zaragoza et al. 2017).
To create models of building tops and sides, many
researchers suggest the fusion technology of multi-
source data, especially using vehicle and airline LiDAR
data. Rao and Liu (2020) proposed combining 3D laser
technology with close-range photogrammetry to create
a 3D model of the basement and murals of Santa
Cristina, Italy. Martinez et al. (2012) utilized feature
recognition to extract building elevation information,
but a large number of manual interventions were still
required in completing a 3D building model.
The data from multi-sources, e.g. a laser scanner,
LiDAR, camera, photos, and surveyors, in upgrading
a historical building is heterogeneous, but the combina­
tion of those data is an inefficient and error-prone job.
Those heterogeneous data should be processed to gen­
erate 3D building models in a more efficient method.
2.3. Machine learning and deep learning for
modeling historical buildings
Machine learning, a subclass of artificial intelligence,
is self-learning based on data. The machine learning
algorithm is divided into three categories which are
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 3
supervised, unsupervised, and reinforcement learn­
ing, and they are applied widely in detection, predic­
tion, and generation. As a subcategory of machine
learning, deep learning is a neural network with neu­
rons having several layers between input and output,
robustly providing automatic learning of features.
Machine learning provides a practical platform for
detecting features in images, texts, and languages.
After reviewing several examples of photogrammetry
and remote sensing carried out at Leibniz University
Hannover, Heipke and Rottensteiner (2020) claimed
that deep learning, particularly CNN, improved sig­
nificantly the photogrammetric processing in surface
reconstruction, scene classification, object extraction,
and object tracking and recognition. Yang,
Rottensteiner, and Heipke (2021) proposed a deep
learning framework with a two-step strategy to verify
land use information. The first step was to detect the
high-resolution aerial images by a CNN, and
the second step was to classify land use by
another CNN.
Machine learning is a promising technique in the
conservation of historical buildings and heritages
(Mesanza-Moraza, Garcia-Gomez, and Azkarate
2021). Egorova and Shcherbinin (2016) suggested
creating virtual heritage replicas by 3D modeling to
promote their preservation. After a systematic review
of various machine learning techniques applied in
historical buildings, Mishra (2021) highlighted the
research directions to improve the robustness of sev­
eral predictive models such as determination of the
mechanical properties of building materials, damage
scenarios, and damages on the building surfaces. Yu
et al. (2022) proposed to use the deep networks to
automatically perform the restoration of the destroyed
historical records in Dunhuang Mogao Grottoes in
China. Karadag (2023) used conditional generative
adversarial networks, a subset of machine learning,
to predict missing and damaged parts of 200 historical
buildings within early Ottoman tombs and their doc­
umentation was generated accurately through 3D
point cloud data. With the growth of 3D capturing
devices such as 3D laser scanners and UAVs equipped
with high-resolution cameras, photogrammetry 3D
modeling has increased the productivity of modeling
historical buildings. Galanakis et al. (2023) explored
deep learning to detect the detailed boundary of
stones in the Apollo Temple at Delphi, reaching
stone-level accuracy. Koutsoudis et al. (2021) pre­
sented a machine-learning approach to produce
a large-scale 3D model of the urban area of 0.5
square km with a multilayer texture map from the
multispectral aerial images.
4 W. HU AND R. HU
2.4. Heterogeneous data fusion by the CNN method
As a subset of machine learning designed for proces­
sing structured arrays of data, CNNs are at the heart
of deep learning algorithms, which are widely used in
computer vision and language processing because of
their superior performance. CNNs have three main
types of layers including a convolutional layer,
a pooling layer, and a fully connected layer. Song
et al. (2020) concluded that CNN was a good algo­
rithm for multi-source heterogeneous data fusion.
Janssens et al. (2016) utilized the fusion of multi-
source heterogeneous data developed by a CNN
model to bear fault detection of the vibration signal.
Zhang et al. (2019) proposed a fusion of 2D CNN and
3D DenseNet for dynamic gesture recognition. Lv
et al. (2019) developed a two-route CNN model to
classify bank accounts with heterogeneous data,
where suspicious bank accounts were identified using
transaction data through the CNN model. Nti, Felix
Adekoya, and Asubam Weyori (2021) proposed a new
multi-source information fusion framework to predict
stock prices based on a hybrid CNN structure and
long-short memory. Saba et al. (2019) developed
a damage detection and recognition model according
to the patient skin images based on CNN. Li et al.
(2017) proposed a novel efficient deep fusion CNN for
multimodal 2D + 3D facial expression recognition.
Previous literature reveals that the multi-source data
method based on CNNs have been widely applied in
computer image recognition, those images are faces,
skins, and gestures, but how the large-scale building
images are recognized to create 3D models is not clear,
which demands further study in this paper.
Figure 1. Flow chart of deep fusion of multi-source heterogeneous data using residual 3D CNN.
3. Research framework and data sources
3.1. Research framework for creating 3D building
models
The framework for creating 3D models for histor­
ical buildings consists of four steps (Figure 1). The
first step is to acquire heterogeneous data from
various sources including a 3D laser scanner on
the ground, aerial cameras on a UAV, handy cam­
eras, and total stations. The data can be classified
into three types: point cloud data, image data, and
survey data. The second step is data preprocessing
through a 3D laser scanning system, which pro­
duces a large amount of sparse point cloud data.
The point cloud data from 3D scanners will be
reduced and filtered and noises in the point clouds
should be deleted and some key points must be
calibrated. After data preprocessing, a 3D point
cloud draft model will be developed by data inte­
gration. The third step is to recognize various fea­
tures of the historical building and fuse those data
into a refined 3D building model by a deep learning
algorithm, called the residual 3D CNN, including
input, 3D CNN, RELU, maximum pooling layer,
fully connected layer, and output. A refined build­
ing model could represent detailed building facades
with features of color, texture, shape, and spatial
relationship and it also includes the spatial structure
information and 3D topological relationship of
building components. The fourth step is to create
a perfect 3D building model on an HBIM platform,
integrating architectural data, building components,
and facilities.

3.2. Data sources for modeling a historical building
The main data sources for modeling a historical build­
ing are a 3D laser scanner, a UAV, a total station, and
field surveyors.
3.2.1. 3D laser scanning
The 3D laser scanning technology has been widely used
in creating building facade models and interior decora­
tion models of historical buildings. Its main contents are
the reconstruction of facades and interior components.
(1) Digital models of building facades
Since its facades and roofs are so unique and its
structure and components are so complicated that
a historical building is difficult to be modeled by
a forward parametric drawing method, but it is suitable
for reverse modeling technology, e.g. 3D laser scanning.
To solve the problems of complex spatial topological
relations and mutual blocks among building compo­
nents, in a reverse modeling method the point clouds
are first encapsulated into surface patches, and then the
holes in the patches, including internal holes, edge
holes, and other data holes, are repaired.
(2) Digital models of building interior decoration and
components
A historical building e.g. the Jade Buddha Temple has
a large number of valuable decorations and compo­
nents, including statues, pillars, shrines, wall reliefs,
roof murals, special roof shapes, ancient beams and
columns, and unique structures, whose models are
only suitable for 3D laser scanning technology. After
considering the integrity of compatible data integrity
and lightweight data, a data acquisition scheme called
first overall and after local is often adopted. The internal
structure of a historical building should be scanned in
a panoramic fine view at first and the building interior
details are scanned in a high precision afterward so that
this scheme can ensure data integrity while reducing
data volume.
3.2.2. UAV tilt photogrammetry
The UAV tilt photogrammetry technology is a new tool
in surveying, mapping, and remote sensing, which com­
bines traditional aerial photography and close-range
measurement technology, breaking through the limita­
tion of the ortho-images taken only from a vertical
angle, and acquiring images by single or multiple lenses
mounted on a flight platform in five different angles
including vertical, front-looking, left-looking, right-
looking and rear-looking. A tilted camera on a UAV
may have five lenses, three lenses, two lenses, or a single
lens. Except for the single lens, the angles of other lenses
are fixed. The single lens on a UAV completes five-view
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 5
photogrammetry by adjusting the angle of the camera
lens.
In aerial photography, the UAV flies at an altitude
higher than the building roof and its camera lens angle
is fixed, there are many blind spots because of the
limitations of the ground trees and the view angles.
The blind spots in photography make some textures in
a building model miss, which seriously affects the mod­
eling results. For example, in the UAV ortho-images,
the ground resolution in the UAV aerial photograph is
inversely proportional to the UAV altitude. When the
airborne camera focal length and the photograph pixels
are fixed, along with the increase of the height between
the ground object and the UAV, the aerial photograph
resolution decreases continuously. Therefore, the reso­
lution on a building top is higher but on its bottom is
lower in a UAV aerial photograph, which results in
a conclusion that the model texture on the building
top is good but on its bottom is poor. Other surveying
or photography tools are needed to tackle this problem.
3.3. Surveying by total stations and other methods
The total station is a combination of a digital theodolite
and an electronic distance-measuring device, useful for
measuring the vertical and horizontal angles and slope
distances. Data collected and processed in a total station
can be downloaded to computers for further processing.
A total station can provide more accurate measurement
than photogrammetry tools e.g. 3D scanners and UAVs,
and it can be used to measure the distance between
building reference points or compensate for the missed
data in the photogrammetry tools.
4. Multi-dimensional data fusion based on CNN
CNN is a typical hierarchical structure, which classifies
and identifies data by automatically extracting features
layer by layer. 2D CNN is often used to classify still
pictures, but it has some limitations in feature extraction
of serial data such as video and space because it can’t
identify the spatial relationship between each data in
sequences. 3D CNN can capture the correlation between
sequence data. Because its convolution kernel has one
more spatial dimension than 2D CNN, it has more net­
work parameters, which increases the computational
consumption and its computation is slow.
In a three-convolution layer CNN, the first convolu­
tion layer in a CNN detects low-order features such as
edges, angles, curves, etc. The input of the second con­
volution layer is the output of the first layer, and
the second layer can detect the combination of low-
order features such as circles and polygons. The input
6 W. HU AND R. HU
of the third convolution layer is the output of the second
layer, and the third layer can detect the spatial features
e.g. spatial structure. With the increase of convolution
layers, more high-order features are detected, which are
more complex and abstract since they are combinations
of lower-order features. 1D CNN is mainly used to deal
with natural language and time-series models. 2D CNN
is mainly used in computer vision and image processing,
and various CNN varieties emerge, which is the most
commonly used method in various image processing
competitions. 3D CNN is used to process 3D data,
which is mainly used to capture features from the spatial
dimension and the temporal dimension after analyzing
videos. Therefore, 3D CNN is the most effective method
to process hyperspectral remote-sensing images.
4.1. Method of 3D convolution
The input of 3D CNN is many 3D swept cubes, which
can simultaneously extract features in three scales.
Through the 3D convolution kernel, features can be
extracted from successive frames and feature cubes can
be connected to multiple 3D models in the upper layer
so that all information on a whole-space piece can be
captured.
The convolution kernel in 3D CNN is a 3D cube. In
the network, each feature cube in the convolution layer
can be connected with several adjacent 3D figures from
the previous layer and then be operated in convolutions.
The 3D convolution process is shown in Figure 2. The
position value of a feature cube is felt locally by con­
volution of the same position of multiple 3D graphics
frames in the upper layer. The 3D convolution is defined
by Equation (1).
XP XQ XR
1 1 1
Vxyz ¼ p¼0 q¼0 r¼0
wpqr uðxþpÞðyþqÞðzþrÞ
Figure 2. 3D convolution process.
The position value of a feature map at the i-th layer
and the j-th sequence at positions (a, b, c) is Vijabc ,
represented by Equation (2).
� Xm XPi 1 XQi 1 XTi 1 pqt �
ðaþpÞðbþqÞðcþtÞ
Vijabc ¼ f dij þ k¼1 p¼0 q¼0 t¼0
Y ijk � Vð i 1Þ;k
(2)
where m is the number of feature maps. Pi and Qi are
convolution kernel sizes, representing the length and
pqt
width of the convolution kernel, respectively.Yijk is the
convolution kernel connected with the (i-1)-th layer and
ðaþpÞðbþqÞðcþtÞ
the k-th sequence feature map. Vði 1Þ;k is the
value of the (i-1)-th layer and the k-th feature map at
the position of ða þ pÞðb þ qÞðc þ tÞ. dij is the offset
between the i-th layer and the j-th feature map.
The down-sampling layer is usually used in conjunc­
tion with the convolution layer. Due to the limitation of
convolution kernel size, the size of feature maps is
limited after convolution processing. So it is necessary
to compress the features of feature maps effectively in
a pooling layer. Because the convolution operation for
large-scale images in large-scale data sets is too slow, the
network training speed will be seriously affected. To
improve the network training speed and reduce the
complexity of the network, it is best to use these two
layers together. The down-sampling operation has
translation invariance, that is, the extracted features
can remain unchanged while moving a target in the
image in a small displacement, which enhances its
robustness to displacement. Therefore, the down-
sampling layer is an effective method to reduce the
dimensionality of data images, which reduces the
redundancy of sample information in the process of
dimensionality reduction, so it has a certain resistance
to over-fitting. The sampling layer uses Equation (3).
(1)
ymnl ¼ max ðxm�sþi;n�tþj;l�rþk Þ (3)
0�i�S1;0�j�S2;0�k�S3

Where s, t, and r are the sampling steps in three
directions, respectively; x is a 3D input vector; y is the
output after sampling.
The main functions of the down-sampling layer are
to reduce the image resolution, reduce the amount of
network operation data, and enhance the robustness of
the network structure. However, the down-sampling
layer should not be added too frequently, because the
down-sampling will zoom the image size in multiple
scales and fuzz the feature information seriously.
The pooling layer is to extract features by
a convolution operation. Although the parameters
have been reduced by previous layers, they are still
relatively complicated in calculating when they are
directly inputted into a classifier. At this time, the pool­
ing function is generally required to convolute and
process the data so that the feature is mapped. It is
a cluster operation, known as down-sampling, which
means that a location and the eigenvalues of its neigh­
borhoods are statistically analyzed and its location is
updated by this value. The common pooling methods
are average pooling and maximum pooling. The pooled
layer generally has no filling operation, so the size after
pooling is n s f þ 1 � n s f þ 1.
4.2. Structure of a 3D CNN
To extract diverse information from a set of feature
maps to improve their feature representation, the num­
ber of convolution kernels can be added. The common
criterion in designing a CNN is that deeper layers and
more feature maps demand more convolution kernels in
the convolutional layers, which can combine more low-
level feature maps and generate more diverse high-level
features. The structure of 3D CNN is developed and
shown in Figure 3, including one input layer, five 3D
convolutional layers and five 3D pooling layers that are
interlaced with each other, and two fully connected
layers which yield the classified results.
The first layer is the input layer, composed of multi­
ple 3D images. C1-C5 are five convolution layers, and
C1:16@3x3x3
Input
S1:1x2x2
Output D2 D1:256
Figure 3. The 3D CNN structure.
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 7
the convolution kernel in each of them is 3 × 3 × 3. The
kernel’s number increases from 16 to 256 in an order to
generate more types of high-level features by combina­
tions of low-level features. S1-S5 are the down-sampling
layers, where the maximum pooling method is adopted
to reduce the resolution and scale of feature maps to
decrease computation load and to tolerate more distor­
tion of inputted images. The down-sampling layer uses
a 1 × 2 × 2 window to sample in the spatial dimensions.
The D1 layer is a fully connected layer with 256 neurons,
where the feature cubes outputted from the S5 layer
connect. The D2 layer is the second fully connected
layer and also the output layer. The number of neurons
is 6, which is the same as the number of target classes.
Each neuron in the D2 layer is fully connected with 256
neurons in the D1 layer. After regression and classifica­
tion by a classifier, the behavior category is marked and
obtained in the output.
4.3. Process of a 3D CNN
The building images are inputted into the network,
which produces pictures of various sizes but their mea­
surements are invariant. After that, it goes to the next
three stages. In the first stage, a constructed image
pyramid is sent to the Phase I network to obtain
a boundary box vector and the building frame, where
the building model is calibrated by the regression of
boundary box vectors, and the candidate frames with
high coincidence are merged by the non-maximum
suppression. In the second stage, the building images
are sent to Phase II, where the improper building mod­
els that do not meet the requirements are rejected con­
tinuously. The overlapped candidate frames are merged
by the regression of the boundary box and the non-
maximum suppression, and fewer boundary boxes
remain. In Phase III the boundary box regression and
the non-maximum suppression continue to identify the
building areas, and the final boundary box and key point
positions are obtained. The visualized process in
a historical building is shown in Figure 4.
C2:32@3x3x3 C3:64@3x3x3
S2:1x2x2 S3:1x2x2
C5:256@3x3x3 C4:128@3x3x3
S5:1x2x2 S4:1x2x2
8 W. HU AND R. HU

�� ��2
4.3.1. Classification of buildings �� landmark ��
Llandmark
i ¼ ��ybi yi landmark �� (6)
To judge whether the building image is a binary classi­ 2
fication, a cross-entropy loss function is adopted Where yi landmark is the predicted location of key posi­
(Equation 4).
tions in a building; ybi landmark is the real location of key
� positions in the building. yi landmark 2 R10 means that the
Ldet
i ¼ ðyidet log ðpi Þ þ 1 yidet ð1 log ðpi ÞÞ (4)
coordinates of building components such as doors, win­
where pi is the probability of the building sample xi , dows, pipes, walls, etc.
yidetis a real tag. Since there are various tasks in each level of CNN,
after reviewing different types of images such as back­
grounds, buildings, and non-buildings, the total loss
4.3.2. Regression of boundary boxes function is expressed as follows.
The regression of boundary boxes is to calculate the XN X j j
predicted offset of a candidate box and a real box, TotalLoss ¼ Min i¼1
α β L (7)
j2fdet;box;landmarkg j i i
which is regarded as a regression task. A Euclidean
loss function (Equation 5) is introduced. Where N represents the number of training sets,
j
�� ��2 βi2 f0; 1g is used to indicate different training samples.
�� box �� αj represents the important degree of different
Lbox
i ¼ ��ybi yi box �� (5)
2 components.
yi box is the
Where is the regressive prediction value;b A new type of explanatory loss function is defined as:
box 4
real bounding box coordinates; yi 2 R represent the L ¼ Ls0 þ λc Lc þ λc0 Lc0
upper left corner coordinates, width, and length, Xm eαðwxi þbyi Þ λc Xm
respectively. ¼ i¼1
log Pn αðwxi þbj Þ 2 þ i¼1
jjxi cyi jj
j¼1 e
λc0 Xm 1
4.3.3. Location of key points þ Pn ���� ��
2 i¼1
j¼1 xi cyi �� þ 1
To predict the offset value, a loss function is defined by
(8)
Equation (6)

Figure 4. Process of 3D CNN in a historical building.

 INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 9

Where W is the weight vector of the final inner 4.5. Backpropagation algorithm
convolution layer, m is the number of images in
The backpropagation algorithm is widely used in most
a training batch, n is the number of categories, b is the
networks to update their weights. Its process is to use
offset,xi is the building feature labeled as yi , α is used to
a mean square error loss function to measure the error
control the separability degree, λc is used to adjust the
between the actual output and the real label of an object,
compactness within the class, λc0 is used to adjust the
and then repeatedly iterate and optimize the network
separability degree between classes.
parameters according to the error backpropagation
mechanism.
4.4. Residual learning There is a sample data set in a neural network of
forward propagation, and the loss function by Euclidean
A deep CNN can transform the feature representation distance is obtained by Equation 12.
of samples in the original space into a new feature
space, and then obtain the hierarchical feature repre­ 1 �� ��2
J ðW; b; x; yÞ ¼ ��hw;b ðxÞ y �� (12)
sentation by automatic learning. When compared with 2
other traditional pattern recognition methods, the
deep learning network has more layers and its struc­ where m represents the number of categories of sam­
ture is more complex so it has stronger abilities in ples, hw;b ðxÞ is the network output, and y is the actual
feature learning and feature expression. Therefore, the output.
depth of the network determines the model perfor­ The expression multiplied by 1/2 is to make the
mance. However, along with the network deepening, derivative of the loss function coefficient equal to 1.
the network will degrade and the accuracy of classifi­ The whole loss function of the sample data set consists
cation recognition quickly reaches a maximum, and of two parts (Equation 13): The first is the overall loss
then deeper levels would cause a higher error rate. Of function of the data set, and the second is the weight
course, the higher error rate caused by the network attenuation of the whole network.
degradation is not caused by over-fitting but only by � X �
1 m i i
�
more layers. When the problem of network J ðW; bÞ ¼ J w; b; x ; y
m i¼1
degradation is solved, a deeper network can be trained
λ Xnl 1 Xsl Xslþ1 � ðlÞ �2
successfully. The residual network introduces a cross- þ Wji (13)
2 l¼1 i¼1 j¼1
layer connection, or a shortcut connection, to solve
this problem, which is constructed as follows. Because the function J ðW; b; x; yÞ is continuous and
If a cross-layer connection is added, the relationship derivable, it can be updated by its negative derivative of
between F(x) and G(x) is shown as follows. weights along with the gradient directions (W and b in
F ðxÞ ¼ GðxÞ x (9) the function), also called the gradient descent method.
The parameters W and b are updated by Equation 14
The cross-connected module is called the residual and 15.
module because it calculates the residual error when
not connected. fmi g represents the ownership value of ðl Þ ðl Þ @
Wij ¼ Wij α ðl Þ
J ðW; bÞ (14)
the residual module, then the output of the residual @Wij
module is y.
y ¼ F ðx; fmi gÞ þ x (10)
ðl Þ ðl Þ @
bi ¼ bi α J ðW; bÞ (15)
When calculating the residual module, F ðx; fmi gÞ @bi
ðlÞ

and x should have the same dimension. If their digits

are different, a weight matrix m needs to be introduced. Where α represents the network learning speed or
After x is linearly projected, their digits are adjusted to the learning rate.
be the same. The partial derivative of the overall loss function is
y ¼ F ðx; fmi gÞ þ ws x (11) obtained.
2 3
In the process of forwarding propagation, F ðxÞ þ x is @ 1 X m @
the sum of F ðxÞ and x, where no additional parameters ðlÞ
J ðW; bÞ ¼ 4 ðl Þ
Jðw; b; xi ; yi Þ5
@W m i¼1
@W
are introduced so that the complexity of the original ij ij
ðl Þ
network will not be affected and a backpropagation þ λWij (16)
algorithm can still be used to train the network.
10 W. HU AND R. HU

" # " #
@ 1 Xm @ i i l l 1 @
ðl Þ
J ðW; bÞ ¼ ðlÞ
Jðw; b; x ; y Þ (17) b ¼b α Jðw; b; xi ; yi (23)
@b m i¼1
@b m @bðlÞ
i i i

The weight attenuation part only acts on the para­ The process of backpropagation is to search for the
meter W, and the whole backpropagation process con­ minimum value of the loss function J ðW; bÞ, in which
sists of four steps. the parameters W and b play important roles. At first, all
(1) Regarding each output unit at the nl output layer, parameters Wij are initialized to a random value close to
the error of the i-th unit can be expressed as follows. zero. Then, the gradient descent method is used to
@ � � � � reduce the value of the loss function J ðW; bÞ in repeated
ðn Þ ð nl Þ 0 ðnl Þ
δi l ¼ ðn Þ
J ðW; b; x; y Þ ¼ y i aj � f Z j iteration.
@bZi l
(18)
(2) Regarding each output unit at the hidden layer 5. Pilot project in Shanghai
nl 1, the error of the i-th unit at the l-th layer be There is an upgrading project in Shanghai, China, con­
expressed. sisting of 49 historical buildings. All of them were built
ðnl 1Þ @ a century ago and are slowly deteriorating now. Since
δi ¼ ðnl 1Þ
J ðW; b; x; yÞ they are heritages, their appearance should remain as
@Z
�X i �� � ��
Snl ðnl 1Þ ð nl 1 Þ ðn Þ
the old but their structural components should be
¼ j¼1
Wij � δi �f 0 Zj l (19) replaced and reinforced. Therefore, a reserve engineer­
ing technology is selected to create 3D models to eval­
(3) Partial derivative can be obtained. uate this project. Managers utilize a UAV, a 3D laser
@ ðlÞ ðlþ1Þ scanner, and other devices to capture building data. To
ðl Þ
J ðW; b; x; yÞ ¼ aj δi (20) avoid potential blocks during capturing data, branches
@Wij
around buildings have been trimmed (shown in
Figure 5)
@ ðlþ1Þ
ðl Þ
J ðW; b; x; yÞ ¼ δi (21)
@bi
(4) The weight parameters are updated. 5.1. Multi-source heterogeneous data and
2 3 surveying routes
ðl Þ ðl Þ 1 @ � ðl Þ Before updating a historical building, we survey and
Wij ¼ Wij α4 J w; b; xi ; yi 5 þ λWij
m @W ðlÞ record its facades and features. The surveying task
ij
consists of four types of activities: (1) surveying the
(22)
architectural features including facades by 3D laser

Figure 5. A historical building in the pilot project.


scanners, UAV, and total stations; (2) investigating
the building components and interior by photos; (3)
identifying the building materials by tests; (4) locat­
ing key positions by total stations. An accurate mea­
surement is critical for developing 3D building
models.
5.1.1. Surveying devices and surveying routes
The surveying job is well planned in three steps: the
preliminary surveying step, the comprehensive survey­
ing step, and the supplementary surveying step.
A Phantom 4 pro UAV with a 20 megapixel came,
a FARO FocusS 350 laser scanner and a Leica TS02
total station were utilized in this surveying task.
Surveying routes in a historical block with six buildings
(shown in Figure 6) are carefully designed for surveying
devices to capture building data. We have surveyed each
historical building based on its surveying route. There
are nine ground stations designed for the 5th building
(shown in Figure 7), which are 6 to 10 m away from the
building corners, where the surveying devices would
scan or survey at least two building facades. The point
cloud data from the laser scanner at two stations would
overlap 50% of each façade, and the UAV photogram­
metry could upgrade its color, texture, and resolution.
A field investigation is also critical in collecting building
data e.g. building materials, components, and features.
5.1.2. Control points and calibration errors of 3D
point cloud models
To verify the accuracy of 3D scanning, we adopt the
total station to survey the control points in historical
buildings. For example, we set a vertical line along the
Figure 6. Deployment of surveying routes and ground stations.
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 11
building corner and there are six vertical lines around
the 5th building, and we set eight control points along
a vertical line and there are 48 control points on the
surface of the 5th building.
The height of the control points is evaluated by the
remote elevation measurement method. After compar­
ing the control points between the point clouds and the
total station’s measurement, this building has sloped up
slightly at the eastern façade, and the 3D point cloud
models are calibrated by 48 control points. The calibra­
tion errors of control points’ heights are shown in
Table 1. The result indicates that the maximum calibra­
tion error is 18 mm, the minimum calibration error is 4
mm, and the average calibration error is 13 mm, which
means that the 3D point clouds can meet the require­
ment of building models.
5.1.3. Preliminary tests of 3D point cloud models
The 5th building surface is scanned by a sampling inter­
val of 5 mm and the surface model includes 2.4 million
cloud points. Since the accuracy of the 3D scanner is 1
mm at a single location, a dense point cloud model
would have a higher accuracy and a relative accuracy
could reach 1 mm. To compare the accuracy of various
point cloud models, we develop its surface model by
four kinds of points: 24 million points, 4.8 million
points, 2.4 million points, and 1.2 million points,
whose sample intervals are from 5 mm to 20 mm and
the comparison shows that all surface models can
express this building’s details and their relative accura­
cies can reach 1 mm. After reviewing the results, we
adopted the sampling interval of 5 mm to scan other
historical buildings in this project.
12 W. HU AND R. HU

laser scanning, which would result in a better fusion of

multi-data. The are 316 photos taken after shooting the
building and their average ground resolution is 0.7 mm.

Figure 7. Deployment of the ground stations and the surveying

route around a building.

5.1.4. UAV surveying schemes and routes

After reviewing the surroundings and roofs, we pro­
posed various UAV survey schemes and routes for dif­
ferent buildings. In the UAV flight scheme for the 5th
building, the flight height is 9.324 m for the first circle,
13.465 m for the second circle, 17.378 m for the third
circle, and 22.164 m, the longitudinal overlap rate is
85%, the lateral overlap is 80%, the camera angle is
45°, and the ground sample distance (GSD) is 10 mm.
Since the UAV CMOS sensor size is 1 inch and f = 8.8
mm, the minimum GSD is H � af ¼5.89 mm, which
meets the surveying requirement. Figure 8. Example of UAV circling routes around a building.
The UAV flies in a circling route (shown in Figure 8)
for 38 min, at the speed of 5 m/s. The camera takes model is developed (shown in Figure 9). To improve
photos of the building in the same flight route in five its precision and reduce measurement errors, some key
directions: vertical, tilt forward, tilt back, tilt left, and tilt positions in the building model are calibrated.
right. The 48 control points on the 5th building surface When the building images and data are trained by the
are used to calibrate models in the UAV tilt photogram­ 3D CNN model, the results at each convolution layer are
metry. To secure similar lighting effects, the UAV tilt obtained (shown in Figure 10). The results from the
photography is conducted at the same time as the 3D CNN model can be refined further in an HBIM platform

Table 1. Calibration errors of 48 control points’ heights on the 5th building.

No. of control Calibration error No. of control Calibration error No. of control Calibration error No. of control Calibration error
points (m) points (m) points (m) points (m)
SE01 0.006 SC05 0.013 NW01 0.006 NC05 0.017
SE02 0.013 SC06 0.007 NW 02 0.013 NC06 0.015
SE03 0.014 SC07 0.015 NW 03 0.016 NC07 0.014
SE04 0.015 SC08 0.017 NW 04 0.007 NC08 0.017
SE05 0.017 SW01 0.005 NW 05 0.015 NE01 0.004
SE06 0.015 SW02 0.009 NW 06 0.011 NE02 0.009
SE07 0.013 SW03 0.012 NW 07 0.014 NE03 0.015
SE08 0.017 SW04 0.013 NW 08 0.009 NE04 0.014
SC01 0.004 SW05 0.017 NC01 0.005 NE05 0.013
SC02 0.009 SW06 0.018 NC02 0.015 NE06 0.016
SC03 0.012 SW07 0.017 NC03 0.013 NE07 0.016
SC04 0.015 SW08 0.011 NC04 0.017 NE08 0.015

Figure 9. Extraction of building edges and modeling in draft.
so that a 3D building model with more detailed infor­
mation e.g. building components and facilities will be
created and applied in exporting to a serial of construc­
tion drawings.
5.3. Recovery of the damaged building details
To repair the damaged building details, we use 263
photos to train the CNN model, and then we use 140
photos to check the accuracy of the CNN model.
When damaged building details are identified by
the CNN model, then it spots them and tries to
repair them. For example, several details in a door
header were damaged, which are recognized by the
CNN model, then it recovers them, shown in
Figure 11.
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 13
5.4. Evaluation of various models
To test the accuracy of models generated by the
multi-source data fusion method, we compare the
coordinate errors of 48 control points in the 3D
point cloud model, UAV tilt photogrammetry
model, and the deep fusion model, with the total
station surveying. The comparison of errors in
three dimensions (ΔX, ΔY, and ΔZ) of control points
and their root mean square (RMS) is shown in
Table 2.
The result of the 5th building surveying reveals that
the deep fusion model is much better than the 3D point
cloud model and the tilt photogrammetry model, which
proves that the modeling precision by multi-source data
fusion technology can meet the requirement of a 3D
historical building model.
14 W. HU AND R. HU

Figure 10. 3D CNN process of a building model.

Figure 11. Recovery of damaged building details.

Table 2. Comparison of the coordinate errors of control points in various models of the 5th building surface.
3D point cloud model (m) Tilt photogrammetry model (m) Deep fusion model (m)
No. of control points ΔX ΔY ΔZ ΔX ΔY ΔZ ΔX ΔY ΔZ
SE01 0.008 0.013 0.017 0.021 0.016 0.027 0.011 0.005 0.012
SE02 0.011 0.012 0.024 0.015 0.014 0.017 0.007 0.008 0.015
SE03 0.012 0.008 0.015 0.013 0.027 0.014 0.008 0.009 0.017
SE04 0.012 0.009 0.021 0.016 0.026 0.027 0.011 0.013 0.017
SE05 0.013 0.014 0.015 0.017 0.023 0.025 0.008 0.016 0.012
SE06 0.014 0.016 0.029 0.023 0.022 0.019 0.017 0.012 0.021
SE07 0.011 0.018 0.012 0.024 0.018 0.017 0.013 0.007 0.006
SE08 0.015 0.019 0.022 0.018 0.019 0.014 0.006 0.010 0.007
SC01 0.006 0.016 0.021 0.016 0.012 0.017 0.007 0.013 0.008
SC02 0.008 0.015 0.014 0.025 0.017 0.026 0.005 0.011 0.012
SC03 0.018 0.005 0.027 0.023 0.026 0.017 0.012 0.012 0.021
SC04 0.013 0.004 0.006 0.017 0.021 0.011 0.009 0.007 0.003
SC05 0.011 0.002 0.013 0.018 0.014 0.025 0.012 0.006 0.011
SC06 0.004 0.009 0.021 0.027 0.017 0.028 0.005 0.007 0.013
SC07 0.012 0.015 0.017 0.024 0.026 0.023 0.011 0.013 0.006
SC08 0.014 0.013 0.018 0.029 0.024 0.027 0.008 0.006 0.007
RMS 0.012 0.013 0.019 0.021 0.021 0.022 0.009 0.010 0.013
Comparison of other control points are not listed here.

6. Conclusion and future studies
The key task before repairing a historical building is
how to create a complete 3D model with featured
facades, detailed decorations, and architectural compo­
nents. Since a historical building has so many unique
features, it is hardly identified by manual surveying and
its 3D models are impossible to be created effectively
with manual measurement tools. As an excellent deep
learning method, CNN has widely been used to recog­
nize complicated features. Learning skills in feature

recognition from sufficient cases, improve its capability
of distinguishing different features. However, the build­
ing features extracted by the traditional loss function are
not good enough to be discriminated from other fea­
tures in a large number of data sets, the building infor­
mation cannot be completely represented by those
features. A 3D CNN model with one input layer, five
3D convolutional and pooling layers, and two connected
layers, is developed to fuse all data from various sources
like a UAV, a 3D laser scanner, a total station, and other
images. By improving the CNN structure and the loss
function, the 3D CNN model has constructed 3D build­
ing models for a pilot project of historical buildings in
Shanghai by multi-source data fusion. The pilot project
proves that the 3D CNN method can improve the effi­
ciency and accuracy of modeling historical buildings in
an automatic process. How the 3D building models are
further refined in HBIM platforms will be investigated
by future studies.
Acknowledgments
The authors wish to acknowledge Tongji Architectural Design
(Group) Co., Ltd for extending support in carrying out the
field study. Financial support is from the National Natural
Science Foundation of China (grant number 71971158).
Disclosure statement
No potential conflict of interest was reported by the author(s).
References
Alizadeh-Khameneh, M. A., M. Horemuz, A. B. O. Jensen,
and J. V. Andersson. 2018. Optimal vertical placement of
total station. Review of Journal of Surveying Engineering
144 (3). doi:10.1061/(asce)su.1943-5428.0000255.
Andriasyan, M., J. Moyano, J. E. Nieto-Julian, and D. Anton.
2020. From point cloud data to building information mod­
elling: An automatic parametric workflow for heritage.
Review of Remote Sensing 12 (7):1094. doi:10.3390/
rs12071094.
Bastem, S. S., and A. Cekmis. 2022. Development of historic
building information modelling: A systematic literature
review. Review O Building Research and Information
50 (5):527–58. doi:10.1080/09613218.2021.1983754.
Bruno, N., and R. Roncella. 2019. HBIM for conservation:
A new proposal for information modeling. Review of
Remote Sensing 11 (15):1751. doi:10.3390/rs11151751.
Bruno, S., M. De Fino, and F. Fatiguso. 2018. Historic building
information modelling: Performance assessment for
diagnosis-aided information modelling and management.
Review of Automation in Construction 86:256–76. doi:10.
1016/j.autcon.2017.11.009.
Chen, C. S., Y. H. Chiu, and L. C. Tsai. 2018. Evaluating the
adaptive reuse of historic buildings through multicriteria
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 15
decision-making. Review of Habitat International 81:12–23.
doi:10.1016/j.habitatint.2018.09.003.
Cheng, L., J. Gong, L. Manchun, and Y. Liu. 2011. 3D building
model reconstruction from multi-view aerial imagery and
lidar data. Review of Photogrammetric Engineering and
Remote Sensing 77 (2):125–39. doi:10.14358/pers.77.2.125.
Cheng, L., L. Tong, Y. Chen, W. Zhang, J. Shan, Y. Liu, and
L. Manchun. 2013. Integration of LiDAR data and optical
multi-view images for 3D reconstruction of building roofs.
Review of Optics and Lasers in Engineering 51 (4):493–502.
doi:10.1016/j.optlaseng.2012.10.010.
Clarke, J. A., and D. F. Laefer. 2014. Systematic approach for
large-scale, rapid, dilapidation surveys of historic masonry
buildings. Review of International Journal of Architectural
Heritage 8 (2):290–310. doi:10.1080/15583058.2012.
692849.
Collepardi, M. 1999. Thaumasite formation and deterioration
in historic buildings. Review of Cement & Concrete
Composites 21 (2):147–54. doi:10.1016/s0958-9465(98)
00044-4.
Cornelis, N., B. Leibe, K. Cornelis, and L. Van Gool. 2008. 3D
urban scene modeling integrating recognition and
reconstruction. Review of International Journal of
Computer Vision 78 (2–3):121–41. doi:10.1007/s11263-
007-0081-9.
Egorova, O., and D. Shcherbinin. 2016. Creating technical
heritage object replicas in a virtual environment. Review
of Frontiers of Mechanical Engineering 11 (1):108–15.
doi:10.1007/s11465-016-0363-4.
Franceschi, M., M. Martinelli, L. Gislimberti, A. Rizzi, and
M. Massironi. 2015. Integration of 3D modeling, aerial
LiDAR and photogrammetry to study a synsedimentary
structure in the early Jurassic Calcari Grigi (Southern
Alps, Italy). Review of European Journal of Remote Sensing
48 (1):527–39. doi:10.5721/EuJRS20154830.
Galanakis, D., E. Maravelakis, D. P. Pocobelli, N. Vidakis,
M. Petousis, A. Konstantaras, and M. Tsakoumaki. 2023.
SVD-based point cloud 3D stone by stone segmentation for
cultural heritage structural analysis-The case of the Apollo
Temple at Delphi. Review of Journal of Cultural Heritage
61:177–87. doi:10.1016/j.culher.2023.04.005.
Heipke, C., and F. Rottensteiner. 2020. Deep learning for
geometric and semantic tasks in photogrammetry and
remote sensing. Review of Geo-Spatial Information Science
23 (1):10–19. doi:10.1080/10095020.2020.1718003.
Jaafar, H. A., X. L. Meng, A. Sowter, and P. Bryan. 2017. New
approach for monitoring historic and heritage buildings:
Using terrestrial laser scanning and generalised procrustes
analysis. Review of Structural Control & Health Monitoring
24 (11):e1987. doi:10.1002/stc.1987.
Janssens, O., V. Slavkovikj, B. Vervisch, K. Stockman,
M. Loccufier, S. Verstockt, R. Van de Walle, and S. Van
Hoecke. 2016. Convolutional neural network based fault
detection for rotating machinery. Review of Journal of Sound
and Vibration 377:331–45. doi:10.1016/j.jsv.2016.05.027.
Jo, Y. H., and S. Hong. 2019. Three-dimensional digital doc­
umentation of cultural heritage site based on the conver­
gence of terrestrial laser scanning and unmanned aerial
vehicle photogrammetry. Review of Isprs International
Journal of Geo-Information 8 (2):53. doi:10.3390/
ijgi8020053.
16 W. HU AND R. HU
Jordan-Palomar, I., P. Tzortzopoulos, J. Garcia-Valldecabres,
and E. Pellicer. 2018. Protocol to manage Heritage-Building
Interventions using heritage building information
Modelling (HBIM).” Review of Sustainability 10 (4):908.
doi:10.3390/su10030908.
Karadag, İ. 2023. Machine learning for conservation of archi­
tectural heritage. Review of Open House International
48 (1):23–37. doi:10.1108/ohi-05-2022-0124.
Karimi, F., N. Valibeig, G. Memarian, and A. Kamari. 2022.
Sustainability rating systems for historic buildings:
A systematic review. Review of Sustainability
14 (19):12448. doi:10.3390/su141912448.
Koutsoudis, A., G. Ioannakis, P. Pistofidis, F. Arnaoutoglou,
N. Kazakis, G. Pavlidis, C. Chamzas, and N. Tsirliganis.
2021. Multispectral aerial imagery-based 3D digitisation,
segmentation and annotation of large scale urban areas of
significant cultural value. Review of Journal of Cultural
Heritage 49:1–9. doi:10.1016/j.culher.2021.04.004.
Lee, P. C., W. Xie, T. P. Lo, D. B. Long, and X. F. Tang. 2019.
A cloud model-based knowledge mapping method for his­
toric building maintenance based on building information
modelling and ontology. Review of Ksce Journal of Civil
Engineering 23 (8):3285–96. doi:10.1007/s12205-019-2457-0.
Leonov, A. V., M. N. Anikushkin, A. V. Ivanov,
S. V. Ovcharov, A. E. Bobkov, and Y. M. Baturin. 2015.
Laser scanning and 3D modeling of the Shukhov hyperbo­
loid tower in Moscow. Review of Journal of Cultural
Heritage 16 (4):551–59. doi:10.1016/j.culher.2014.09.014.
Li, H., J. Sun, X. Zongben, and L. Chen. 2017. Multimodal 2D
+3D facial expression recognition with deep fusion convo­
lutional neural network. Review of Ieee Transactions on
Multimedia 19 (12):2816–31. doi:10.1109/tmm.2017.
2713408.
Lim, J. S., S. Gleason, M. Williams, G. J. Matas, D. Marsden,
and W. Jones. 2022. UAV-based remote sensing for mana­
ging alaskan native heritage landscapes in the
Yukon-Kuskokwim Delta. Review of Remote Sensing
14 (3):728. doi:10.3390/rs14030728.
Lv, F., J. Huang, W. Wang, Y. Wei, Y. Sun, B. Wang, and
P. Pławiak. 2019. A two-route CNN model for bank
account classification with heterogeneous data. Review of
Plos One 14 (8):e0220631. doi:10.1371/journal.pone.
0220631.
Martinez, J., A. Soria-Medina, P. Arias, and A. Felippe
Buffara-Antunes. 2012. Automatic processing of terrestrial
laser scanning data of building facades. Review of
Automation in Construction 22:298–305. doi:10.1016/j.aut
con.2011.09.005.
Martinez-Carricondo, P., F. Carvajal-Ramirez, L. Yero-
Paneque, and F. Aguera-Vega. 2021. Combination of
HBIM and UAV photogrammetry for modelling and doc­
umentation of forgotten heritage. Case study: Isabel II dam
in Nijar (Almeria, Spain). Review of Heritage Science 9 (1).
doi:10.1186/s40494-021-00571-8.
Mesanza-Moraza, A., I. Garcia-Gomez, and A. Azkarate.
2021. Machine learning for the built heritage archaeological
study. Review of Acm Journal on Computing and Cultural
Heritage 14 (1):1–21. doi:10.1145/3422993.
Milic, V., K. Ekelow, M. Andersson, and B. Moshfegh. 2019.
Evaluation of energy renovation strategies for 12 historic
building types using LCC optimization. Review of Energy
and Buildings 197:156–70. doi:10.1016/j.enbuild.2019.05.017.
Mishra, M. 2021. Machine learning techniques for structural
health monitoring of heritage buildings: A state-of-the-art
review and case studies. Review of Journal of Cultural
Heritage 47:227–45. doi:10.1016/j.culher.2020.09.005.
Nieto-Julian, J. E., D. Anton, and J. Jose Moyano. 2020.
Implementation and management of structural deforma­
tions into historic building information models. Review of
International Journal of Architectural Heritage
14 (9):1384–97. doi:10.1080/15583058.2019.1610523.
Nti, I. K., A. Felix Adekoya, and B. Asubam Weyori. 2021.
A novel multi-source information-fusion predictive frame­
work based on deep neural networks for accuracy enhance­
ment in stock market prediction. Review of Journal of Big
Data 8 (1). doi: 10.1186/s40537-020-00400-y.
Ozmen, C., A. E. Akan, and A. I. Unay. 2011. Analysis of
a historic masonry building. Review of Gradevinar
63 (5):449–58.
Patrucco, G., A. Gomez, A. Adineh, M. Rahrig, and
J. L. Lerma. 2022. 3D data fusion for historical analyses of
heritage buildings using thermal images: The Palacio de
Colomina as a case study. Review of Remote Sensing
14 (22):5699. doi:10.3390/rs14225699.
Pesci, A., G. Teza, and E. Bonali. 2011. Terrestrial laser scan­
ner resolution: Numerical simulations and experiments on
spatial sampling optimization. Review of Remote Sensing
3 (1):167–84. doi:10.3390/rs3010167.
Pu, S., and G. Vosselman. 2009. Knowledge based reconstruc­
tion of building models from terrestrial laser scanning data.
Review of Isprs Journal of Photogrammetry and Remote
Sensing 64 (6):575–84. doi:10.1016/j.isprsjprs.2009.04.001.
Rao, C. P., and Y. Liu. 2020. Three-dimensional convolutional
neural network (3D-CNN) for heterogeneous material
homogenization. Review of Computational Materials
Science 184:109850. doi:10.1016/j.commatsci.2020.109850.
Saba, T., M. Attique Khan, A. Rehman, and S. Larabi Marie-
Sainte. 2019. Region extraction and classification of skin
cancer: A heterogeneous framework of deep CNN features
fusion and reduction. Review of Journal of Medical Systems
43 (9). doi: 10.1007/s10916-019-1413-3.
Song, W., L. Zhang, Y. Tian, S. Fong, L. Jinming, and
A. Gozho. 2020. CNN-based 3D object classification using
hough space of LiDAR point clouds. Review of Human-
Centric Computing and Information Sciences 10 (1). doi:10.
1186/s13673-020-00228-8.
Sztwiertnia, D., A. Ochalek, A. Tama, and P. Lewinska. 2021.
HBIM (heritage building information modell) of the Wang
stave church in Karpacz – case study. Review of
International Journal of Architectural Heritage
15 (5):713–27. doi:10.1080/15583058.2019.1645238.
Templin, T., and D. Popielarczyk. 2020. The use of low-cost
unmanned aerial vehicles in the process of building models
for cultural tourism, 3D web and augmented/mixed reality
applications. Review of Sensors 20 (19):5457. doi:10.3390/
s20195457.
Walter, N. 2014. From values to narrative: A new foundation
for the conservation of historic buildings. Review of
International Journal of Heritage Studies 20 (6):634–50.
doi:10.1080/13527258.2013.828649.
Wang, N. N., X. F. Zhao, P. Zhao, Y. Zhang, Z. Zou, and
J. P. Ou. 2019. Automatic damage detection of historic
masonry buildings based on mobile deep learning. Review

of Automation in Construction 103:53–66. doi:10.1016/j.
autcon.2019.03.003.
Wang, Q., C. M. Yang, J. G. Lu, F. W. Wu, and R. Xu. 2020.
Analysis of preservation priority of historic buildings along
the subway based on matter-element model. Review of
Journal of Cultural Heritage 45:291–302. doi:10.1016/j.cul
her.2020.03.003.
Wu, B., L. Xie, H. Han, Q. Zhu, and E. Yau. 2018. Integration
of aerial oblique imagery and terrestrial imagery for opti­
mized 3D modeling in urban areas. Review of Isprs Journal
of Photogrammetry and Remote Sensing 139:119–32. doi:10.
1016/j.isprsjprs.2018.03.004.
Yang, C., F. Rottensteiner, and C. Heipke. 2021. A hierarchical
deep learning framework for the consistent classification of
land use objects in geospatial databases. Review of Isprs
Journal of Photogrammetry and Remote Sensing
177:38–56. doi:10.1016/j.isprsjprs.2021.04.022.
Yang, L., Y. Sheng, and B. Wang. 2016. 3D reconstruction of
building facade with fused data of terrestrial LiDAR data
and optical image. Review of Optik 127 (4):2165–68. doi:10.
1016/j.ijleo.2015.11.147.
Yang, X. C., P. Grussenmeyer, M. Koehl, H. Macher,
A. Murtiyoso, and T. Landes. 2020. Review of built heritage
modelling: Integration of HBIM and other information
techniques. Review of Journal of Cultural Heritage
46:350–60. doi:10.1016/j.culher.2020.05.008.
INTERNATIONAL JOURNAL OF ARCHITECTURAL HERITAGE 17
Yang, X., L. Yi-Chou, A. Murtiyoso, M. Koehl, and
P. Grussenmeyer. 2019. HBIM modeling from the surface
mesh and its extended capability of knowledge
representation. Review of Isprs International Journal of Geo-
Information 8 (7):301. doi:10.3390/ijgi8070301.
Yu, T. X., C. Lin, S. J. Zhang, C. X. Wang, X. H. Ding, H. L. An,
X. X. Liu, T. Qu, L. Wan, S. You, et al. 2022. Artificial intelli­
gence for Dunhuang cultural heritage protection: The project
and the dataset. Review of International Journal of Computer
Vision. 130(11):2646–73. doi:10.1007/s11263-022-01665-x.
Zaragoza, I. M.-E., G. Caroti, A. Piemonte, B. Riedel,
D. Tengen, and W. Niemeier. 2017. Structure from motion
(SfM) processing of UAV images and combination with
terrestrial laser scanning, applied for a 3D-documentation
in a hazardous situation. Review of Geomatics Natural
Hazards & Risk 8 (2):1492–504. doi:10.1080/19475705.
2017.1345796.
Zhang, E., B. T. Xue, F. Z. Cao, J. H. Duan, G. F. Lin, and
Y. F. Lei. 2019. Fusion of 2D CNN and 3D DenseNet for
dynamic gesture recognition. Review of Electronics
8 (12):1511. doi:10.3390/electronics8121511.
Zou, Z., X. F. Zhao, P. Zhao, F. Qi, and N. N. Wang. 2019.
CNN-based statistics and location estimation of missing
components in routine inspection of historic buildings.
Review of Journal of Cultural Heritage 38:221–30. doi:10.
1016/j.culher.2019.02.002.