See discussions, stats, and author profiles for this publication at: [Link]

net/publication/348210139

Reconstructing Photogrammetric 3D Model by Using Deep Learning

Chapter · January 2021

DOI: 10.1007/978-3-030-57509-0_27

CITATIONS READS

2 439

2 authors:

June-Hao Hou Chili Cheng

National Yang Ming Chiao Tung University National Chiao Tung University
43 PUBLICATIONS 49 CITATIONS 6 PUBLICATIONS 10 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Chili Cheng on 21 March 2023.

The user has requested enhancement of the downloaded file.

 Reconstructing Photogrammetric 3D Model
by Using Deep Learning

June-Hao Hou and Chi-Li Cheng

Abstract designers when making decisions. The method to gain data

or model for design becomes more and more important.
In this paper, we propose a deep learning-based method to
There are several ways to get a model of the site that
reconstruct 3D models generated with photogrammetry
architects and designers could get access to. For instance,
techniques. The mechanism to reconstruct the model is
OpenStreetMap provides a quite user-friendly interface for
detecting corners and then segment the polyline contours
gaining or sharing spatial information; the global terrain data
based on the detected corners. Moreover, we take
is opened as SRTM format for everyone. However, most of
advantage of the generative design tool (Grasshopper)
these sources cannot offer the most recent and detailed
to build a training gym system, constantly producing
building models; in order to get more recent models, certain
unique datasets to train the neural network. In the part of
architects begin applying drones with photogrammetry
feature engineering, we use series of external angles as
software to build the model around the design site.
the geometrical feature to train the model, which means
Nonetheless, these models generated by the photogram-
that the model processes a small number array instead of
metric method have certain disadvantages, like the redun-
using images or any kind of huge multi-dimensional data.
dant faces, defects, and errors, etc. These issues make this
The result of this research reveals that the neural network
kind of model extremely difficult to use in a design process.
model trained by our training gym system can learn how
In our previous research [1], we proposed a contour-based
to detect corners on building contours successfully. The
mesh reconstruction with a gaussian mixture model; conse-
details of the training gym, dataset format, dataset
quently, the method is able to filter out irrelevant models and
generator, the deep learning model, and the potential
then use the most representative contours of buildings to
usages will be elaborated in the following content.
rebuild them (see Fig. 1). In this research, we ponder a
method to improve our previous simplification method in the
Keywords reconstruct process. Moreover, the efficiency of the method

Geometry simplification Deep learning Polyline
segmentation Mesh reconstruction Photogrammetry
is considered because we attempt to make it practical and
able to facilitate designers. Thus, there are several aspects of
the reconstruction process that should be scrutinized.

1 Introduction 1.1 Related Work

Nowadays, data-driven design plays a crucial role in archi- This research is to improve our previous research that pro-
tectural design; using various information to analyze per- poses a contour-based method to reconstruct and simplify
formances of different aspects in design can truly facilitate the photogrammetric model [1]. Even though the method in
our previous research is able to reduce photogrammetric
J.-H. Hou models efficiently, it lacks the capability to simplify curved
National Chiao Tung University, Hsinchu City, 30010, Taiwan
e-mail: jhou@[Link] walls on buildings. The polyline of curved walls remain a
series of points after simplification. As a result, the issue in
C.-L. Cheng (&)
Graduate Institute of Architecture, National Chiao Tung this research is how to rebuild the curved parts as surfaces
University, Hsinchu City, 30010, Taiwan decently. The difficulty with the issue is that these models
e-mail: micky@[Link]

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 295
S. Eloy et al. (eds.), Formal Methods in Architecture, Advances in Science, Technology & Innovation,
[Link]
296 J.-H. Hou and C.-L. Cheng

Fig. 1 A method of mesh simplification for photogrammetric models

generated with photogrammetry techniques include different contours to reconstruct buildings to reduce the complexity.
levels of noise depending on the quality and the number of The reason is that most mesh-based methods seem not to be
photos, as well as parameters in the photogrammetry soft- able to tolerate noise well. Since we reduce the complexity a
ware. Accordingly, the method requires the capability to lot, machine learning could focus more on dealing with noise
tolerate the variety of factors on the models. In our recon- on a series of angles on a contour.
struction process, the core task is corner detection; the pro-
cess can detect corners accurately. The number of faces of Method for Processing Polylines
the reconstructed building model will be reduced because Wolin et al. proposed a simple method to find corners by the
that the nodes on the curved wall will be grouped to form a short-straws concept, dealing with polylines that are sup-
surface. There are some existing methods to detect corners posed to be polygon efficiently [6]. Zhang et al. presented
for different scenarios. another method to detect corners with an arc length-based
algorithm. It addresses the issue of measuring discrete cur-
Method for Detecting Corners on Images vature [7]. For simplifying and combining polylines, Joa-
For processing contours and detecting corners, Antoni Buades chim Spoerhase et al. suggest the Simplification of Polyline
et al. proposed an algorithm to process images, deriving Bundles. In their research, they deal with polyline simplifi-
meaningful contours efficiently [2]. In the research, they cation and the issues that a bunch of polylines are adjacent to
employed 3D convolution kernels to derive relevant features each other [8]. It is quite inspiring because they are different
of the contours; furthermore, their algorithm is able to process from other similar simplification research that only focuses
images with noise, resulting in distinct contours of objects. on a single polyline. Besides, there is an approach utilizing
Rosten et al. [3] investigate a way to speed up the process drawing speed data to enhance the validity of corner
that utilizes machine learning to detect corners on the ima- detection by Tevfik Metin Sezgin. In the research, the
ges. They came up with a way to neglect unnecessary fea- detection process is based on scale-space analysis, process-
tures of the corners but keep the validity. These image-based ing deeper information so that it is able to tolerate the noise
methods have a kind of disadvantage that the 2D matrix and derive the corners preciously [9].
always includes empty value. The scenario in this research Recently, machine learning was introduced to solve tasks
can avoid this disadvantage because every external angle on for processing polylines. Long Zeng et al. proposed a machine
a contour is relevant. learning-based method and used a huge number of Human-
labelled datasets to train the model to detect corners on
Method for Detecting Corners in 3D Meshes drawing lines [10]; however, using numerous human-labeled
In addition to deriving distinct corners of contours on 2D datasets is too time-consuming to operate. A case about using
images, the topic of how to refine 3D is investigated by many polyline contour simplification for reconstructing the building
researchers. For instance, research by Yongzhi Wang et al. model was presented by Biao Xiong et al.; two algorithms in
investigating a method to simplify the mesh model based on their research depend on footprint maps and LiDAR to seg-
the triangle analysis [4]; their goal is to simplify but keep the ment contour in order to construct the models appropriately
salient features as much as possible. Another research for [11]. The research illustrates practical methods of constructing
processing mesh by Xiao-chao Wang et al. proposed a high-quality models. Although these mentioned methods are
method to detect corners on noisy meshes via normal tensor able to derive corners for different scenarios, they still include
voting and neighbor supporting [5]. In this research, we deal some parts to be improved when using them to simplify
with the 3D model as well; nonetheless, we choose polyline building models from photogrammetry.
Reconstructing Photogrammetric 3D Model by Using Deep Learning 297

First of all, mesh reduction methods are quite inefficient in the parametrical environment, facilitating designers
because they require sophisticated algorithms to process further.
complex geometrical data. On the other hand, in our previ-
ous research, we focus on polyline contours because most
variations of building models on vertical are slight. There- 2 Methodology
fore, a few contours of a building are adequate to represent
the major features of the building. The slicing-based method To simplify the model generated with photogrammetry
that we have presented is able to simplify the building model efficiently, three strategies are crafted and employed. The
significantly. The performance of our method is more effi- first thing that we are dealing with is simplifying the dataset;
cient than other methods because we implement neural most of the machine learning-based method use image or
networks to deal with the relatively simple data (a series of complex feature data to train their model. Instead, we use a
angles on polyline contours); which means that 2D geometry series of angles on a polyline as features to detect corners.
is adequate to facilitate the reconstruction process. Hence, Then, the labelled data to be used in our method are gen-
this research will focus on dealing with 2D contours as well. erated by computer automatically. Besides, a training gym is
Second, some polyline-based methods are able to seg- built to integrate dataset generator and deep learning model;
ment polyline accurately but they still have difficulties in the training gym will generate and train the model contin-
segment curved part or tolerance noisy lines. In modern uously until the result is good enough. The other part in this
architectural design, a curved form is inevitable to face; as a research is the reconstruction process based on the trained
result, this issue is necessary to solve. There are some model generated with the training gym (see Fig. 2).This
approaches requiring data other than geometrical features method is different from others’ because the model of this
such as speed data and footprint maps. This information method will detect corners according to the context of the
requires special equipment or approach, making it difficult to entire contour; however, the feature for detection is much
use by designers or architects. In this research, we propose a simpler than others’ methods; the characteristic will bring
method using relatively simple data form, but the results about the validity and efficiency when detecting corners.
would be accurate.
Third, it is great that machine learning has been utilized
in certain related researches; Nonetheless, their methods are 2.1 Contour Generator
processing unnecessary complex information and requiring a
lot of human-labelled data. In this research, we propose a Instead of using human-labelled data, the labelling process is
training gym system to address the issue. computerized in this research by using Grasshopper as a data
The purpose of doing this research is to improve a generator; the program keeps generating various contours
reconstruction method in our previous research. In the pre- (see Fig. 3). An universal training aid is designed for
vious research, we were able to simplify complex pho- enhancing the versatility when detecting corners; based on
togrammetrically models significantly yet the contour of its’ our experience when dealing with building contours, there
results are polylines. It might remain certain issues; for one are three kinds of variable usually appear on a contour that
thing, when the user wants to explode the model to get a might influence the detection, such as angle, curvature, and
curved wall, the result will be a lot of segments. And when a configuration. In addition, certain tiny parts might be
user views the results by wireframe or shaded mode, there neglected by the detection process; therefore, we design the
will be a bunch of irrelevant edges messing up the visual- training aid with a tiny part to enhance the sensitivity of
ization. Additionally, if users want to generate a pattern to paying attention to the details of a contour (see Fig. 4). In
design the façade on the model, these wall surfaces that are the dataset generator, the training aid will be varied by
represented as NURBS format will make them easier to use automatically setting different seeds for random components

Fig. 2 Training gym and

reconstruction process
298 J.-H. Hou and C.-L. Cheng

Fig. 3 Dataset generator

Fig. 4 Variable training aid for

versatile purpose

and influencing the result. The results are unified contours of Each contour will be divided into 100 nodes then all these
four variable parts. The dataset generator can perform effi- 100 nodes are connected as a new polyline. After exploding
ciently; hundreds of results can be generated in a second. the polylines, the result will be 100 segments. The means to
The image of dataset samples is visualized in Fig. 5. derive the series radians from it is extending each segment
In addition to these geometric characteristics, noise also line and measuring the internal radian between the segment
plays a crucial role to influence corner detection. To deal and the next one. We define the clockwise turn as positive
with the issue, a post-process to add noise is employed (see radian and the anticlockwise turn as negative radian (see
Fig. 6). The scale of noise is varied so it can deal with Fig. 7).
various rugged contours with various scales of noises. After
the post-process, the data is quite challenging for computa-
tional detection although it might not be an issue to a human. 2.3 Labelling and Processing Dataset
After generating the rugged contours, they will be outputted
with the corner labels for the next step. The method to label the dataset is to find the closest corners
from the original contour (see Fig. 8). The coordinates of
these corners can be derived by exploding the originally
2.2 Feature Extraction generated contour then the vertex will be used as the corner
coordinates. Then, we utilize the closest point component in
The feature for corner detection in our method is a series of grasshopper to gain the corresponding index number in the
radians (the range is from +p to −p), representing the turning dataset. The corner data is derived by shifting the series of
angle of each node. Although it seems that the node with a radians based on the index number; consequently, the crucial
high value of radian could be a corner, it always depends features to suggest it as a corner will appear at the start and
because of noise. The first step is to normalize the dataset. the end of the data. On the other hand, these features in the
Reconstructing Photogrammetric 3D Model by Using Deep Learning 299

Fig. 5 Various generated contours

Fig. 6 Adding noise on contours

to enhance the tolerance

on the dataset. That is to say that the neural network model

will consider the context of the entire contour, making the
estimation more thoughtful.

2.4 Training Gym System in Grasshopper

In this research, the neural network model is trained

automatically by the training gym in grasshopper. In the
training gym, it will keep generating new kinds of contour
Fig. 7 Using external angle as feature of contour to challenge the neural network model. There is an issue
occurring in the contour that most of the nodes do not
belong to the corner. If we use the original distribution of
middle of the data can offer information about noise or the data it will result in poor sensitivity for corner detec-
inferior features, suggesting it might be a corner. The corner tion. In order to solve this issue, we separate the corner
part is labelled as 0 or 1. If it is a corner data, it will be part and non-corner part into two sets, and then, the
labelled as 0; on the other hand, the data which does not non-corner part will be reduced to fit the number of corner
belong to the corner will be labelled as 1. So far, the parts (see Fig. 9), avoiding the result that the neural net-
information for training the neural network is adequate. The work keeps guessing anything as non-corner. Moreover,
neural network will be able to figure out the relationship during training, the dataset will never repeat to avoid
between the corner number and the series of radians based overfitting; it is a significant characteristic in our method
300 J.-H. Hou and C.-L. Cheng

Fig. 8 Contextual dataset

Fig. 9 Training process

that it truly takes advantage of utilizing a generative design order to make the neural network model able to be employed
tool for machine learning. in the reconstruction process independently.
The neural network model in the training gym is built The following is the image showing the progression of
with the plugin GH-CPython and PyTorch. The rest of the learning (see Fig. 10). In each epoch, the neural network
parts such as dataset generator, noise generator, iterative model will be tested, visualized with a colour gradient and
mechanism, are made of original Grasshopper components. vertical bars. The warmer the colour is, the possible it is a
The neural network that we apply is a simple multiple layers corner; besides, the value is visualized by the height of a bar.
with three hidden layers; the output layer has only one node Therefore, the improvement in each epoch can be observed
with a sigmoid activity function to restrain the number in the intuitively. In the beginning, the result is basically random;
range from 0 to 1. We anticipate that a neural network is as time went on, the corner detection accuracy increases so
adequate for this task because the model needs to focus on that we can use the neural network in the reconstruction
the series of radians. The task is much simpler than the process.
multiple layers neural network that is presented by
Geofrey E. Hinto; besides, the neural network that Geofrey 2.4.1 Reconstruction Process
used is able to classify 10 kinds of digits from 2D images The reconstruction process in our method utilizes the
with 784 (28 * 28) pixels [12] while our neural network neural network model trained by the training gym. The first
deals with only 100 input data and classifies 2 kinds of step is corner detection. The contour to detect should be
situations (corner or non-corner). As a result, we consider normalized as a 100-radians format, and then the data will
that it is unnecessary to increase the complexity of the neural be evaluated by the neural network model. There is only
network like certain fancy methods. one output of the neural network, whether it is a corner or
Since the training gym is set, the rest is to let the process not. In the defining step, we set a threshold value to decide
progresses toward perfection. In fact, it can get a nice result where to segment (see Fig. 11). The original contour will
in a half-hour. During the training process, the neural net- be separated into individuals; subsequently, each polyline’s
work model will be saved in standard pkl format (a format to nodes will be used as control points of individual Bézier
store artificial neural network of PyTorch framework) in curves.
Reconstructing Photogrammetric 3D Model by Using Deep Learning 301

Fig. 10 Learning progression

Fig. 11 Segmentation based on

corner estimation

a universal polyline segmentation tool. The main charac-

3 Results teristics of the result generated by our method is that it can
significantly reduce the number of faces and rebuild the
A training dataset generator and a training gym system are contour as properly segmented surfaces efficiently. A part of
proposed in this research; it can constantly improve the the efficiency is caused by the fact that the neural network
neural network and never repeat the data to avoid overfitting model in our method is speeded up by GPU.
during training. Moreover, a kind of training aid is designed
for simulating any circumstance when detecting corners on
building contours. In our method, the neural network and 4 Discussion
data format are designed to be as simple as possible for
efficiency. Our method simplifies the process of training a neural net-
The capability of our method has been tested and work for a building model reconstruction process; Although
approved that it is able to detect corners on various contours. the neural network is trained with the dataset generated by
Even though the neural network is trained by our training our dataset generator, it is adequate to detect corners on
gym system instead of real data, it can find corners from real various unseen polylines. It reveals the potential to use a
data successfully (see Fig. 12). generative design tool to facilitate the training process. Our
Additionally, we have tested them for non-building-like method is helpful when dealing with models generated
and concave contours (see Fig. 13). The following is the through photogrammetry in some aspects. First, our method
comparison between the other two existing methods to brings about the neat visualization of building models.
reduce polyline in Grasshopper. Our method can reconstruct Secondly, since the surfaces on the building models are easy
the contours neatly, reducing the number of faces signifi- to derive, the designer can utilize a parametric tool to
cantly. Furthermore, our method can segment the open analyse or generate façade on these segmented wall surfaces.
polylines as well (see Fig. 14); it might be qualified to use as In addition, these surfaces of the wall make them easier to
302 J.-H. Hou and C.-L. Cheng

Fig. 12 Finding corners on real data

Fig. 13 Using various contours to test the model

Reconstructing Photogrammetric 3D Model by Using Deep Learning 303

Fig. 14 The result of

reconstructing open contour

Fig. 15 Taking advantage of our

method for parametric design

export to other geometry formats, such as Revit, and Auto- this method to be an incremental process method. The
CAD; designers are no longer struggling with complex mesh incremental process is like subdivision; however, the division
issue. The following photograph is a set of examples of is based on our deep learning-based segment method; by
demonstrating how to generate a parametric pattern on these doing this, the resolution will increase exponentially in a few
surfaces (see Fig. 15). Despite the fact that our method is iterations. On the other hand, in this research, the way to
suitable for most polylines, it has a reservation about the reinforce the sharpness on the corner has not been investi-
result; these control points of each surface belong to an gated. It has been considered for future improvement.
unsegmented original polyline. Therefore, when a user is not There is another application for reverse engineering that
satisfied with some result, these original nodes can be could utilize our method (see Fig. 16). It could cope with the
derived from the control points of the surface. In the future, issue caused by the fact most simplify and segment methods
this method could be utilized as a standard means for are dealing with a huge number of triangles on mesh sur-
reconstructing the photogrammetric mesh model. faces. Instead, our method is based on the contour; our
method is able to detect several corners from one slicing at
the same time. In the future work, how to optimize the
5 Future Work slicing process will be explored. The similar approach to
build a 3D model is presented by Carswell et al. [1].
There are certain issues in our method that can be improved in Although the mesh model generated by the photogram-
the future. First, there are a few defects might happen around metry can be significantly simplified through our method,
the corner that sometimes remains tiny pieces segments. the chaos of texture is not yet solved. The issue might be
Second, the resolution of detection in our method is 100 solved by using these surfaces generated with our method to
nodes; although it is adequate for most cases, there are some resample the texture. Once the issue is solved, it will be able
rare cases, might require higher resolution corner detection. to facilitate several usages such as movie and game
The issue might be coped with through the way that makes industries.
304 J.-H. Hou and C.-L. Cheng

Fig. 16 Taking advantage of our

method for reverse engineering

In term of algorithm feature engineering, we utilize radian 4. Wang, Y., et al. (2019). Fast mesh simplification method for
as the feature in this method and count on deep learning to three-dimensional. Scientific Programming, 3, 1–12.
5. Wang, X., et al. (2012). Feature detection of triangular meshes via
tackle with the noise. The other possible method that we neighbour supporting. Journal of Zhejiang University: Science C,
speculate is using discrete Fourier transform to derive fea- 6, 440–451.
tures and filter out noise like the concept of signal processing 6. Wolin, A., et al. (2008). ShortStraw: A simple and effective corner
in pre-treatment phase then using deep learning as well; it finder for polylines, sketch based interfaces and modeling (pp. 33–
40).
will be investigated in the future work. 7. Zhang, S., et al. (2015). Corner detection using arc length-based
angle estimator. Journal of Electronic Imaging, 24.
Acknowledgements The research is partially supported by the Min- 8. Spoerhase, J., et al. (2019). Simplification of polyline bundles.
istry of Science and Technology under Grant Number MOST 35th European Workshop on Computational Geometry, Utrecht,
109-2634-F-009-015 through Pervasive Artificial Intelligence Research The Netherlands, arXiv preprint arXiv:1907.05296
(PAIR) Labs, Taiwan. 9. Sezgin, T. M. (2001). Feature point detection and curve approx-
imation. Department of Electrical Engineering and Computer
Science.
10. Zeng, L., et al. (2019). Robust corner and tangent point detection
References for strokes with deep. Graduate School at Shenzhen, Tsinghua
University.
1. Carswell, D., et al. (2006). 3D solid fin model construction from 11. Xiong, B., et al. (2016). Footprint map partitioning using airborne
2D shapes using non-uniform rational B-spline surfaces. Advances laser scanning data. ISPRS Annals of Photogrammetry, Remote
in Engineering Software, 37, 491–501. Sensing & Spatial Information Sciences, 3, 241–247.
2. Buades, A., et al. (2018). Contours, corners and T-junctions 12. Hinton, G. E. (2007). Learning multiple layers of representation.
detection algorithm. Image Processing on Line, 24–36. TRENDS in Cognitive Sciences, 11, 428–434.
3. Rosten, E., et al. (2008). Faster and better a machine learning
approach. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 32, 105–119.

View publication stats