Automation in Construction 86 (2018) 11–32

Contents lists available at ScienceDirect

Automation in Construction
journal homepage: www.elsevier.com/locate/autcon

Interior construction state recognition with 4D BIM registered image T

sequences
⁎
Christopher Kroppa, Christian Kochb, , Markus Königa
a
Computing in Engineering, Ruhr-University Bochum, Germany
b
Intelligent Technical Design, Bauhaus-Universität Weimar, Germany

A R T I C L E I N F O A B S T R A C T

Keywords: Deviations from planned schedules in construction projects frequently lead to unexpected financial dis-
Progress monitoring advantages. However, early assessment of delays or accelerations during the phase of construction enables the
Building information modelling adjustment of subsequent and dependent tasks. Manually performed, this involves many human resources if as-
Computer vision built information is not immediately available. This is particularly valid for indoor environments, where a
Indoor construction
general overview of tasks is not given.
Image registration
In this paper, we present a novel method that increases the degree of automation for indoor progress mon-
Image recognition
itoring. The novel method recognizes the actual state of construction activities from as-built video data based on
as-planned BIM data using computer vision algorithms. To achieve that, two main steps are incorporated. The
first step registers the images with the underlying 4D BIM model. This means the discovery of the pose of each
image of a sequence according to the coordinate system of the building model. Being aware of the image origin,
it allows for the advanced interpretation of the content in consecutive processing. In the second step, the relevant
tasks of the expected state of the 4D BIM model are projected onto the image space. The resulting image regions
of interest are then taken as input for the determination of the activity state.
The method is extensively tested in the experiment section of this paper. Since each consecutive process is
based on the output of preceding steps, each process of the introduced method is tested for its standalone
characteristics. In addition, the general manner of applicability is evaluated by means of two exemplary tasks as
a concluding proof of the success of the novel method. All experiments show promising results and direct to-
wards automatic indoor progress monitoring.

1. Introduction finishing works represent a high-ranking section of every construction

Building Information Modelling (BIM) is an essential step to the
digital management of construction projects. One big advantage of BIM
is the holistic collection, linking and provision of data for different
planning, construction and operation tasks. In the context of con-
struction management, the application of 4D building models by linking
activities of a schedule with corresponding building elements is very
common. Based on 4D building models the construction sequence can
be analyzed and progress monitoring can be supported.
In current practice, progress monitoring is mainly performed
manually. To this end, weekly or daily paper-based progress reports are
collected by field personnel by hand [1]. This procedure leads to a high
workload of the field personnel to fulfil the need for a reasonable fre-
quency of inspections. Hence, manual progress monitoring involves
either a massive amount of human intervention or inspection updates
cannot be performed as frequently as required. In particular, interior
project that is sensitive to schedule disorders. The average share in a
typical total budget lies between 25% and 40% [2]. As a result, a higher
degree of automation in progress monitoring would result in reasonable
cost savings. The derivation of the actual state requires the automatic
comparison of the as-planned information that is present in BIM with
the actual as-built state of the building.
Recently, the research community has been investigating methods
that aim to achieve a higher degree of automation in progress mon-
itoring. Several approaches and studies that address the comparison of
as-built and BIM-based as-planned data for this purpose have been
presented. For example, sensing technologies like Radio-Frequency
Identification (RFID) [3,4], Ultra-Wideband (UWB) [5–8]), Wi-Fi [9],
ZigBee [10], laser scanners [11,12], Global Positioning System [13],
image [14–19], video [20–23], and depth image [24] capturing devices
are used to obtain data of the as-built state.
Each of these technologies has its advantages and limitations in a

⁎
Corresponding author.
E-mail address: c.koch@uni-weimar.de (C. Koch).

https://doi.org/10.1016/j.autcon.2017.10.027
Received 16 May 2017; Received in revised form 25 October 2017; Accepted 28 October 2017
Available online 09 November 2017
0926-5805/ © 2017 Elsevier B.V. All rights reserved.
C. Kropp et al.
specific environment. For instance, radio-based technologies are able to
determine the presence of objects through walls and can, therefore, be
deployed indoors, but are limited to presence detection or localization.
Contrarily, stationary line-of-sight 3D laser sensing technologies obtain
accurate measurements within general site scenes, but are too inflexible
for indoor applications. Although depth images from portable devices
are more flexible, they are currently limited in range. The main aspects
of BIM-based automation of progress monitoring for a specific sensing
technology and a target environment that need to be considered are:
1.1. Registration of as-built data to BIM model
Mapping of the sensed data to the BIM object instances and asso-
ciation to scheduled activities. The comparison of as-built and as-
planned states is only possible if registration is performed. Objects from
the BIM model need to be identified in the raw data. Detection solely of
the type or class is not sufficient for precise progress statements. When a
column is detected, the determined state needs to be associated with
one specific instance of all available columns.
1.2. Activity coverage
The raw data delivered by the sensing technology must have the
capability of supporting the state detection of all activities that are
supposed to be performed in the actual environment. Sensing technol-
ogies like RFID or laser scans are not able to determine states defined by
sole appearance changes, for example, wall painting.
1.3. Coverage of the location's spatial structure
The sensing technology must meet the requirements of the en-
vironment's structure. It needs to handle permanent and temporary
construction obstacles and distances of the actual environment. For
instance, utilization of common RFID technology for triangulation of
objects on construction sites delivers positions with an error of about
2 m [25], resulting in not being able to determine whether materials are
already mounted or still stored. However, low distance RFID might help
detect the presence of materials or construction objects within a range
of up to 1 m (ISO/IEC 18000–3), which is useful in interior finishing
inspections.
1.4. Dependency on infrastructure
Sensing technologies may depend on infrastructure in order to
collect data. For the usage of these technologies, it needs to be con-
sidered if the state of the building provides the required infrastructure.
For instance, Wi-Fi-based localization of objects by triangulation as-
sumes the presence of appropriate hardware and a dense distribution in
every part of the building, which contrasts with the stage of interior
finishing.
Currently, there exists no proper solution that covers the difficult
situation imposed by view obscuring walls and registration issues along
with the dense repetition of the similar (but distinct) building elements.
This makes registration challenging for line-of-view sensing technolo-
gies and requires complex infrastructure for radio-based technologies.
Moreover, there are challenges associated with the wide range of con-
struction activities existing indoors.
Vision-based as-built data acquisition methods can meet all the four
main aspects leveraging computer vision and image processing methods
based on sequential imagery. Registration is possible on discovered
correspondences between the image and the known geometry of the
digital building model. A wide range of activities can be covered by
appearance-based processing methods or scene and object structure
recognition approaches. In addition, image sensing devices are highly
portable and can easily elude barriers for a better view and do not
depend on infrastructure. In Table 1, sensing technologies are rated for
12
Automation in Construction 86 (2018) 11–32
each of the four stated main aspects regarding BIM-based progress
monitoring.
In this context, vision-based methods appear to represent the ap-
propriate means to target indoor construction state recognition.
However, current methods for indoor construction state recognition are
independent of, and not associated with 4D BIM. This paper presents an
approach to automate 4D BIM based progress monitoring.
2. Related work
Automation in progress monitoring has been the subject of research
in recent years. Various sets of different approaches have been pre-
sented that are based on various acquisition and processing techniques.
In this section, a summary of present approaches is given and the
technologies are introduced that this work refers to.
2.1. Data acquisition
The acquisition of as-built data is a fundamental step to ensure ac-
curate progress monitoring. Since the choice of the acquisition tech-
nique influences all subsequent steps, the applied technique needs to be
well-chosen. Thus, the capabilities of the technique and the resulting
raw as-built data need to be analyzed. There are several surveys that
give an overview on available techniques and its applications in-
corporated for progress monitoring [26–28]. These overviews mainly
list Radio-frequency Identification (RFID) technologies, laser scanning
methods, as well as image and video capture systems as promising
technologies for as-built assessment. With RFID, construction elements
can be inexpensively and robustly tagged due to low sensor complexity,
but requires mounting tags and does not enable determination of con-
struction state [29]. Applications for radio frequency based technolo-
gies during the construction of buildings are the localization of mate-
rials [4,29–32], personnel [33] and inventory [34]. The disadvantages
of RFID make these technologies unattractive for detailed progress
measurement and RFID is seen to be a supporting approach for other
techniques in recent approaches for construction progress measurement
[35]. Further radio frequency based technologies are utilized to capture
the as-built data, such as global navigation satellite systems (GNSS) and
Wi-Fi. For indoor construction scenes, GNSS are not available because
of the missing line of sight [36] and Wi-Fi is usually not present in
buildings under construction. Laser scanning offers high accuracy point
cloud measurements, but consumes a certain amount of time for each
scan [37] and/or has problems at spatial discontinuities [38]. Besides
high costs for laser equipment, heavy involvement of personnel induces
additional expenses. However, several approaches apply laser scan
technology for 3D object recognition using BIM and for comparison of
information inferred from scanned data with designed 4D BIM
[12,39–41] to track progress. Another promising form of as-built data is
images and videos. Images and videos are acquired in several ways. The
advantages of images and videos as the basis for progress measurement
lie on low costs, short time, and the ease of collection. Continuous
advents of high definition cameras and high performance processing
units signify the potential of these types of data with an enhanced ac-
curacy of information and fast processing. The technology's main ad-
vantages are low cost and portability. Additionally, no special knowl-
edge is needed to operate and taking photos of construction sites for
documentation reasons is a common practice. The camera system may
either consist of a monocular [42] or a stereo sensor [17]. The per-
spective can be fixed and known [16,43–45], or uncertain [37,46–48].
Besides the approach of using single photos, videos have been recently
used to capture as-built scenes [18,49]. Occlusions may occur due to
the line-of-sight characteristic, but can be handled by traversing the
indoor scene. Both automated registration and recognition can be
achieved by computational processing of acquired data.
C. Kropp et al.
Table 1
Rating of main aspects regarding interior finishing for different sensing technologies (with + as appropriate and – as inappropriate).
AutoID WiFi/UWB/ Laser scans Unregistered images
GPS
Registration + + + −
Activity-coverage − − − −
Spatial-coverage + + − −
Infrastructure independent + − + +
2.2. Image-to-image registration
In this subsection, an overview of different vision-based methods
that estimate camera motion based on the registration from one image
to another through a scene is presented. The following approaches of
Visual Odometry (VO), Visual Simultaneously Localization and
Mapping (VSLAM) and Structure from Motion (SFM) are introduced.
Although certain similarities between these approaches are present,
research in the field of vision-based localization spreads into specialized
branches that each make distinct assumptions.
SFM reconstructs the camera motion and the structure of the scene.
The order of the input image set is usually not regarded, which leads to
an interrelation of all images with each other. One key that has led to
automatic SFM was the introduction of advanced feature descriptors,
which allow for the robust matching of unordered image sets, since they
provide invariance in terms of scale and rotation. By a final global
optimization procedure, regarding the minimization of the re-projec-
tion error, the whole model is adjusted. This optimization procedure is
usually performed with (sparse) bundle adjustment, which adjusts cam-
eras and features in space by an overdetermined system based on
multiple observations in various views.
VO has been introduced by [50] and is described as an algorithm
that estimates the trajectory of a camera traversed through its en-
vironment. The basic approach is available in a monocular binocular
system, but other approaches consider more simultaneously analyzed
camera observations [51]. VO incrementally updates the new camera
pose with each new image. After the track of a feature is lost, i.e. oc-
clusion occurs or the field of view is exited, its 3D information is dis-
carded [52]. In order to reduce the drift that occurs during the incre-
mental addition of new images, some approaches apply regional
optimization on a window of the last couple of frames [53].
VSLAM is a vision-based method that represents a branch con-
sidering both estimation of the camera location whilst building a map.
With this technique, a global map is built that consists of landmarks,
which represent the estimated position of feature correspondences in a
sequence of image frames. The information that is stored in the map
allows for active search of feature recognition. Since landmarks are kept
in the map although they are not visible (i.e. not in the field of view or
simply not recognized yet), they can be recognized in future image
frames. Some references also call VSLAM a real-time or online version
of SFM [54] due to the fact that the structure of the environment is kept
in the map. Applications in SFM often aim for reconstruction of objects,
whereas VSLAM keeps the landmarks in the map for the purpose of
recognition of the same pose, i.e. for loop-closures of the path. Hence,
VSLAM only needs to involve as many features as necessary to estimate
the localization, which can end up in very sparsely populated maps. An
approach that targets the characteristics of building structures is pre-
sented in [55] and is based on lines of three dominant directions. This
approach shows promising results in controlled environments, but bears
several unsolved problems reported with the recognition of lines.
2.3. Image-to-model registration
Registering acquired data to a model combines an as-is view with
as-designed information. This states the foundation for comparisons
13
Automation in Construction 86 (2018) 11–32
Registered, unordered images Registered image sequences/videos
− +
+ +
− +
+ +
between both sides. In recent years, several research contributions have
been presented that address machine-based approaches for the regis-
tration of the sensed data and the corresponding model data. These
approaches leverage correspondences of different geometric primitives
(points/lines/planes) to determine the translation, rotation and scale in
reference to the model. On the one side, there are assisted approaches
that depend on manual intervention for the registration. On the other
side, there are approaches that work automatically, but have restric-
tions. However, some approaches are already being used in outdoor
progress monitoring research projects.
In recent years, several research results have been published dealing
with the registration of acquired site data with as-planned building
models. In their experiments, predefined viewpoints for cameras are
used. Some conceptual works target towards automatic comparison of
images taken from the construction site with 4D building models with
static cameras. The position and orientation of the cameras are assumed
to be known. However, the comparison still requires manually per-
formed steps, like manual camera installation on the site [56,57],
correspondence establishment [44], manual work package creation
[58]. In Lukins et al. [57], a fixed camera is used to classify changes in
the scene appearance as structural events related to the building model
using the pose estimation of David & DeMenthon [59]. Lukins et al.
[57] report about failed alignments due to building model complexity
and cluttered scenes, and thus perform additional optimization. The
approach introduced in [16] obtains prior knowledge of building
components and their occupancy within a scene from a 4D building
model registered to the camera. The paper argues that fixed cameras on
a construction site for tracking progress are disadvantageous due to
inflexibility in response to changing structures. In contrast to the pre-
viously described approaches, the approaches in [46] uses a large set of
unconstrained images that are manually registered to a 4D BIM model
with an automatically reconstructed point cloud. In [40], a laser scan
derived point cloud is registered to a 3D building model. Hereby, the as-
planned model is formatted as stereo lithography, before the 3D model is
registered with the scanned model using manually selected point cor-
respondences. This approach is extended in [41] for full automation,
where the registration is split into a coarse and a fine registration step
based on iterative closest point (ICP). The approach of Kim [60] deals
with the fully automated registration similar to the previous approach,
but re-samples the point cloud and the 3D building model data to
provide a common data layout. There are several cases in which this
approach is not applicable [61]. Moreover, the ICP algorithm needs a
good initial guess to converge [62]. In this context, [63] conducted a
survey on methods for as-built generation of BIM models including the
setup where as-designed BIM models are available.
Other research contributions give solutions to the problem of
camera pose estimation for each image. Pose estimation approaches
that work with correspondences between the model features and their
projections to the image sensor are also called the Perspective-n-Point/
Line (PnP/PnL) problem. A set of three correspondences is the smallest
subset of correspondences used in solving the problem. A higher
number of features used in algorithms typically improves the accuracy
of the solution. However, in case the correspondences contain outliers,
the accuracy is negatively affected [64]. In Chen [65], a general solu-
tion for the PnL problem is presented. Dhome et al. Liu et al. [66]
C. Kropp et al.
presented an approach that delivers a unique solution using eight line
correspondences, which Ansar et al. [67] decreases to four. Mirzaei
et al. [68] try to reduce the effect of outliers. Recently, progress has
been achieved by [69] in terms of robustness and accuracy with im-
precise correspondences. The approach of [64] is based on lines only
and is declared as very robust to outliers, but requires at least nine line
correspondences, whereas [70] considers points and lines. Fischler and
Bolles [71] introduce the RANSAC paradigm and show its abilities for
outlier rejection by determining the pose of a camera for cartography
purposes on the basis of the PnP problem for given correspondences.
Furthermore, there exist image-to-model registration approaches
that establish correspondences in addition to the estimation of the pose
[72–81]. Some approaches incorporate point features, others use line
features or both. The approach presented in [72] combines an algo-
rithm to establish the correct point correspondences [73] and pose es-
timation proposed in [74]. It performs a local search and determinis-
tically converges to a local minimum solution for an initial guess of the
object pose. This does not guarantee a globally optimal solution. Hence,
a random start scheme is proposed, when no prior pose estimation is
provided. This increases the search space significantly, but can find a
globally optimal solution. This approach is extended for the case of
coplanar model points [75], but is only applicable for this case. The
application is modified to work for line features in David [76]. The
authors state that for the random start approach fewer initial guesses
are required than with the use of point features. Diaz and Abderrahim
[77] modify the initial algorithm in a way that it fits the requirements
of 3D object tracking. Other similar approaches try to reduce the search
space by different strategies, using gaussian mixture models [78],
branch & bound [79], evolutionary algorithms [80], and differential
evolution [81]. When prior pose information is provided, they are more
robust to occlusions, clutter and repetitive patterns [78].
2.4. Progress recognition
The research community has shown the promising applicability of
vision based recognition and developed several approaches that focus
on the vision-based recognition of construction components and activ-
ities.
Only a few approaches are presented that consider the detection or
recognition of construction components. In Roh et al. [1], object de-
tection is considered especially for indoor purposes of progress mon-
itoring. The approach detects air condition equipment and is in general
only applicable for small objects with distinct characteristics. Kropp
et al. [84] present an approach that aims to recognize structural fin-
ishing components. The approach is evaluated with the recognition of
heating devices and is able to differentiate between different activity
states. The cascaded material state determination approach presented
in Kropp et al. [83] is extended by Hamledari et al. [82] to detect
electrical outlet, but is not linked to BIM and thus not able to perform
recognition nor identification of components.
Some work has been carried out to include construction work
packages and activities into the recognition process to detect the pro-
gress and change of construction elements. Work package and object
progress (or change) detection or recognition are considered in those
approaches [43,57,85]. Two major trends can be observed that consider
the estimation of progress of work packages and activities. The existing
approaches either analyze the result of the work by means of the re-
sulting object or they do track the labor itself and derive completion
according to the type and intensity of actions. The approaches pre-
sented by Ibrahim [85] and Zhang [43] handle visual work package
progress inspection by present 3D models and a fixed camera. These
approaches investigate images of a work package under construction
from the same view over the lapse of time. Nevertheless, it is reflected
that this approach has several challenges by heavy clutter, uncontrolled
and unforeseeable lighting variations, dynamic target appearance and
frequent occlusions [85]. Additionally, there is the concern of coverage
14
Automation in Construction 86 (2018) 11–32
with the fixed camera and that it is not feasible to confirm every
component. Furthermore, limited success rates are supported [43].
Moreover, there is only completion of whole components, but no in-
dication of percentage completion. It is concluded, that total automa-
tion is not possible with this approach and only visible changes are
detected. In addition, the approach is limited since it is not generic
enough for other activities.
In other research contributions, vision-based activity recognition is
indirectly guessing the work in progress while workers are observed
permanently by fixed on-site cameras [6,24,86]. In Chen et al. [6],
UWB is used to fuse the camera images with three dimensional space,
whereby depth images are considered in the approaches of Khosrow-
poura et al. [24] and Escorcia et al. [86]. Progress monitoring per-
formed with these approaches involves a lot of time applying the
camera system on one task. The workers need to be observed perma-
nently by the camera to guess the current activity. For each application,
in particular for progress monitoring, the necessity of the resulting
discretization level needs to be minded. Additionally, in practice,
privacy policies in many countries will not allow for the observation
and monitoring of the workers at any time. For indoor progress mon-
itoring the present indirect progress estimation approaches do not seem
to be appropriate. They are not applicable in a common framework for
automation in progress monitoring with a single and common tech-
nique. Instead, directly deriving the state from one moment of ob-
servation seems to be a more practical approach.
Although 4D BIM offers sources of rich information that may sup-
port the recognition of the current state of a building component or
activity, research efforts in vision-based state recognition are mainly
independent of 4D BIM. This particularly affects the recognition of
objects and materials. Hence, their association to further information
accessible by the link to 4D BIM and its objects and tasks is not con-
sidered and is insufficiently investigated in research. Furthermore, a
wide range of materials and structural objects needs to be recognized
for indoor progress monitoring, which is not covered by just a single
recognition method.
2.5. Line segment extraction
Line segment extraction is a task within the field of image proces-
sing that detects appearances of straight contours. The change of pixel
intensity values from high to low and vice versa states the basis for
processing algorithms. The whole images are scanned for all possible
lengths and poses of line segments. Line segment detection algorithms
follow different strategies to select gradient pixels for potential line
association and connect them to segments. This requires the para-
metrization of each algorithm, which influences the output result. A
fundamental parameter used in many algorithms is the gradient
threshold that discards pixels, when the gradient value is too low.
Sensitive thresholds regard low gradient line contours, but include false
lines due to noise and clutter, and are accompanied by a long runtime.
Contrarily, thresholds that only regard high gradients do not detect a
complete set of lines. In recent years, state-of-the-art line segment de-
tection algorithms have been presented [87,88]. Grompone von Gioi
et al. [87] have presented a line segment detector that works on the
directions of the gradients for the environment of each pixel and con-
nects those that are adjacent and share a similar angle. The line seg-
ment detector based on the work of Akinlar and Topal [88] computes
anchor points that are in contrast to surrounding pixels and connects
the found points to lines, when they are connected by pixels with fitting
gradients. Other works focus on the recognition of lines from image to
image concerning their appearances for matching purposes [89–91].
These approaches not only regard the pixels that state the line segment,
but also consider the surrounding pixels along the line to create de-
scriptors. The disadvantage using those descriptors is the dependency
on the line length. Line segments cannot be recognized on basis of the
descriptor when the observable line length varies.
C. Kropp et al.
2.6. Problem statement and objectives
For controlling project costs and time, on-site inspections are ne-
cessary to determine the current progress of interior finishing works.
Currently, there are no methods available that sufficiently assist and
continuously automate the progress monitoring. Existing methods
mostly deal with outdoor construction sites. Furthermore, these
methods are neither directly applicable indoors nor validated for in-
terior sites. Moreover, they are predominantly concerned with accurate
geometry reconstruction and subsequent as-design comparison, but lack
the capability of activity state recognition and project delay prediction.
Furthermore, there is only a small amount of building activities covered
and no approach considers a generic framework for involving the var-
ious set of building activities indoors. Additionally, BIM data is used for
as-built/as-planned matching and for visualization purposes. However,
none of the concerned approaches considers the usage of 4D BIM data
to support activity state determination referring to the 4D BIM sche-
dule. Moreover, research in the field of indoor navigation does not
provide solutions that match the required aspects stated in the in-
troduction section. State-of-the-art approaches like the work of Van
Opdenbosch et al. [92] are based on infrastructure for a server con-
nection as well as on obtaining information about a rough localization
and a prior information of the scene in terms of laser scans and images.
This does not allow for the deployment in frequently changing scenes
like congested interior construction sites.
To overcome the mentioned limitations, the conceptual framework
proposed in this paper addresses the following main research objec-
tives:
- Leverage of 4D BIM information to enable smart methods for video
based progress monitoring;
- Robust registration of acquired as-built data with the underlying 4D
BIM model;
- A recognition approach that is able to consider a vast set of con-
struction activities contained in 4D BIM models.
3. Methodology
In this chapter, a novel method that targets a higher degree of au-
tomation for indoor progress monitoring with the consistent integration
of as-planned 4D BIM information into the as-built inspection process is
presented. For the first time, a coordinated set of processes is aligned
that regards challenges and requirements for indoor scenes. The
method considers extensive access to information present in the 4D BIM
model in order to enable and support the progress monitoring pro-
cesses. This leads the automation for indoor progress determination to a
degree that reduces human intervention to a low level.
To achieve this, the underlying concept of the new method regards
all processes, from data acquisition to activity state recognition. A
workflow of the concept is illustrated in Fig. 1. The as-built data is
captured with video cameras, the status of construction activities is
determined by analyzing the content of images. Therefore, a variety of
algorithms from the fields of computer vision, image processing and ma-
chine learning are used for different purposes to ensure the mainly au-
tomatic workflow. As-built video frames are first registered to the BIM
model and in the end the state of activities is recognized. The identified
activity states can be used as an information resource for assisted de-
cision making, which may result in re-scheduling. The updated sche-
dules can then be exploited for subsequent inspections.
Taking the as-built video frames as input, the first block of processes
regards the registration of the image data to the building model. This
reveals the pose of the camera for each captured frame within the
building. To achieve that output of the registration block, an initial
registration is performed. The initial registration represents the only
part with manual intervention during the whole inspection and aims to
provide a rough camera pose. This rough camera pose is then
15
Automation in Construction 86 (2018) 11–32
elaborated into a finer camera pose estimated relative to the building
model geometry. Video frames that do not contain sufficient geometry
of the building model for fine registration are forwarded to a rough
motion estimation process that bypasses the lack of the visibility of
actual building model geometry until fine pose estimation is available
again. This way, switching from fine to rough estimation of the camera
poses enables the automatic registration from the point of initial re-
gistration. After all video frames are processed this way, the registration
of as-built data to the building model is complete.
After registration, the recognition block represents the part of the
method that determines the actual state of activities. Based on the re-
gistered video frames, the single images are analyzed in an appearance-
based approach. First, the search space for objects related to relevant
activities is reduced to avoid analyzing negligible video frames. In ad-
dition, only image areas that do contain the projected model objects are
further regarded for analysis. Second, a rectification of the regarded
image regions is performed to enable a more homogeneous basis of the
image data for the succeeding state recognition. During state recogni-
tion, the input image regions are classified to identify the actual state of
the observed activity.
3.1. Registration
The registration intends to make the as-built acquired image data
comparable to the as-planned information contained in the BIM model.
Thereby, registration means to reveal the pose of the camera that
captured an image frame relative to the coordinate system of the
building model. Ideally, the registration is performed completely au-
tomatically and accurately. Since information extraction solely from
images does not always allow to establish a distinct assignment to a
pose and accurate positioning systems are not available indoors,
manual assistance is assumed within the proposed approach. The ap-
proach represents the idea that the geometry of the currently expected
building model can be leveraged to relate the observed scene in the
assessed images to BIM. However, it is not assured that images from
indoor scenes contain structures that are also part of the BIM model.
Thus, the proposed approach regards a combination of different com-
puter vision algorithms to manage this circumstance.
The registration of each frame is illustrated in Fig. 2. This workflow
describes the processing of an arriving image until a pose is found.
Determined poses of succeeding frames are used as a guess for the
following frames. This enables the automatic registration of the video
frames after an initial registration. The initial registration is performed
for the first frame and delivers an estimation of the first camera pose.
This pose is used for the automatic fine pose estimation process. In case
a fine pose can be obtained from the input image, the image is suc-
cessfully registered and forwarded to the activity state recognition.
When no fine pose could be estimated, rough motion estimation de-
livers the needed pose. For the next frame of the sequence, the pose is
provided as an initial guess. The next subsections give further details on
the registration.
3.1.1. Acquisition
To create as-built data of the current construction status, video
frames are captured with a monocular camera system within the con-
struction scene during inspections. Since the image frames are analyzed
according to the result of construction activities, the inspection is likely
to perform well when results are visible and works are currently not in
action. A video is recorded while the camera system is traversed
through the interior of the building. The personnel that carries out the
inspections needs to be aware of the abilities of trajectory reconstruc-
tion based on computer vision and adjust motions accordingly.
3.1.2. Initial registration
The initial registration delivers a rough estimation of the pose at the
beginning of an image sequence. This is necessary, since the concept
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Re-Scheduling

4D BIM

Building Object
Schedule
model Catalogue

Geometry Relevant Objects

Acquisition
Registration Recognition
As-built Initial Registered Search Space Identified Assisted
Video Frames Registration Video Frames Reduction Activity State Decision Making

Rough Fine Pose ROI in Video State

Camera Pose Estimation Frames Recognition

Rough Motion Fine Image Rectified

Estimation Camera Pose Rectification ROI

Fig. 1. Concept overview for video-based progress monitoring.

further assumes that the structure of the building is ambiguous in dif-
ferent rooms and several observations may look similar. The concept
assumes that the camera poses of consecutive images can be recovered
automatically. Thus, an initial registration covers the first frame in the
image sequence (see Fig. 2) and is performed by manually navigating
through the digital building model. Therefore, the building model is
superimposed onto the regarded image as a wire frame model in an
augmented reality (AR) manner. The user selects the view on the
building model that visually matches the actual camera view. The in-
itial registration is only performed once at the beginning of each in-
spection.
3.1.3. Line extraction
To find correspondences between the model and the as-built scene,
a line segment extraction approach that provides line candidates of the
model lines within as-built images is developed. This novel method
exploits the presence of a rough camera pose. The underlying algorithm
expects the image and a set of model lines projected to the image space
as input, and delivers image line segments as possible candidates for
each model line. Instead of scanning the entire image for line segments,
the appointed method leverages the projected lines from the building
model with an estimated pose. The approach further scans an image
region of interest (ROI) with a directed filter. The ROI is set according
to the guessed line appearance along the perpendicular of the projected
model line. The projected building model lines define a search region
for each line. The search region contains a buffer that considers the
uncertainty of the pose. The model line projection allows to reduce the
search space for each line significantly in terms of the analyzed image
region as well as the direction of the line detection. This has the ad-
vantage that thresholds of image gradients can be set low to find
shallow line appearances.
3.1.4. Fine pose estimation
The fine pose estimation module intends to find an accurate pose of
the camera for a present image of the building interior during con-
struction. Therefore, a rough estimate of the pose from a previous step
is used as an initial guess (see Fig. 2) and makes use of BIM information
to accurately align the camera pose to the building model.
The currently expected state of the building model is analyzed and
geometric primitives are extracted. Furthermore, the image is processed
to determine candidates for the geometric primitives from the building
model. The used geometric primitives are line segments, since line
segments are the dominating geometric primitive in many building
models. This approach takes extracted line segments from the building
model, projects them to the image space and incorporates the method
presented in Subsection 3.1.3 to obtain candidate line segments in the

First Frame 0
Frame Initial
Frame yes
Arrival Registration
?
Pose 0 Pose i-1
no

Frame i Pose
Fine Pose Rough Motion
Found no
Estimation Estimation
?
Pose i
yes
Pose i
Recognition

Fig. 2. Processing of the registration of each frame by pose estimation.

16
C. Kropp et al.
image space. The approach processes this input data and, first, de-
termines the correct correspondent line segments from the model and
the image, and, second, a refined camera pose. For this reason, the fine
pose estimation approach expects a sufficient set of detectable building
model lines in the image.
In a first step, the potential correspondences of line segments, which
are detectable in the image and are present in the building model, are
established. To achieve this, the model line segments are projected onto
the image according to the rough pose estimate (or the pose from the
initial registration for the first frame) and consequently candidates of
line segments visible in the image are determined. In order to obtain
line segments from the building model that are potentially visible from
the camera pose in the captured image, the model data is processed. For
this purpose, it is not sufficient to remove model line segments that are
not visible in the input camera frustum. Rather, model line segments
that are hidden by other structures need to be eliminated, since the
amount of model line segments potentially not visible in the image
should be reduced. Thus, a state-of-the-art hidden-surface algorithm
[93] is incorporated to obtain a set of model lines segments potentially
visible by the camera pose. Concluding, this set states a guess of visible
lines that is dependent on the expected state of the building construc-
tion and the rough pose estimation. Hence, it is possible that model line
segments (a) are modelled and assumed to be visible in the current
construction state, but are actually not visible due to progress devia-
tions or vice versa, and (b) are hidden by non-modelled objects, not
perceptible due to light conditions, etc. Moreover, real image scenes on
construction sites are typically cluttered and contain objects, shadows,
etc. that cannot not be anticipated and thus modelled or modelled in
sufficient detail. For these reasons, the described approach assumes that
in construction scene images not only are more line segments detected,
but also a vast number of outliers are observable. Thus, the approach
presented here is designed to handle a high ratio of outliers and several
line segment candidates from the observed scene for a projected model
line.
Subsequently, image line segments are assigned to model line seg-
ments according to similarity in terms of angle and position distance to
the projected model lines. This assigns each projected model line a set
of image line segment candidates. The resulting problem, finding the
right assignments for the line segment correspondences and simulta-
neously determining the resulting pose is illustrated in Fig. 3.
The problem is solved by incorporating a modified RANSAC algo-
rithm for pose estimation, depicted in Fig. 4. The algorithm starts with
the selection of a random subset of the visible model line segments.
Based on the resulting subset, image line segments are selected from the
set of all candidates that are assigned to a model line to construct a
Fig. 3. Illustration of the absolute pose estimation problem without known corre-
spondences of line segments between the model and the observation. The red line seg-
ment represents a model line and the green line segments candidates for its projection
extracted from the image. (For interpretation of the references to color in this figure
legend, the reader is referred to the web version of this article.)
17
Automation in Construction 86 (2018) 11–32
subset of correspondences. This selection is performed according to the
similarity rank of the candidate to the projection of the model line
segment. Simultaneously, the subset of model lines is analyzed for
known intersections. Incorporating additional point features to the
existing line correspondences improves the accuracy of the results ac-
cording to Xu et al. [70]. Intersections in three-dimensional space re-
sults in an intersection in two-dimensional projection space as well.
Thus, using the image line segments to generate intersections of their
projected lines produces a set of additional point correspondences as an
input for pose determination. Taking the line and point correspondence
subset as the input for a PnX algorithm [70], the model parameters of
the pose can be determined. These model parameters are consequently
tested against all candidate lines. The best fitting candidate lines that
are associated with a model line form the consensus set. Optionally, a
refinement of the model parameters with the consensus set of lines is
performed. If the projection error falls below a certain threshold and
the size of the consensus set is lower than a previously defined size,
appropriate model parameters have been determined, otherwise the
algorithm iterates with a new selected random subset.
3.1.5. Rough motion estimation
Rough motion estimation predicts camera poses for the fine pose
estimation. In the case where fine pose estimation is not successful (see
Fig. 2), these poses remain until it is feasible again to obtain fine poses.
In contrast to the fine pose estimation, which derives poses absolutely
to the building model geometry, rough motion estimation predicts
poses based on relative image-to-image registration. Therefore, the
originally non-transformed and un-scaled camera poses of the image-to-
image registered video frames are transformed and scaled into the
building model coordinate system.
Two distinct frames i and j are considered to estimate the right
transformation and scale. This approach assumes that the absolute pose
Pa = Ra ∣ ca, composed of the absolute rotation Ra and the absolute
camera center ca, from previous fine pose estimation as well as its re-
lative counterpart Pr = Rr ∣ cr, composed of the relative rotation Rr and
the relative camera center cr, from image-to-image registration are
known for both frames. The ratio of the Euclidean distances between
the absolute ca and the relative cr camera centers determines the scale
factor from the relative to the absolute coordinate system
|cai − ca j |
sij = .
|cri − cr j |
For a frame k for which an absolute pose is not available and only Pr
is known, the relative motion from the pose of frame j is determined as
Pr ∆ = Rr ∆ cr ∆ = Pr j−1∙Pr k .
The relative motion is scaled to the absolute motion
Pa∆ = Rr ∆ sij ∙cr ∆.
Knowing the absolute motion from frame j to frame k, the absolute
rough pose is defined as.
Pa rough k = Pa∆ ∙Pa j.
This enables rough pose estimation for all video frames that are not
compliant for fine pose estimation, as well as the provision of predic-
tions for further fine pose estimates.
3.2. Recognition
The recognition block of the general framework contains several
steps in order to prepare the images for activity state determination.
First, the relevant objects and their geometry in the image space are
determined by the search space reduction, afterwards a rectification of
the regarding areas is performed to reduce the recognition problem to
two-dimensional space and finally the state is recognized depending on
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Fig. 4. Extended RANSAC scheme for pose determination with

line candidates.

18
C. Kropp et al.
the type of input activity.
3.2.1. Search space reduction
The images present in a video are captured in an unsupervised way.
Without further comprehension of context knowledge, any object could
possibly appear anytime in every part of every image with any rotation,
scale and translation. Thus, information about the objects that appear in
the images captured cannot be derived easily. The complexity changes
significantly once the image data is registered to the building model.
Reducing the search space within the video sequences relieves the ac-
tual classification from critical ambiguities in image data. Only objects
involved in a certain activity need to be selected. Objects of activities
that were already finished in an earlier point of time, as well as objects
that lie far ahead can certainly be neglected. A specific object is selected
that was identified as relevant for current monitoring by scanning the
schedule for activities. Other objects remain unselected because of
missing currency. Once the objects that need to be observed are known,
it is possible to determine a set of images with objects' visibility. In the
case that objects have to be completely visible for state determination,
the set only contains images with full appearance of the objects. It is
assumed that objects are installed as planned, so that the as-is and as-
planned positions correspond. This approach does not reassess the
correctness of positioning. It only tests for the correct schedule and thus
for presence of objects.
3.2.2. Image rectification
Since the feasibility of comparing objects' appearances greatly
benefits from normalization [94], the present camera pose information
is used to perform rectification on the images as a preparation for
further state determination. This reduces the state recognition to a
simple two-dimensional classification problem. Depending on the ex-
tents of the object of interest defined by the search space reduction, the
original images are transformed into the rectified form (see Fig. 5). The
rectification approximates the original images to a unified view that
ideally is the same for all input images. Since the approximation is
performed on a plane in 3D space, object elements that are not on this
plane are distorted. However, the main part of the rectified images
states a suitable input for classification.
Original Image Rectified Images
19
Automation in Construction 86 (2018) 11–32
3.2.3. State recognition
The state recognition is the actual process where the identification
of the progress is taking place with a classification of the input image
content. Due to the assumption that the image content is rectified to an
approximated plane, the classification can be performed in two-di-
mensional space. Thus, a vast amount of approved mainstream re-
cognition strategies can be applied. However, the approach presented
in this paper regards two different branches of recognition to consider
the circumstance that a wide set of activities needs to be recognized.
3.2.3.1. Object recognition for activity state determination. The
recognition of objects in this approach concerns the physical presence
of the investigated construction element. It assumes that the element's
structure remains mainly the same for each instance of an object due to
a pre-defined classification. However, partially covered areas of the
object or different lighting conditions restrain the classification. This
requires that the whole object is visible. Objects that do not fit into the
image cannot be analyzed using the underlying approach.
3.2.3.2. Material recognition for activity state determination. Material
recognition in this approach addresses the identification of an activity
state. Contrary to object recognition, material does not have one
specific shape, but its texture is distributed over the surface it is
applied on. Sometimes there is a fine line between the distinction of an
object and material, especially when material consists of small
countable objects. Nonetheless, it ultimately depends on the details of
the building model and the construction schedule. Since the underlying
approach expects an activity-specific recognition, this is not defined
automatically. A further characteristic that particularly represent the
challenges in indoor scenes is that the whole extent of the material
surface often cannot be observed by one image (camera-dependent).
Thus, partial visibility of the material needs to be considered in
recognition strategies. However, each material has its own structural
characteristics that can be leveraged for feature extraction, e.g. bricks
are usually aligned in rows and have a rectangular shape. Hence,
material recognition in the proposed framework for state recognition
expects a particular implementation for each material.
Fig. 5. Rectification of two images from different poses to one
reference.
Original Image
C. Kropp et al.
4. Experiments and results
The aim of the experiments was to evaluate the output of the single
steps of the presented framework for validity and the viability of the
whole framework as one unit. The evaluation of the registration steps is
presented first, followed by the test results for the recognition, con-
cluding in the final general experimental output.
4.1. Registration
To evaluate the registration block of the framework, different steps
were tested. This set of steps contains the line extraction, fine pose
estimation, and rough motion estimation. In this section, the experi-
ments were performed on single images.
4.1.1. Line extraction
The line extraction step was tested to evaluate the detection rate of
the proposed approach. Therefore, a test set of 510 images of photos
with a resolution of 4288 × 2848 pixels and video frames (each frame
processed independently) with a resolution of 1920 × 1080 pixels was
created. To enable straight line segment extraction, the images were
undistorted before they were processed.
The results of the proposed algorithm are compared to the line ex-
traction algorithms of Grompone von Gioi et al. [87] and Akinlar et al.
[88]. Those two algorithms are not directly comparable with the pre-
sented method. To obtain comparable results, a distance metric is used
that counts inliers under a certain threshold of the distance to the ex-
pected line as inliers. A positive detection was encountered as soon as at
least line was found/filtered in the search region. The expected lines are
a set of manually defined line segments and state the ground truth. It
was assured that a contour at the defined expected line pose was pre-
sent. The results listed in Table 2 compare the presented method with
two other line segment extraction methods, namely LSD [87] and
EDLines [88]. The time needed to perform the test for [87,88] is ex-
pressed as the extraction time for the whole image and for the presented
method for one line. This varies by the amount of lines that need to be
extracted for an image. However, the algorithm offers opportunities for
parallelization. The detection rate benefits from the low threshold that
can be applied using the presented method. Fig. 6 shows image ex-
amples with sheer lines that could be extracted with the presented
method, but not with the other two compared methods. Unstable
transitions along the length of the edge (see Fig. 6a), edges with low
contrast (see Fig. 6b), and blurry edges (see Fig. 6c) are found more
often with the presented method.
However, the target application for the line segment extraction al-
gorithm is not the detection of lines that are known to be present, but of
model lines that are not certainly present, fully observable, or precise in
pose. Thus, uncertainties induce a larger distance of the expected model
line. The more the line search region is extended, the more candidates
for a line expectation are found.
4.1.2. Fine pose estimation
In this section, the fine pose estimation step is evaluated in terms of
robustness regarding uncertainty in the input camera pose. Therefore,
camera pose deviation is simulated on a set of defined ground truth
camera poses. A set of images is composed and their corresponding
camera poses within the building model are determined by an
Table 2
Line extraction results.
Detection rate Time for each line (avg.)
EDLines [88] 0.18 246.238 ms
LSD [87] 0.41 300.98 ms
Proposed method 0.9259 2.02 ms
20
Automation in Construction 86 (2018) 11–32
exhaustive application of the fine pose estimation method.
The simulated deviations include rotation and translation on the
camera pose. To test the impact of different types of motion, the de-
viations were applied on single axes. The illustrations of Fig. 7 show the
applied maximum deviations applied to each axis on an exemplary
image. The rotational deviation was induced by a maximum of −20°/
+20° and the translation was set to a maximum of − 1 m/+1 m. The
first row of Fig. 7(a-c) shows the model line projections with the
maximum camera pose deviation simulated by the rotation of the x-, y-
and z-axis and the second row (d-f) shows the translation on the x-, y-
and z-axis.
The goal of this experiment was to recover the original deviation-
less camera pose from the divergent input camera pose. To define an
appropriate measure of deviation for the estimated camera pose, the
distance of the endpoints of the model lines (point distance) between
the original pose and the transformed estimated pose was chosen (see
Fig. 8). This unites the rotational and translational error in one value.
The tests were performed on the same set of images as used in the
line extraction experiments (510 images). The search space for the line
extraction was adaptively set according to the point distance between
the original and the input pose of the image space endpoints of the
projected model lines. For the case where not enough model lines were
projected within the image borders, the images were skipped to avoid
infinite error values for erroneous line configuration input to the
RANSAC algorithm.
The charts in Fig. 9 and Fig. 10 show the results of the mean point
distance of all tested images. The deviations were plotted increasing
from the negative to the positive maximum. The lowest point distance
error could be expected to lie at zero deviation. However, since the
RANSAC approach is non-deterministic, this is not guaranteed and re-
sults in the small error range are present.
4.1.3. Rough motion estimation
In this section, the rough motion estimation approach is evaluated.
The main aspect investigated is whether the pose predictions are still
appropriate when a longer passage of frames does not allow for the fine
pose estimation. The main test procedure works as follows: The image
sequence is initialized with fine pose estimation. As soon as fine pose
estimation stops, the rough motion estimation steps in and continues
the estimation of the pose until fine pose estimation is available again.
Therefore, a test set for three different scenarios of camera motion
was generated: translational motion, rotational motion, and both,
translational motion with simultaneous rotation (see Table 3). The
input of the test procedure contains a video file, an initial pose, the
relative camera trajectory, a simple 3D line model, and a frame j that
determines the end of possible fine pose determination. It is assured
that until frame j and for the very last frame of the image sequence, all
model lines are visible and the fine pose estimation is providing suc-
cessful results. To measure the capabilities of the rough motion esti-
mation, the last image is registered to the 3D line model and the dif-
ference between the resulting registered pose and the guessed rough
pose is evaluated.
During the test procedure, fine registration is applied automatically
for a certain number of frames. With the first and last frame of this
subsequence, i and j are determined. The baseline between the camera
center of both frames defines the scale. The trajectory of the subsequent
frames is guessed by the rough motion estimation procedure.
The image sequences were recorded with an off-the-shelf smart-
phone camera capable of recording with a video resolution of
3840 × 2160 pixels and 30 frames per second. The camera was ap-
pointed in a fixed focal length and the intrinsic camera parameters were
determined in advance.
Two different image-to-image registration methods, VO and SFM,
were incorporated in the test procedure. The results of the rough mo-
tion estimation experiments are listed in Table 4. The table contains the
measured length of the trajectory according to the registration, the
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Fig. 6. Image examples with sheer lines that could be extracted with the presented method, but not with the other two compared methods: (a) unstable transitions, (b) low contrast, and
(c) blurry edges.

number of frames the image sequence contains, the first and second
frame used for registration with fine pose estimation, the appointed
technique, and the corresponding error measured in the last frame.
However, the fine pose estimation process could re-register the motion
of the camera to the building model in the last frame in all tested cases
to determine the error.
The results of this experiment show the induction of drift by the
rough motion estimation. However, the deviation measured was very
low such that the fine pose estimation could still re-register absolutely
to the model. This shows that the incorporation of rough motion esti-
mation is possible for a longer sequence. Afterwards, the association
back to the model is feasible. The experiment includes a limited set of
motion. Most motion necessary to assess visual as-built data on the site,
though, is covered by these examples.
4.2. Recognition
The evaluation of the recognition block of the general framework
was performed by experiments of the object and material recognition.
The activity state recognition was conducted on the differentiation of
three stages of a drywall installation and on a two-stage classification of
a radiator mounting. In this section, the experiments were performed
on single images.
4.2.1. Drywall state determination with material recognition
Drywall installation activities are usually finished after completing
three states: a) the installation of panels (Fig. 11a, b) the plastering of
the drywall (Fig. 11b and c) the painting of the drywall (Fig. 11c).
Different visual features that result in a challenging recognition char-
acterize the different states of drywall installation activities. The main
challenges for determining the state of the drywall are the low structure

Fig. 7. Projections of input model lines with divergent camera poses by rotation (a–c) and translation (d–f).

21
C. Kropp et al.
Point distance
Rotation and translation distance
Fig. 8. Illustration of the point distance as the applied error metric.
Fig. 9. Mean point distance of the line endpoints with applied transformation of the es-
timated camera pose with translational deviation input.
Fig. 10. Mean point distance of the line endpoints with applied transformation of the
estimated camera pose with rotational deviation input.
and the homogeneous color distribution over the wall. Furthermore, in
indoor environments, walls may not completely be visible if a special
lens is not applied.
According to the description of the introduced state recognition
framework, the method is explained as a workflow to achieve state
differentiation. The classification workflow is illustrated as a flow chart
in Fig. 12. In the first step, the image is loaded and cut according to the
provided ROI (see example in Fig. 13). The second step includes seg-
mentation that prevents the inclusion of unexpected objects, like lad-
ders, painting pots or cables that may occlude the desired drywall and
22
Automation in Construction 86 (2018) 11–32
thus disturb a clear view on the expected texture characteristics and
influence results. After these preparatory steps, the sheer drywall can be
investigated. Therefore, the subsequent step performs a compass edge
feature extraction and comparison. The compass edge features are ex-
tracted with the information from the registered motion information
that provides the camera view and rotation relative to the wall. With
that information, an edge filter can be adapted to find the horizontal
and vertical gaps as well as the round screw spots at the drywall. Also
within this step, a preliminarily trained support vector machine classifier
with the ability to decide between paneled drywalls and the other two
stages pursuant to the extracted edge features is applied.
The paneled state is the earliest outcome state in this classification
workflow, which exits if this state is reached. For the differentiation
between the remaining states, further steps within the workflow are
necessary. The further state distinction is focused on increasing the
contrast of the areas that are painted to the plastered panels. Histogram
based adaptive contrast normalization is used to expose the contrast at
the plastered areas of the wall. Afterwards, a histogram considering the
pixel intensity is used as a feature vector for classification. The result
holds a precision and recall of at least 95%, which allows for the usage
of the method as a process in the general framework. Exemplary results
are illustrated in Fig. 14. A deeper insight into the method is given in
[83].
4.2.2. Radiator installation state determination with object recognition
The second investigated case for state recognition is based on the
method presented in Kropp et al. [84]. The approach is intended to
derive the current state of an activity by recognizing the presence of
structural objects with changing appearance from different views by
making an approximation of its real perspective transform. It assumes
that views onto an object from a certain similar spatial region result in
comparable object appearances, such that it is sufficient to train a
classifier for achieving good recognition results. It considers available
information about the object from BIM, as well as motion information
to reduce the complex three-dimensional recognition problem into a
simple two-dimensional detection problem. The method was evaluated
on the example of the recognition of radiators. Fig. 15 shows different
activity states of radiators that need to be recognized. Different possible
states of the radiator are reduced to two states and concluded into the
radiator being present and non-present. Examples for the non-present
state can be an unprepared (Fig. 15a) or prepared installation back-
ground (Fig. 15b), whereas the present radiator can be found wrapped
in packaging foil (Fig. 15c) or unwrapped (Fig. 15d).
The workflow that is related to the current approach is displayed in
the flow chart in Fig. 16. In the first step within this flow chart, the view
onto the object is classified by an aspect approximation. This step is
slightly modified compared to the original design to refer to the concept
of aspect graphs [95]. Unlike the drywall recognition process, not the
standard rectified image ROI is used, but an aspect-normalized ROI of
the image. As the result, the most dominant approximated surface is
rectified. Hereby, information from the BIM object geometry and the
registered motion information is used to obtain the relative pose of the
object to the camera. The approximated rectification is intended to
reduce the three-dimensional appearance into a normalized two-di-
mensional representation of the object. Before visual features are ex-
tracted, this rectangle is illuminated to remove illumination inequal-
ities. Finally, the method of Histogram of Oriented Gradients is applied to
extract information about visual characteristics and used to train a
support vector machine classifier. This enables the derivation of the
presence of the object and hence, a state of the activity. The experi-
ments result in a precision of 0.96 and a recall of 0.99, which satisfies
the usage of the method as a process in the general framework. In
Fig. 17 exemplified results of the described method are illustrated.
Kropp et al. [84] contains a detailed description of the explained steps.
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Table 3
Test cases for rough motion estimation experiments.

Test case Overview Example images of the sequence

Translation

Rotation

Translation & rotation

Table 4
Results of the rough motion estimation experiments.

Test scenario Measured trajectory length # Frames 1st fine pose 2nd fine pose Baseline between 1st and 2nd fine pose Technique Last frame error
[m] frame frame frame[m] [m]

Translation 18.098 721 1 30 0.281 VO 0.267

SFM 0.043
Rotation 3.327 713 1 12 0.024 VO 0.195
SFM 0.640
Trans. & Rot. 16.617 541 1 8 0.181 VO 0.405
SFM 0.288

Fig. 11. Example images of (a) installed drywall panels, (b) plastered drywall, and (c) painted drywall.

4.3. Case study on construction site videos
For the evaluation of the whole framework, three exemplary videos
were selected that document the radiator and drywall activity states. In
Table 5, the content of image sequences is listed. All the videos were
recorded at the renovation of the IC building on the campus of the
Ruhr-Universität Bochum with the monocular rear camera of a tablet
PC. The first video shows the non-present radiator and installed drywall
panels, the second video shows the non-installed radiator and the
plastered drywall state, and the third video contains the appearance of
the radiator and the painted drywall state. The goal of this experiment
was to test the ability of the final statement about the activity state
under the influence of all previous steps starting from the initial re-
gistration, over fine and rough pose estimation towards the activity
state recognition.
In the beginning of the sequences, a sufficient set of visible model

23
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Drywall State Recognition

4D BIM

Building Object
Schedule Catalogue
model

Geometry Relevant Objects

Compass Edge Intensity

Rectified
Segmentation Feature Panelled? Histogram Plastered?
ROI false Extraction false
Extraction
true true
Segmented Compass Edge Intensity Identified
Rectified ROI Distribution Histogram Activity State

Fig. 12. Workflow for the classification of the drywall installation completion state.

Fig. 13. Example image of a drywall with original appearance (a), BIM extracted ROI (b) and applied segmentation (c).

Fig. 14. Exemplified classification results: (a) recognized installed drywall state, (b–c) recognized plastered drywalls state, (d) recognized painted drywall state.

Fig. 15. Different activity states for the test configuration: (a) unprepared background, (b) prepared background, (c) radiator wrapped in packaging foil, and (d) unwrapped radiator.

lines appears, such that a proper fine pose estimation was guaranteed.
Since earlier test results showed a positive and reliable application of
the rough motion estimation, only a few manually set frames were
defined for fine pose registration. The image sequences contain some
obstacles that partially cover the relevant activities. The first sequence
contains Euro pallets on the floor that cover the second drywall par-
tially. In the second sequence, some beams cover all activities due to the
camera motion. The third sequence contains a ladder in front of the
present radiator in most frames. The videos contain critical sequences
like backlight, rotation-dominant motion of the camera, views with

24
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Fig. 16. Workflow for the classification of the heating installation completion state with example image data.

Fig. 17. Exemplified classification results: (a-d) true negatives (TN), (e) false negatives (FN), (f–h) true positive (TP).

Table 5 containing the relevant activity subject, as well as the number of frames
Overview of the activity states within the three exemplary image sequences. for which a correct determination of the current state was derived. The
voted result shows the ultimate result for the inspection of the activity
Sequence Location Date Radiator Drywall 1 Drywall 2
calculating the absolute majority of state determinations. For all se-
#1 room 5–169 November 27, Not installed Installed Installed quences, activities and frames, the activities were classified correctly.
2012 Thus, also the voted results are correct. Although the camera pose de-
#2 room 5–169 December 20, Not installed Plastered Plastered termination was not always stable due to low feature areas and thus the
2012
#3 room 2–169 December 20, Installed Painted Painted
ROIs of the analyzed state were not always precise, they were accurate
2012 enough for a robust statement about the activity state.
Based on the image sequences, the results were composed in videos
that illustrate the BIM registration and activity state recognition pro-
low-textured walls and an insufficient set of visible model lines. cess. For the registration section, the observed activities were high-
However, for recognition terms, the states of the desired objects and lighted in the original frames. An equivalent view was created with a
materials were expected to be properly recognized. BIM viewer and additionally, a bird's eye view that shows the motion of
Table 6 shows the results of the test. The table contains the total the camera within the building model was created. For the recognition
number of frames contained in the image sequence of the videos, the section, the rectified areas of the observed activities are visualized. It
number of frames that were automatically selected for potentially was highlighted when the recognition of the state was successfully

25
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Table 6
Overview of the results within the three exemplary image sequences.

Seq. Sequence length Radiator Drywall 1 Drywall 2

[frames]
Visible Correctly classified Voted Visible Correctly classified Voted result Visible Correctly classified Voted result
[frames] [frames] result [frames] [frames] [frames] [frames]

#1 808 274 274 Not 326 326 Installed 15 15 Installed

present
#2 1732 429 429 Not 345 345 Plastered 346 346 Plastered
present
#3 828 236 236 Present 223 223 Painted 70 70 Painted

Fig. 18. First example result frame of the first

image sequence of the case study (small white
text is summarized in Table 7).

Fig. 19. Second example result frame of the first

image sequence of the case study study (small
white text is summarized in Table 7).

performed and statistics about the recognition are printed. Eleven
frames are exemplarily shown (Figs. 18–28). The first three examples
show the composed results from the first sequence (Figs. 18–20), the
succeeding four examples show the composed results from the second
sequence (Figs. 21–24), and the last four examples show the composed
results from the third sequence (Figs. 25–28).
The information printed in the example images is summarized in
Table 7. The table contains the related figure and the associated se-
quence the frame was taken from. Moreover, the visibility and the re-
cognition statistics of the three analyzed activities are contained in the
table. The visibility means the visibility of the activity-related con-
struction objects in the estimated image region of the current frame. In
case of the drywalls, the visibility is represented in a percentage value,
whereas the visibility of the radiator is only given when the whole
object is visible (yes/no). The recognition statistics hold the voting
results for the regarded activity states that were recognized for the
qualified image up to the current frame.
5. Discussion
The results of the experiments in the preceding section show the
overall viability of the framework. However, analyzing the results of
the experiments, several issues, which the proposed method has yet to
address, were encountered. This mainly includes the as-built acquisi-
tion and the registration block of the framework. During the processing
of the image sequences, it could be observed that very rough motion of
the camera, especially rotation-dominant motion in front of the close-
range objects like walls in combination with the monocular camera
system can lead to bad results. Scale varies during the trajectory re-
construction of the video sequence, especially the scale cannot be well

26
C. Kropp et al.
reconstructed during rotational motion. For practical reasons, per-
sonnel of future inspections on the construction site should be made
aware of this.
Furthermore, the approach for scale determination accomplished in
rough motion estimation assumes that the scale is fixed until the next
fine pose estimation process. The relative vision-based motion estima-
tion techniques regarded, particularly VO, induce an unpredictable
degree of drift for each succeeding frame that can grow rapidly de-
pending on many factors. Thus, it is recommendable to return to fine
27
Automation in Construction 86 (2018) 11–32
Fig. 20. Third example result frame of the first
image sequence of the case study (small white
text is summarized in Table 7).
Fig. 21. First example result frame of the second
image sequence of the case study study (small
white text is summarized in Table 7).
Fig. 22. Second example result frame of the
second image sequence of the case study (small
white text is summarized in Table 7).
pose estimation as early as possible.
Since the accumulated search space for line segment extraction
varies according to the induced uncertainty and the amount of pro-
jected model lines that are imposed onto the image differs from image
to image, the presented approach might not be the fastest option in all
cases. Sometimes, when a vast amount of lines is to be extracted, it can
be advantageous in terms of processing time to incorporate a con-
servative line segment extraction approach with subsequent filtering.
C. Kropp et al.
6. Conclusions and outlook
In order to determine the actual state of buildings under construc-
tion, this paper contributes a promising novel method that increases the
degree of automation for inspections. The method is based on the 4D
BIM model information and image sequences that are merged and
processed to derive rich information about single construction tasks of
interest. Each image of a sequence is registered to the 4D BIM model in
terms of its origin in the building model coordinate system and its point
28
Automation in Construction 86 (2018) 11–32
Fig. 23. Third example result frame of the second
image sequence of the case study (small white
text is summarized in Table 7).
Fig. 24. Fourth example result frame of the
second image sequence of the case study (small
white text is summarized in Table 7).
Fig. 25. First example result frame of the third
image sequence of the case study (small white
text is summarized in Table 7).
in time in the construction schedule. Solely once, manual intervention
is needed for initialization to register a sequence. Consecutive images
are registered in a fully automatic manner.
With the BIM registered image sequences, the search for relevant
tasks is reduced to regions of interest that contain tasks of interests
according to the schedule and associated objects in the building model.
Again, registration information comes into play for the rectification of
the determined image regions. This leads to less complex problems for
the activity state recognition.
C. Kropp et al.
The behavior of single independent steps of the new method was
studied as well as the performance of the general framework for each
step using the results of preceding steps. Both, the independent as well
as the dependent results show promising applicability for widely au-
tomatic progress monitoring combining BIM and image sequences.
However, potential errors in each of the subsequent steps are propa-
gated to the following step. So far, this has not been analyzed and tested
in detail yet, but leaves room for further improvements and future
discussions.
29
Automation in Construction 86 (2018) 11–32
Fig. 26. Second example result frame of the third
image sequence of the case study (small white
text is summarized in Table 7).
Fig. 27. Third example result frame of the third
image sequence of the case study (small white
text is summarized in Table 7).
Fig. 28. Fourth example result frame of the third
image sequence of the case study (small white
text is summarized in Table 7).
The results show that an automation of progress monitoring sup-
ports the availability of quickly interpreted as-built inspections. This
enables short intervals and, thus, frequent updates on the as-built
states. Performing inspections from the beginning of finishing works on
a daily basis reveals frequent insights about activities performed on the
construction site. However, in the case where activities are actually
performed ahead of the schedule, occlusions might be possible and
would cause problems. A potential solution in future work would be to
use certain time windows of the planned schedule including activities
C. Kropp et al. Automation in Construction 86 (2018) 11–32

Table 7
Details of the printed information within the examples of the composed result frames.

Figure Example sequence Drywall #1 Radiator Drywall #2

Visibility Installed/Plastered/Painted Visibility Not installed/Installed Visibility Installed/Plastered/Painted

Fig. 18 1 50% 1/0/0 Yes 1/0 28% 0/0/0

Fig. 19 1 55% 216/0/0 No 133/0 0% 15/0/0
Fig. 20 1 1% 326/0/0 Yes 222/0 0% 15/0/0
Fig. 21 2 54% 0/12/0 Yes 12/0 31% 0/11/0
Fig. 22 2 59% 0/134/0 No 73/0 0% 0/21/0
Fig. 23 2 0% 0/227/0 Yes 196/0 0% 0/21/0
Fig. 24 2 21% 0/227/0 No 429/0 54% 0/325/0
Fig. 25 3 47% 0/0/4 Yes 0/4 24% 0/0/0
Fig. 26 3 54% 0/0/140 No 0/99 0% 0/0/0
Fig. 27 3 4% 0/0/221 Yes 0/143 6% 0/0/0
Fig. 28 3 0% 0/0/221 No 0/234 41% 0/0/48

and object ahead of the schedule.
Due to the complexity of the framework, there is a vast availability
of starting points for optimization. This can reach from more robust
video recording techniques that will come up soon to real time im-
plementation of the whole framework to achieve augmented reality
applications on mobile devices. However, the framework is generic and
modular, which allows for the smooth interchange of single steps in
many ways. Nevertheless, there is a need to generate an extensive test
set of activities to cover for state recognition.
Acknowledgements
The authors gratefully acknowledge the financial support by the
German Research Foundation (DFG) for this work under the grants KO
4311/4-1 and KO 3473/8-1.
References
[1] S. Roh, Z. Aziz, F. Peña-Mora, An object-based 3D walk-through model for interior
construction progress monitoring, Autom. Constr. 20 (2011) 66–75, http://dx.doi.
org/10.1016/j.autcon.2010.07.003.
[2] P. Greiner, P. Mayer, K. Stark, Ablaufplanung, Baubetriebslehre - Proj, 4th ed.,
Vieweg + Teubner Verlag, 9783834806581, 2009, pp. 115–170.
[3] S.N. Razavi, C.T. Haas, Multisensor data fusion for on-site materials tracking in
construction, Autom. Constr. 19 (2010) 1037–1046, http://dx.doi.org/10.1016/j.
autcon.2010.07.017.
[4] S.N. Razavi, C.T. Haas, Using reference RFID tags for calibrating the estimated lo-
cations of construction materials, Automation in Construction, 2011, pp. 677–685, ,
http://dx.doi.org/10.1016/j.autcon.2010.12.009.
[5] T. Cheng, M. Venugopal, J. Teizer, P.A. Vela, Performance evaluation of ultra
wideband technology for construction resource location tracking in harsh en-
vironments, Autom. Constr. 20 (2011) 1173–1184, http://dx.doi.org/10.1016/j.
autcon.2011.05.001.
[6] T. Cheng, J. Teizer, G.C. Migliaccio, U.C. Gatti, Automated task-level activity
analysis through fusion of real time location sensors and worker's thoracic posture
data, Autom. Constr. 29 (2013) 24–39, http://dx.doi.org/10.1016/j.autcon.2012.
08.003.
[7] A. Shahi, J.S. West, C.T. Haas, Onsite 3D marking for construction activity tracking,
Autom. Constr. 30 (2013) 136–143, http://dx.doi.org/10.1016/j.autcon.2012.11.
027.
[8] R. Maalek, F. Sadeghpour, Accuracy assessment of ultra-wide band technology in
tracking static resources in indoor construction scenarios, Autom. Constr. 30 (2013)
170–183, http://dx.doi.org/10.1016/j.autcon.2012.10.005.
[9] S. Woo, S. Jeong, E. Mok, L. Xia, C. Choi, M. Pyeon, J. Heo, Application of WiFi-
based indoor positioning system for labor tracking at construction sites: a case study
in Guangzhou MTR, Automation in Construction, 2011, pp. 3–13, , http://dx.doi.
org/10.1016/j.autcon.2010.07.009.
[10] X. Shen, W. Chen, M. Lu, Wireless sensor networks for resources tracking at building
construction sites, Tsinghua Sci. Technol. 13 (2008) 78–83, http://dx.doi.org/10.
1016/S1007-0214(08)70130-5.
[11] S. El-Omari, O. Moselhi, Integrating 3D laser scanning and photogrammetry for
progress measurement of construction work, Autom. Constr. 18 (2008) 1–9, http://
dx.doi.org/10.1016/j.autcon.2008.05.006.
[12] F. Bosché, M. Ahmed, Y. Turkan, C.T. Haas, R. Haas, The value of integrating scan-
to-BIM and scan-vs-BIM techniques for construction monitoring using laser scan-
ning and BIM: the case of cylindrical MEP components, Autom. Constr. (2014),
http://dx.doi.org/10.1016/j.autcon.2014.05.014.
[13] N. Pradhananga, J. Teizer, Automatic spatio-temporal analysis of construction site
equipment operations using GPS data, Autom. Constr. 29 (2013) 107–122, http://
dx.doi.org/10.1016/j.autcon.2012.09.004.
[14] K. Han, M. Golparvar-Fard, D. Castro-Lacouture, J. Irizarry, B. Ashuri (Eds.),
Automated monitoring of operation-level construction progress using 4D BIM and
daily site photologs, Construction Research Congress 2014, American Society of
Civil Engineers, Atlanta, Georgia, 2014, pp. 1033–1042, , http://dx.doi.org/10.
1061/9780784413517.106.
[15] Z. Zhu, S. German, I. Brilakis, Detection of large-scale concrete columns for auto-
mated bridge inspection, Autom. Constr. 19 (2010) 1047–1055, http://dx.doi.org/
10.1016/j.autcon.2010.07.016.
[16] Y.M. Ibrahim, T.C. Lukins, X. Zhang, E. Trucco, A.P. Kaka, Towards automated
progress assessment of workpackage components in construction projects using
computer vision, Adv. Eng. Inform. 23 (2009) 93–103, http://dx.doi.org/10.1016/
j.aei.2008.07.002.
[17] H. Son, C. Kim, 3D structural component recognition and modeling method using
color and 3D data for construction progress monitoring, Autom. Constr. 19 (2010)
844–854, http://dx.doi.org/10.1016/j.autcon.2010.03.003.
[18] I. Brilakis, M.W. Park, G. Jog, Automated vision tracking of project related entities,
Adv. Eng. Inform. 25 (2011) 713–724, http://dx.doi.org/10.1016/j.aei.2011.01.
003.
[19] S. El-Omari, O. Moselhi, Data acquisition from construction sites for tracking pur-
poses, Eng. Constr. Archit. Manag. 16 (2009) 490–503, http://dx.doi.org/10.1108/
09699980910988384.
[20] M.W. Park, I. Brilakis, Construction worker detection in video frames for initializing
vision trackers, Autom. Constr. 28 (2012) 15–25, http://dx.doi.org/10.1016/j.
autcon.2012.06.001.
[21] E. Rezazadeh Azar, B. McCabe, Automated visual recognition of dump trucks in
construction videos, J. Comput. Civ. Eng. 26 (2012) 769–781, http://dx.doi.org/10.
1061/(ASCE)CP.1943-5487.0000179.
[22] M. Golparvar-Fard, A. Heydarian, J.C. Niebles, Vision-based action recognition of
earthmoving equipment using spatio–temporal features and support vector machine
classifiers, Advanced Enginering Informatics. 27 (2013) 652–663, http://dx.doi.
org/10.1016/j.aei.2013.09.001.
[23] J. Gong, C.H. Caldas, C. Gordon, Learning and classifying actions of construction
workers and equipment using bag-of-video-feature-words and Bayesian network
models, Adv. Eng. Inform. 25 (2011) 771–782, http://dx.doi.org/10.1016/j.aei.
2011.06.002.
[24] A. Khosrowpoura, J.C. Niebles, M. Golparvar-Fard, Vision-based workface assess-
ment using depth images for activity analysis of interior construction operations,
Autom. Constr. 48 (2014) 74–87, http://dx.doi.org/10.1016/j.autcon.2014.08.003.
[25] A.R. Andoh, X. Su, H. Cai, A boundary condition-based algorithm for locating
construction site objects using RFID and GPS, Construction Research Congress 2012
(28) (2012) 808–817, http://dx.doi.org/10.1061/9780784412329.082.
[26] M. Kopsida, I. Brilakis, P. Vela, A review of automated construction progress and
inspection methods, Proc. 32nd CIB W78 Conference on Construction IT, 2015, pp.
421–431.
[27] J. Teizer, Status quo and open challenges in vision-based sensing and tracking of
temporary resources on infrastructure construction sites, Adv. Eng. Inform. 29
(2015) 225–238, http://dx.doi.org/10.1016/j.aei.2015.03.006.
[28] J. Yang, M.W. Park, P.A. Vela, M. Golparvar-Fard, Construction performance
monitoring via still images, time-lapse photos, and video streams: now, tomorrow,
and the future, Adv. Eng. Inform. 29 (2015) 211–224, http://dx.doi.org/10.1016/j.
aei.2015.01.011.
[29] E. Ergen, B. Akinci, R. Sacks, Tracking and locating components in a precast storage
yard utilizing radio frequency identification technology and GPS, Autom. Constr. 16
(2007) 354–367, http://dx.doi.org/10.1016/j.autcon.2006.07.004.
[30] D. Grau, L. Zeng, Y. Xiao, Automatically tracking engineered components through
shipping and receiving processes with passive identification technologies, Autom.
Constr. 28 (2012) 36–44, http://dx.doi.org/10.1016/j.autcon.2012.05.016.
[31] J. Song, C.T. Haas, C.H. Caldas, Tracking the location of materials on construction
job sites, J. Constr. Eng. Manag. 132 (2006) 911–918, http://dx.doi.org/10.1061/
(ASCE)0733-9364(2006)132:9(911).
[32] D. Grau, C.H. Caldas, C.T. Haas, P.M. Goodrum, J. Gong, Assessing the impact of
materials tracking technologies on construction craft productivity, Autom. Constr.

30
C. Kropp et al.
18 (2009) 903–911, http://dx.doi.org/10.1016/j.autcon.2009.04.001.
[33] J. Teizer, D. Lao, M. Sofer, Rapid automated monitoring of construction site ac-
tivities using ultra-wideband, 24th Int. Symp. on Automation and Robotics in
Construction, ISARC 2007, 2007 9788190423519, pp. 23–28.
[34] A.A. Oloufa, M. Ikeda, H. Oda, Situational awareness of construction equipment
using GPS, wireless and web technologies, Automation in Construction, 2003, pp.
737–748, , http://dx.doi.org/10.1016/S0926-5805(03)00057-8.
[35] S. El-Omari, O. Moselhi, Integrating automated data acquisition technologies for
progress reporting of construction projects, Automation in Construction, 2011, pp.
699–705, , http://dx.doi.org/10.1016/j.autcon.2010.12.001.
[36] G. Deak, K. Curran, J. Condell, A survey of active and passive indoor localisation
systems, Comput. Commun. 35 (2012) 1939–1954, http://dx.doi.org/10.1016/j.
comcom.2012.06.004.
[37] M. Golparvar-Fard, J. Bohn, J. Teizer, S. Savarese, F. Peña-Mora, Evaluation of
image-based modeling and laser scanning accuracy for emerging automated per-
formance monitoring techniques, Autom. Constr. 20 (2011) 1143–1155, http://dx.
doi.org/10.1016/j.autcon.2011.04.016.
[38] P. Tang, D. Huber, B. Akinci, R. Lipman, A. Lytle, Automatic reconstruction of as-
built building information models from laser-scanned point clouds: a review of
related techniques, Autom. Constr. 19 (2010) 829–843, http://dx.doi.org/10.1016/
j.autcon.2010.06.007.
[39] Y. Turkan, F. Bosche, C.T. Haas, R. Haas, Automated progress tracking using 4D
schedule and 3D sensing technologies, Automation in Construction, 2012, pp.
414–421, , http://dx.doi.org/10.1016/j.autcon.2011.10.003.
[40] F. Bosche, C.T. Haas, B. Akinci, Automated recognition of 3D CAD objects in site
laser scans for project 3D status visualization and performance control, J. Comput.
Civ. Eng. 23 (2009) 311–318, http://dx.doi.org/10.1061/(ASCE)0887-3801(2009)
23:6(311).
[41] F. Bosché, Automated recognition of 3D CAD model objects in laser scans and
calculation of as-built dimensions for dimensional compliance control in con-
struction, Adv. Eng. Inform. 24 (2010) 107–118, http://dx.doi.org/10.1016/j.aei.
2009.08.006.
[42] T.C. Lukins, E. Trucco, Towards automated visual assessment of progress in con-
struction projects, Proc. of the British Machine Vision Conference, 2007, pp. 1–10, ,
http://dx.doi.org/10.5244/C.21.18.
[43] X. Zhang, N. Bakis, T.C. Lukins, Y.M. Ibrahim, S. Wu, M. Kagioglou, G. Aouad,
A.P. Kaka, E. Trucco, Automating progress measurement of construction projects,
Autom. Constr. 18 (2009) 294–301, http://dx.doi.org/10.1016/j.autcon.2008.09.
004.
[44] D. Rebolj, N.Č. Babič, A. Magdič, P. Podbreznik, M. Pšunder, Automated con-
struction activity monitoring system, Adv. Eng. Inform. 22 (2008) 493–503, http://
dx.doi.org/10.1016/j.aei.2008.06.002.
[45] M. Golparvar-Fard, F. Peña-Mora, Application of visualization techniques for con-
struction progress monitoring, Computing in Civil Engineering, 2007, pp. 216–223,
, http://dx.doi.org/10.1061/40937(261)27.
[46] M. Golparvar-Fard, F. Peña-Mora, S. Savarese, Application of D4AR - A 4-dimen-
sional augmented reality model for automating construction progress monitoring
data collection, processing and communication, Journal of Information Technology
in Construction 14 (2009) 129–153 (ISSN:14036835).
[47] M. Golparvar-Fard, F. Peña-Mora, C.A. Arboleda, S. Lee, Visualization of con-
struction progress monitoring with 4D simulation model overlaid on time-lapsed
photographs, J. Comput. Civ. Eng. 23 (2009) 391–404, http://dx.doi.org/10.1061/
(ASCE)0887-3801(2009)23:6(391).
[48] M. Golparvar-Fard, S. Savarese, F. Peña-Mora, Automated model-based recognition
of progress using daily construction photographs and IFC-based 4D models,
Construction Research Congress 2010, 2010, pp. 51–60, , http://dx.doi.org/10.
1061/41109(373)6.
[49] F. Dai, A. Rashidi, I. Brilakis, P. Vela, Comparison of image-based and time-of-flight-
based technologies for three-dimensional reconstruction of infrastructure, Journal
of Construction Engineering and Management 139 (2013) 69–79, http://dx.doi.
org/10.1061/(Asce)Co.1943-7862.0000565.
[50] D. Nister, O. Naroditsky, J. Bergen, Visual odometry, Proc. 2004 IEEE computer
society conference on computer vision and pattern recognition, 2004, pp. 652–659,
, http://dx.doi.org/10.1109/CVPR.2004.1315094.
[51] D. Scaramuzza, F. Fraundorfer, Visual odometry part II, IEEE Robot. Autom. Mag.
18 (2011) 80–92, http://dx.doi.org/10.1109/MRA.2011.943233.
[52] J. Engel, J. Sturm, D. Cremers, Semi-dense visual odometry for a monocular camera,
Proc. IEEE International Conference on Computer Vision, 2013, pp. 1449–1456, ,
http://dx.doi.org/10.1109/ICCV.2013.183.
[53] S. Niko, K. Konolige, S. Lacroix, P. Protzel, Visual Odometry Using Sparse Bundle
Adjustment on an Autonomous Outdoor Vehicle, Elektrotechnik Und
Informationstechnik, (2005), pp. 157–163, http://dx.doi.org/10.1007/3-540-
30292-1_20.
[54] H. Strasdat, J.M.M. Montiel, A.J. Davison, Real-time monocular SLAM: why filter?
Proc. - IEEE International Conference on Robotics and Automation, 2010, pp.
2657–2664, , http://dx.doi.org/10.1109/ROBOT.2010.5509636.
[55] H. Zhou, D. Zou, L. Pei, R. Ying, P. Liu, W. Yu, StructSLAM: visual SLAM with
building structure lines, IEEE Trans. Veh. Technol. 64 (2015) 1364–1375, http://
dx.doi.org/10.1109/TVT.2015.2388780.
[56] P. Podbreznik, D. Rebolj, Automatic comparison of site images and the 4D model of
the building, in: R. Scherer, P. Katranuschkov, S.E. Schapke (Eds.), Conference on
Information Technology in Construction, Dresden, Germany, 2005, pp. 235–239
(3860054783).
[57] T.C. Lukins, E. Trucco, Towards automated visual assessment of progress in con-
struction projects, Proceedings of the British Machine Vision Conference, 2007, pp.
142–151, , http://dx.doi.org/10.5244/C.21.18.
31
Automation in Construction 86 (2018) 11–32
[58] X. Zhang, N. Bakis, T.C. Lukins, Y.M. Ibrahim, S. Wu, M. Kagioglou, G. Aouad,
A.P. Kaka, E. Trucco, Automating progress measurement of construction projects,
Autom. Constr. 18 (2009) 294–301, http://dx.doi.org/10.1016/j.autcon.2008.09.
004.
[59] P. David, D. Dementhon, Object recognition in high clutter images using line fea-
tures, Proc. IEEE International Conference on Computer Vision, 2005, pp.
1581–1588, , http://dx.doi.org/10.1109/ICCV.2005.173.
[60] C. Kim, C. Kim, H. Son, Fully automated registration of 3D data to a 3D CAD model
for project progress monitoring, Autom. Constr. 35 (2013) 587–594, http://dx.doi.
org/10.1016/j.autcon.2013.01.005.
[61] B. Bellekens, V. Spruyt, M. Weyn, A survey of rigid 3D pointcloud registration al-
gorithms, Ambient 2014, Fourth International Conference on Ambient Computing,
Applications, Services and Technologies 2014, 2014 9781612083568, pp. 8–13.
[62] M.Y. Yang, Y. Cao, J. McDonald, Fusion of camera images and laser scans for wide
baseline 3D scene alignment in urban environments, ISPRS J. Photogramm. Remote
Sens. 66 (2011), http://dx.doi.org/10.1016/j.isprsjprs.2011.09.004.
[63] V. Pătrăucean, I. Armeni, M. Nahangi, J. Yeung, I. Brilakis, C. Haas, State of re-
search in automatic as-built modelling, Adv. Eng. Inform. 29 (2015) 162–171,
http://dx.doi.org/10.1016/j.aei.2015.01.001.
[64] B. Přibyl, P. Zemčík, M. Čadík, Camera pose estimation from lines using Plücker
coordinates, Proc. Br. Mach. Vis. Conf. 2015, The British Machine Vision
Association and Society for Pattern Recognition, Swansea, GB, 2015, pp.
45.1–45.12, , http://dx.doi.org/10.5244/C.29.45.
[65] H.H. Chen, Pose determination from line-to-plane correspondences: existence
condition and closed-form solutions, IEEE Trans. Pattern Anal. Mach. Intell. 13
(1991) 530–541, http://dx.doi.org/10.1109/34.87340.
[66] Y. Liu, T.S. Huang, O.D. Faugeras, Determination of camera location from 2-D to 3-
D line and point correspondences, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990)
28–37, http://dx.doi.org/10.1109/34.41381.
[67] A. Ansar, K. Daniilidis, Linear pose estimation from points or lines, IEEE Trans.
Pattern Anal. Mach. Intell. 25 (2003) 578–589, http://dx.doi.org/10.1109/TPAMI.
2003.1195992.
[68] F.M. Mirzaei, S.I. Roumeliotis, Globally optimal pose estimation from line corre-
spondences, Proc. - IEEE International Conference on Robotics and Automation,
2011, pp. 5581–5588, , http://dx.doi.org/10.1109/ICRA.2011.5980272.
[69] L. Zhang, C. Xu, K.M. Lee, R. Koch, Robust and efficient pose estimation from line
correspondences, Lect. Notes Comput. Sci. (Including subser. Lecture Notes on
Artificial Intelligence, Lecture Notes Bioinformatics), 2013, pp. 217–230, , http://
dx.doi.org/10.1007/978-3-642-37431-9_17.
[70] C. Xu, L. Zhang, L. Cheng, R. Koch, Pose estimation from line correspondences: a
complete analysis and a series of solutions, IEEE Trans. Pattern Anal. Mach. Intell.
(2016), http://dx.doi.org/10.1109/TPAMI.2016.2582162.
[71] M.a. Fischler, R.C. Bolles, Random sample consensus: a paradigm for model fitting
with applications to image analysis and automated cartography, Commun. ACM 24
(1981) 381–395, http://dx.doi.org/10.1145/358669.358692.
[72] P. David, D. Dementhon, R. Duraiswami, H. Samet, SoftPOSIT: simultaneous pose
and correspondence determination, Int. J. Comput. Vis. 59 (2004) 259–284, http://
dx.doi.org/10.1023/B:VISI.0000025800.10423.1f.
[73] S. Gold, a. Rangarajan, C.P. Lu, S. Pappu, New algorithms for 2D and 3D point
matching:: pose estimation and correspondence, Pattern Recogn. 31 (1998)
1019–1031, http://dx.doi.org/10.1016/S0031-3203(98)80010-1.
[74] D.F. Dementhon, L.S. Davis, Model-based object pose in 25 lines of code, Int. J.
Comput. Vis. 15 (1995) 123–141, http://dx.doi.org/10.1007/BF01450852.
[75] D. Oberkampf, D.F. DeMenthon, Iterative pose estimation using coplanar feature
points, Comput. Vis. Image Underst. 63 (1996) 495–511, http://dx.doi.org/10.
1006/cviu.1996.0037.
[76] P. David, P. David, D. DeMenthon, D. DeMenthon, R. Duraiswami, R. Duraiswami,
H. Samet, H. Samet, Simultaneous pose and correspondence determination using
line features, IEEE Computer Society Conference on Computer Vision and Pattern
Recognition 2, 2003, pp. 424–431, , http://dx.doi.org/10.1109/CVPR.2003.
1211499.
[77] J.C. Diaz, M. Abderrahim, Modified SoftPOSIT algorithm for 3D visual tracking,
2007 IEEE international symposium on intelligent, Signal Process. (2007), http://
dx.doi.org/10.1109/WISP.2007.4447523.
[78] F. Moreno-Noguer, V. Lepetit, P. Fua, Pose priors for simultaneously solving
alignment and correspondence, Lecture Notes in Computer Science (Lecture Notes
on Artificial Intelligence, Lecture Notes Bioinformatics), 2008, pp. 405–418, ,
http://dx.doi.org/10.1007/978-3-540-88688-4-30.
[79] M. Brown, D. Windridge, J.Y. Guillemaut, Globally optimal 2D-3D registration from
points or lines without correspondences, Proc. IEEE International Conference on
Computer Vision, 2016, pp. 2111–2119, , http://dx.doi.org/10.1109/ICCV.2015.
244.
[80] A.M.D.J.C, C. Rossi, EvoPose: a model-based pose estimation algorithm with cor-
respondences determination, IEEE International Conference on Mechatronics and
Automation, ICMA 2005, Niagara Falls, Ont., Canada, 2005, pp. 1551–1556, ,
http://dx.doi.org/10.1109/ICMA.2005.1626786.
[81] J. Xia, X. Xu, J. Xiong, Simultaneous Pose and Correspondence Determination Using
Differential Evolution, Proc. - International Conference on Natural Computation,
Chongqing, 2012, pp. 703–707, , http://dx.doi.org/10.1109/ICNC.2012.6234643.
[82] H. Hamledari, B. Mccabe, S. Davari, Automated computer vision-based detection of
components of under-construction indoor partitions, Autom. Constr. 74 (2017)
78–94, http://dx.doi.org/10.1016/j.autcon.2016.11.009.
[83] C. Kropp, C. Koch, M. König, Drywall state detection in image data for automatic
indoor progress monitoring, in: R. Issa, I. Flood (Eds.), Computing in Civil and
Building Engineering, ASCE, Orlando, Florida, United States, 2014, pp. 347–354, ,
http://dx.doi.org/10.1061/9780784413616.044.
C. Kropp et al.
[84] C. Kropp, M. König, C. Koch, Object Recognition in bim registered videos for indoor
progress monitoring, Proc. 20th International Workshop on Intelligent Computing
in Engineering, Vienna, Austria, 2013 9783200031456, pp. 1–10.
[85] Y.M. Ibrahim, T.C. Lukins, X. Zhang, E. Trucco, A.P. Kaka, Towards automated
progress assessment of workpackage components in construction projects using
computer vision, Adv. Eng. Inform. 23 (2009) 93–103, http://dx.doi.org/10.1016/
j.aei.2008.07.002.
[86] M.V. Escorcia, M. Dávila, J. Niebles Golparvar-Fard, Automated vision-based re-
cognition of construction worker actions for building interior construction opera-
tions using RGBD cameras, Construction Research Congress, 2012, pp. 879–888, ,
http://dx.doi.org/10.1061/9780784412329.089.
[87] R. Grompone Von Gioi, J. Jakubowicz, J.M. Morel, G. Randall, LSD: a fast line
segment detector with a false detection control, IEEE Trans. Pattern Anal. Mach.
Intell. 32 (2010) 722–732, http://dx.doi.org/10.1109/TPAMI.2008.300.
[88] C. Akinlar, C. Topal, EDLines: a real-time line segment detector with a false de-
tection control, Pattern Recogn. Lett. 32 (2011) 1633–1642, http://dx.doi.org/10.
1016/j.patrec.2011.06.001.
[89] Z. Wang, F. Wu, Z. Hu, MSLD: A robust descriptor for line matching, Pattern
Recogn. 42 (2009) 941–953, http://dx.doi.org/10.1016/j.patcog.2008.08.035.
[90] L. Zhang, R. Koch, An efficient and robust line segment matching approach based
32
Automation in Construction 86 (2018) 11–32
on LBD descriptor and pairwise geometric consistency, J. Vis. Commun. Image
Represent. 24 (2013) 794–805, http://dx.doi.org/10.1016/j.jvcir.2013.05.006.
[91] C. Zhao, H. Zhao, J. Lv, S. Sun, B. Li, Multimodal image matching based on mul-
timodality robust line segment descriptor, Neurocomputing 177 (2016) 290–303,
http://dx.doi.org/10.1016/j.neucom.2015.11.025.
[92] D. Van Opdenbosch, G. Schroth, R. Huitl, S. Hilsenbeck, A. Garcea, E. Steinbach,
Camera-based indoor positioning using scalable streaming of compressed binary
image signatures, 2014 IEEE International Conference on Image Processing ICIP
2014, 2014, pp. 2804–2808, , http://dx.doi.org/10.1109/ICIP.2014.7025567.
[93] F. Dévai, An optimal hidden-surface algorithm and its parallelization, Lecture Notes
in Computer Science (Lecture Notes in Artificial Intelligence, Lecture Notes
Bioinformatics), 2011, pp. 17–29, , http://dx.doi.org/10.1007/978-3-642-21931-
3_2.
[94] S. Savarese, L. Fei-Fei, 3D generic object categorization, localization and pose es-
timation, Proc. IEEE International Conference on Computer Vision, IEEE, 2007, pp.
1–8, , http://dx.doi.org/10.1109/ICCV.2007.4408987.
[95] D. Eggert, L. Stark, K. Bowyer, Aspect graphs and their use in object recognition,
Ann. Math. Artif. Intell. 13 (1995) 347–375, http://dx.doi.org/10.1007/
BF01530835.