See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/369568091

Thermal reality capturing with the Microsoft HoloLens 2 for energy system
analysis

Article  in  Energy and Buildings · March 2023

DOI: 10.1016/j.enbuild.2023.113020

CITATIONS READS

0 21

4 authors, including:

Tristan Kinnen Christoph Blut

RWTH Aachen University RWTH Aachen University
4 PUBLICATIONS   1 CITATION    18 PUBLICATIONS   38 CITATIONS   

SEE PROFILE SEE PROFILE

Joerg Blankenbach
RWTH Aachen University
111 PUBLICATIONS   831 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

As-built BIM View project

DigiPark View project

All content following this page was uploaded by Tristan Kinnen on 30 March 2023.

The user has requested enhancement of the downloaded file.

 Energy & Buildings 288 (2023) 113020

Contents lists available at ScienceDirect

Energy & Buildings

journal homepage: www.elsevier.com/locate/enb

Thermal reality capturing with the Microsoft HoloLens 2 for energy

system analysis
T. Kinnen ⇑, C. Blut, C. Effkemann, J. Blankenbach
Geodetic Institute and Chair for Computing in Civil Engineering & Geo Information Systems, RWTH Aachen University, Germany

a r t i c l e i n f o a b s t r a c t

Article history: Using digital methods to analyze the often highly complex structures of technical building equipment
Received 4 January 2023 (TBE) holds great potential for identifying energy savings. For digitizing TBE, we evaluated the
Revised 17 March 2023 Microsoft HoloLens 2 (MHL2), since it promises to be a mobile and cost-effective solution and found that
Accepted 23 March 2023
the MHL2 can be a suitable instrument for as-built acquisition, depending on the use-case’s accuracy
Available online 27 March 2023
requirements. The depth sensor of the MHL2 operates in a low single-digit centimeter accuracy range,
while the position tracking inaccuracy is more significant. Furthermore, we extended the acquisition sys-
Keywords:
tem with an external thermal camera to generate both, RGB and thermal point clouds. Additionally, we
Microsoft HoloLens 2
As-built acquisition
built a custom 3D calibration setup that allows us to calibrate both cameras in parallel and achieve
Thermographic point clouds promising accuracies for the level of detail of the resulting RGB and thermal point clouds and thus for
Technical building equipment the energy system analysis approach.
Sensor calibration Ó 2023 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND
Co-registration license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction that is comprehensible to humans, a geometric, topological and

The digitalization in the construction sector today is mainly
characterized by the BIM method. BIM is already regularly being
used for planning new buildings and across a building’s life cycle
[1]. Generally, BIM requires a continuous data flow. In particular,
after construction has finished, a consistent documentation of the
built situation (as-built) for creating an accurate digital twin and
ensuring its up-to-dateness is crucial. Recently, energy-related
topics have come into focus, for example, concepts that enable
the identification of potential for energy savings. The technical
building equipment (TBE) is a key factor in the improvement of
the energy efficiency of a building [2]. However, many construction
projects that were planned with BIM have encountered various
issues in which the built situation deviated from the planning
model [34]. One of the reasons is that TBE typically is highly com-
plex and the BIM method still has shortcomings in the field of
building services, for instance, the missing integration of aspects
such as building automation (BAS) and operational optimizations
[1]. Accordingly, linking the advantages of digital working methods
and energetic system optimization of building technologies entails
a considerable amount of hidden potential.
In order to process the amount of information and communica-
tion data concerning the complex structures of the TBE into a form
⇑ Corresponding author.
E-mail address: kinnen@gia.rwth-aachen.de (T. Kinnen).
semantic as-built acquisition of the TBE is an essential first step.
The comprehension of topological relationships requires a collec-
tion of geometric information with as little concealment as possi-
ble. A common method for this purpose is the use of terrestrial
laser scanner (TLS) or mobile laser scanners (MLS). For enabling
the derivation of semantic information, RGB and thermographic
image data can be utilized by including them into the point cloud.
The additional acquisition of thermographic data of the TBE fur-
thermore permits the individual components to be categorized
on the basis of the detection of heat and cold flows. As a result,
the systematic structure of the respective systems can be recon-
structed and energy-related vulnerabilities in the plant’s system
can be identified simultaneously. In general, data acquisition by
means of scanning or imaging sensor technology is comparatively
simple and can be performed rapidly nowadays. However, the
instruments needed are either expert systems (e.g., total stations),
expensive (e.g., geodetic laser scanner, mobile laser scanners), or
not well qualified for many applications (e.g., drone in indoor envi-
ronments). In contrast, the mixed reality (MR) glasses ‘‘Microsoft
HoloLens 200 (MHL2) represent a potentially cost-effective and
promising alternative, since the device already includes the neces-
sary hardware, such as a depth sensor and a RGB camera for cap-
turing colorized point clouds. Furthermore, it offers the
advantages of a head-mounted system with a high degree of
mobility and a user-friendly handling.

https://doi.org/10.1016/j.enbuild.2023.113020
0378-7788/Ó 2023 The Author(s). Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
T. Kinnen, C. Blut, C. Effkemann et al.
List of Abbreviations
2D two-dimensional
3D three-dimensional
ADK Automotive Development Kit
API Application Programming Interface
BAS Building Automation System
FPS Frames per Second
Hz Hertz
IMU Inertial Measurement Unit
MHL Microsoft HoloLens
MHL2 Microsoft HoloLens 2
mK Millikelvin
MLS Mobile Laserscanning
MP Megapixel
In this paper, we investigate and evaluate the application
potential of the MHL2 for as-built acquisition, specifically for
energy-related TBE. For this, we analyzed its sensors and their
interfaces. Based on this, we present an off-the-shelf system con-
sisting of the MHL2 and an external thermal imaging camera for
parallel acquisition of depth, RGB, and thermal data. With this
approach, we can deliver an easy-to-use and cost-effective mobile
reality capture system that does not require on-site professionals.
In addition, in order to form a basis for energy system analysis
from the acquired data, we developed and evaluated a 3D setup
for calibration of RGB and thermal sensors. Finally, with the help
of this method, we were able to develop an automated method
for generating point clouds colored with both RGB and thermal
data as a basis for the analysis of energy systems.
2. Related work
Since Microsoft released the first generation of the MHL in 2016,
it has been continuously researched for its diverse potential appli-
cations in a wide range of fields. Researchers have already recog-
nized the potential of the MHL for the acquisition of geometry
data of construction projects and investigated it under various
research focuses.
As shown in [5], a challenge with the first generation of MHL
was that the captured object or environment data was only avail-
able to the user in the form of meshes instead of point clouds.
However, a mesh is a discretization of a geometric domain into
small simple shapes [6]. Therefore, the process of generating a
mesh is always based on the existence of a quantity of geometric
point data. Accordingly, the MHL scans its surroundings with the
built-in depth sensor and initially generates a point cloud before
automatically converting it into a triangular mesh. As access to
the raw data of the MHL sensors is not provided, [7] titled the uti-
lization of the MR glasses as a research tool as severely limited.
Furthermore, the first generation of the MHL suffers from a lack
of high-quality hardware to enable accurate geometric data acqui-
sition, resulting in an inability to compete with the geometric
accuracy achieved by a TLS as demonstrated by [7–10]. In [9] the
MHL was investigated for as-built acquisition by scanning an
empty apartment several times and comparing the generated
meshes to a point cloud captured by a TLS. The presented results
of the paper show a marginal variance between different acquisi-
tion processes of an identical environment with the MHL. The
observed mean variance between the acquired data of the MHL
and the data of the TLS amounted to 2.4 cm inside of a single room.
With the addition of neighboring rooms and larger distances
between the surface to be captured and the depth sensor of the
MHL, the deviations increased. In conclusion, it was found that
2
Energy & Buildings 288 (2023) 113020
MR Mixed Reality
PX Pixel
RGB Red-Green-Blue
TBE Technical Building Equipment
TLS Terrestrial Laserscanning
ToF Time of Flight
UV Axis designation of a two-dimensional coordinate sys-
tem
UWP Universal Windows Platform
WDP Windows Device Portal
the transition from triangular meshes captured with MHL to
semantically enriched, topologically correct building models is still
a challenge and holds much potential for future research. The
result of [9] are confirmed by [8] and [11] by recording an average
distance of the meshes of the MHL to the registered point cloud of a
laser scanner in the low to mid-single-digit centimeter range. With
this achieved accuracy, the MHL could fulfill the requirements for
an as-built acquisition as well as for a subsequent rule-based
reconstruction of a semantically enriched and topologically correct
model. [8] furthermore investigated the capabilities of the MHL
regarding its localization (in the sense of pose determination) in
indoor environments as well as its spatial mapping of indoor
scenes. To analyze the stability of the distance measurement, a
wall with an initially fully cooled down MHL was captured for
100 minutes in order to compare the temporal variation of the
resulting distance data relative to the first frame. The results
demonstrated that the MHL only achieves stable range measure-
ments after an operating time of more than 60 minutes. Evalua-
tions of the MHL2 in terms of suitability as an instrument for as-
built acquisition have not been published to date. However, first
results for the evaluation of the sensors of the MHL2 are shown
by [12], whose investigations revealed an average deviation of
2 cm for its hand-tracking accuracy.
The function for creating point clouds colored with data from
RGB images has become a standard feature of many commercial
software products from laser scanner manufacturers. [5] and [7],
on the other hand, presented two different methods to realize
the generation of colored point clouds in real time with sensor data
of the MHL. The method of [7] is based on the external acquisition
of geometric data by a higher quality depth sensor and an addi-
tional RGB camera connected to a computer. Via a WiFi connection,
the data is transferred to the MHL in real time. The external RGB
camera acts as an interface for the mapping process by calibrating
with the RGB camera of MHL via a manually set up checkerboard.
This enables the data from the external depth sensor to be trans-
formed first to the coordinate system of the external RGB camera
and then to the coordinate system of the MHL through the existing
calibration. One advantage of this system is that higher quality
results can be achieved by using external sensors. However, the
system design limits mobility during the acquisition process by
carrying external sensors. In contrast, the method developed by
[5] focuses exclusively on the data from the built-in sensors of
the MHL by coloring the dynamic meshes in real time using the
images from the built-in RGB camera. For this purpose, the tech-
nique of UV mapping is used, which projects a two-dimensional
image onto the surface of a three-dimensional model for texture
mapping as also shown by [13]. The abbreviation UV denotes the
axes of the two-dimensional image. Consequently, the system first
T. Kinnen, C. Blut, C. Effkemann et al.
creates a UV texture into which the RGB data are transferred. Then
the colored UV texture is mapped onto the mesh. To realize this
process in real time, an adaptive mapping scheme is used that dis-
tributes each triangle of the dynamic mesh evenly over the texture
and adapts to the new geometries without affecting existing color
data. Approaches to generate colored point clouds from MHL2 sen-
sor data have not yet been published. However, [14] presented an
end-to-end framework for transferring and interacting with 3D
point clouds to the MHL2 system.
The development of methods for the generation of thermo-
graphic point clouds are in principle documented in different con-
texts. [15] discussed the topic of how a 3D point cloud of the
environment with a TLS and a thermographic panorama taken with
a nodal point adapter can be linked together. [16] has shown a
method to generate a 3D building model with color and thermal
information from the acquired data enabled by a handheld system
consisting of a ‘‘Microsoft Kinect” and a simple infrared camera.
[17], on the other hand, presented a fully automated method for
generating point clouds colored by RGB and thermal data. In this
case, a laser scanner, an RGB camera and a thermal camera were
permanently mounted on a freely controllable robot. Therefore,
the advantage that all sensors are calibrated is given. However,
the robot is not variable in height and can therefore only record
all data from a constant small distance above the ground, which
means that parts of the environment to be captured could not be
recorded.
Consequently, some methods for generating thermal point
clouds in general do already exist. However, no procedure has
yet been developed that makes it possible to link thermographic
image recordings with the depth points recorded by the MHL2.
Furthermore, in contrast to its predecessor, the MHL2 has not yet
been analyzed for its potential for its application as an as-built
acquisition system.
3. Material and methods
In this chapter, we first introduce the sensor system of the
MHL2 and its technical data. Subsequently, we present our method
for the sensory extension of the acquisition system and for the
automated sensor data fusion for the generation of colored point
clouds.
3.1. Working with the MHL2
The MHL2 MR glasses are a head-mounted and therefore hands-
free operable computer that creates a digital layer of information
within the real world, which allows users to interact with holo-
grams that remain in a specific position and behave like physical
objects [18]. From a technical point of view, the constant acquisi-
tion of the spatial environment is fundamental for operating in
MR. Accordingly, the hardware of the MHL2 includes various sen-
sors that are integrated in the visor of the glasses. An overview of
the sensors of the MHL2 and their functions is shown in Fig. 1 and
Table 1.
This technical equipment enables real-time 3D mapping of the
immediate environment and self-localization within the acquired
3D data [8]. The position of the MHL2 can be determined from
the data of the four grayscale cameras and the inertial measure-
ment unit (IMU). Therefore, it represents a highly potential system
for the as-built data acquisition in the construction sector and
offers the possibility of direct visualization of the results and a
higher mobility level compared to other data acquisition systems.
According to the sensor data, the built-in infrared cameras are
responsible for the eye tracking. Consequently, these cameras are
focused on the eyes of the user and are not suitable for recording
3
Energy & Buildings 288 (2023) 113020
Fig. 1. Overview of the sensors of the MHL2.
Table 1
List of sensors of the MHL2 and their functions [19].
Sensors Quantity Resolution Function
Grayscale visible light 4 – Head-Tracking
cameras
Infrared cameras 3 – Eye-Tracking
Time-of-flight depth sensor 1 1 MP Depth acquisition
Accelerometer, Gyroscope 1 – Orientation
and Magnetometer determination
(Inertial Measurement
Unit (IMU))
RGB camera 1 8 MP Color image and
Image, video recording
1080p30
Video
thermal information of their surroundings. For creating thermal
point clouds, it is necessary to integrate an additional external
sensor.
The depth sensor of MHL2 operates in two different modes for
near and long-distance ranges. The near range mode is aimed to
track the hand movements of the user (Articulated Hand Tracking).
Opposite is the long-distance range mode, which provides the geo-
metric mapping of the environment. In both cases, the respective
distance is determined via the Time of Flight (ToF). The arrange-
ment and the different lenses of the MHL2 camera sensors result
in different fields of detection of the environment. These overlap
each other partly to a greater and partly to a lesser extent. A
detailed illustration of this is given in [9]. The fact that the detec-
tion fields of the RGB camera and the depth sensor completely
overlap provides an advantage by ensuring that a high amount of
pixel of a RGB image can be assigned to the acquired depth values.
Our approach of a co-registration for simultaneous mapping of
depth, RGB and thermal data requires an access to the raw data
from the MHL2 sensors. For the advancement of academic or
industrial research and development processes in thematic fields
in which the MHL2 may be applied, Microsoft has published the
option of a Research Mode. With this option, access to the raw data
of the MHL2 sensors is provided by various application program-
ming interfaces (APIs) and tools [20]. The research mode can be
activated in the Windows Device Portal (WDP), which needs to
be previously set up in the settings of the MHL2 under the options
of the developer mode.
Since each sensor operates in an individual coordinate system,
transformation matrices that include rotational as well as transla-
tional parameters are generated for each acquired frame. There-
with, the different sensor data must be transformed into a
common coordinate system. In this context, the left front grayscale
camera serves as the reference system for the MHL2. The data can
be subsequently transformed from the reference coordinate sys-
T. Kinnen, C. Blut, C. Effkemann et al. Energy & Buildings 288 (2023) 113020

tem of the left front greyscale camera into world coordinates, if the
location of at least one position in the acquired area is defined. To
obtain access to the data provided by the research mode, we used
the StreamRecorder application from the HoloLens2ForCV1 reposi-
tory, which can be manually set to save the data of the ToF depth
sensor and the RGB camera. In this way, the data from the MHL2 sen-
sors can be recorded for a user-defined period of time, and subse-
quently be stored on the device. After the acquisition, the data can
be downloaded from the WDP. The data of the ToF depth sensor is
available as 2D depth images. So, the data package of an acquisition
process with the StreamRecorder application includes 2D images of
the depth camera and the RGB camera as well as all necessary geo-
metric values needed for the transformation of different local sensor
data into a common coordinate system, including transformation
matrices and calibration information of the RGB camera. Based on
this data structure and the user-friendly interface of the StreamRe-
corder application (Fig. 2), the MHL2 represents a promising acquisi-
tion system that does not require qualified personnel to operate.
Fig. 3. FLIR Boson ADK thermal camera.

3.2. System expansion interfaces

As mentioned in 2.1, the infrared sensors of the MHL2 are not
suitable for capturing thermal data. Therefore, we have extended
the system with the FLIR Boson ADK thermal camera (Fig. 3). The
technical data of the FLIR Boson ADK are listed in the following:
 Resolution: 640  512px
 Frame rate: 60 Hz
 Sensitivity: 50mK
 Aperture angle: 32°
 IP67 case
3.2.1. Mounting the FLIR Boson ADK to MHL2
Since we did not intend to limit the factor of the high amount of
mobility that the MHL2 entails as a system for the data acquisition,
we followed the approach of attaching the FLIR Boson ADK thermal
camera directly to the MHL2. However, in order to achieve high
quality thermographic point clouds, a constant offset and orienta-
tion of the sensors of the MHL2 and the FLIR Boson ADK thermal
camera over the time of the acquisition process is important. We
adapted and optimized a publicly available 3D-printed camera
mount to attach the thermal camera to the MHL2. Particular atten-
tion was paid to maximize the robustness and stability of the
mount for the FLIR Boson ADK for the co-registration. In addition,
a small tilt angle was added to increase the overlap between the
FLIR Boson ADK and the depth and RGB camera of the MHL2.
In the following, the individual components of our 3D printed
mount are illustrated in Fig. 4 and our resulting acquisition system
is presented in Fig. 5.
During the acquisition process, images from the depth camera
and the RGB camera of the MHL2 are captured in parallel via the
StreamRecorder application, from which RGB point clouds can be
generated in postprocessing. For the additional generation of ther-
mal point clouds, a parallel acquisition of thermal images is
required. Unfortunately, the extension possibilities of the MHL2
are limited, so that the USB-based FLIR Boson ADK cannot be con-
nected directly. Therefore, we connected the FLIR Boson ADK to an
external computer and incorporated it into the system architec-
ture. The external computer is responsible for reading and trans-
mitting the thermal images to the MHL2. For this, we developed
a client application that runs on the external computer and a
Universal Windows Platform (UWP) server application for the
MHL2. The communication is based on the Transmission Control

Fig. 2. User Interface of the StreamRecorder-App.

4
T. Kinnen, C. Blut, C. Effkemann et al.
Fig. 4. 3D-printed mount for attaching the FLIR Boson ADK thermal camera to the MHL2.
Fig. 5. MHL 2 with FLIR Boson ADK thermal camera.
Protocol. In this way, a signal is transmitted from the MHL2 to the
external computer at the time when the user starts an acquisition
process in the StreamRecorder application, initiating the parallel
acquisition of thermographic image data. The system times of the
external computer and MHL2 are synchronized, so that each depth,
RGB and thermal image can be stamped with the acquisition time.
These timestamps are based on the standard defined by Windows
(one hundredth of a nanosecond). With this approach, data from
the depth, RGB and thermal cameras can be uniquely mapped to
each other as part of a co-registration process. As a result, the color
images can be combined with the depth data to generate colored
point clouds. An example of the raw data of the depth, RGB and
thermal sensors for one frame is visualized in Table 2 for the acqui-
sition of a heater.
In order to use the FLIR Boson ADK thermal camera for captur-
ing thermal images intended for coloring point clouds, it is neces-
sary to reconfigure the system settings for the color visualization
for each new data acquisition, as it is basically set to define the
color scale always in relation to the warmest and coldest areas of
a respective exposure field. This setting leads to the risk that the
color representation of the same objects may differ in the different
frames, resulting in incoherent colors in the point cloud. To fix this,
the damping factor must be adjusted in a way that the color repre-
sentation is applicable for the current scene.
5
Energy & Buildings 288 (2023) 113020
3.2.2. Calibration
Since the thermal camera is mounted on top of the MHL2, it is
shifted vertically. To combine the thermal images with the depth
images, it is necessary to determine the exact transformation of
the thermal camera sensor and the depth camera sensor. This is
done during a system calibration, before starting the capturing
process. In general, camera calibrations can be performed image-
based on the basis of corresponding feature points. The RGB cam-
era of the MHL2 is suitable as a reference in this context, since the
RGB images, in contrast to the depth images, enable an unambigu-
ous detection of the feature points. Furthermore, the RGB camera is
already calibrated to the depth camera as a part of the system of
the MHL2 and its calibration parameters are provided within the
data of the StreamRecorder application. As part of the thermal
camera calibration, the intrinsic and extrinsic as well as the deriva-
tion parameters are determined. The intrinsic parameters include


the camera-specific parameters, such as the focal length f x ; f y


and the image principal point cx ; cy . The extrinsic parameters,


on the other hand, describe the translational t x ; t y ; tz and rota-
tional ðRÞ offset between the RGB camera and the thermal camera
expressed by a transformation matrix, which is composed by a
rotation matrix and a translation vector. For RGB cameras, black
and white checkerboard patterns printed on paper are therefore
often used, as the corners of the squares can be detected easily
and robustly as feature points. However, simple paper based black
T. Kinnen, C. Blut, C. Effkemann et al.
Table 2
Raw data of the depth, RGB and thermal camera.
Depth camera
and white checkerboard pattern that cannot generate any heat are
unsuitable for the calibration of thermal cameras, since the sensor
reacts to thermal radiation. For this reason, we developed a calibra-
tion system (Fig. 6). Light bulbs were inserted in an irregular man-
ner to serve as heat sources that can be detected by the thermal
camera. In addition, uniquely identifiable markers were placed
on the front side with the light bulbs located in their respective
centers. These centers were determined photogrammetrically in
advance from the image set of a high-resolution single lens reflex
camera. The resulting positions for the centers of the markers serve
as ground truth for all calibration processes of the RGB camera of
the MHL2 and the FLIR Boson ADK thermal camera. The respective
marker centers can be determined from the images of the MHL2
RGB camera in reference to the ground truth data within a calibra-
tion process. In the thermal images, the light bulbs are each repre-
sented as heat sources in the form of dots. Since the centers of the
markers and of the light bulbs are identical, the centers can be
determined from the images of the thermal camera equivalent to
the RGB images in reference to the ground truth. The extrinsic
and intrinsic calibration parameters of the thermal camera can
Fig. 6. Setup for calibrating the FLIR Boson ADK thermal camera to the RGB camera of the MHL2.
Energy & Buildings 288 (2023) 113020
RGB camera Thermal camera
subsequently be determined. To ensure that the most accurate
parameters possible result from a calibration process, it is essential
that the positions of the various sensors are constant between the
respective images. For this reason, we placed the acquisition sys-
tem on a stationary surface for the duration of the recording.
3.3. Generation of colored point clouds
The process for generating point clouds colored with both RGB
and thermal data starts with the selection of a specific image of
each sensor. While the RGB camera of the MHL2 captures 30
images (or frames) per second (FPS), the frame rate of the depth
camera is 1 to 5 FPS and is therefore not constant [21]. By relating
the respective scan times to the floor area of the acquired environ-
ment of a scan process, inferences can be made that the amount of
taken depth images depends on the movement speed of the MHL2
and thus on the head movements of the user during the acquisition
process. The data from different scans show that the frame rate of
the depth camera increases equivalently to a faster acquisition
speed. As a result, the number of sensor data used as a starting
6
T. Kinnen, C. Blut, C. Effkemann et al. Energy & Buildings 288 (2023) 113020

point for the generation of colored point clouds is unequal, so that StreamRecorder software application at the start of each acquisi-
a selection must be made as to which color images are assigned to tion process, so that data can be transferred from the reference sys-
the respective depth images. As a solution to this problem, we tem of the left front grayscale camera via the world coordinate
selected the corresponding images of the color sensors based on system to the coordinate system of the RGB camera. Subsequently,
the known timestamps of the files. The reference element for this the 3D points are converted to the coordinate system of the RGB
operation is the grayscale image of the depth sensor. Based on camera with the transformation matrices provided by the MHL2.
the timestamp of this image, the respective timestamp with the Using the known extrinsic calibration parameters, which include
smallest temporal difference is determined in the lists of all times- the rotational and translational spatial offset between the MHL2
tamps of the data sets of the RGB camera and the thermal camera. RGB and the FLIR Boson ADK thermal camera, the depth data can
All following coordinate transformations are based on a pinhole be transferred in parallel to the coordinate system of the thermal
camera model as presented in [22] (Fig. 7). The overall schematic camera at this point. Continuing, the depth points are projected
process workflow of our mapping method is shown in Fig. 8. onto the two-dimensional image planes of the RGB and thermal
For transforming 2D image points ðu; v Þ to 3D coordinates camera using the respective intrinsic camera parameters. In order
ðX C ; Y C ; Z C Þ in the corresponding camera coordinate system can to optimize the mapping results, each frame of the FLIR Boson ADK
be performed by using equation (1) [23,24]: thermal camera is previously corrected for distortion with the help
2 3 2 32 3 of the determined distortion parameters of the calibration process.
u fx 0 cx XC In this manner, the pixels of the color sensor frames that are pre-
6 7 6 76 7
s4 v 5 ¼ 4 0 fy cy 54 Y c 5 ð1Þ sent in the temporally allocated viewing area of the depth sensor
1 0 0 1 Zc can be identified and each depth point can be within the intersect-
ing areas is thereupon assigned a color value from the respective
The transformation of 3D coordinates to world coordinates color sensor data.
ðX w ; Y w ; Z w Þ or to a different camera coordinate system, on the For generating a point cloud that has points are simultaneously
other hand, requires the multiplication with the extrinsic parame- assigned color values from the image data of the RGB and the FLIR
ters as shown in equation (2) [23,24]: Boson ADK thermal camera, we have followed the approach of
2 3 transferring the depth data and the respective assigned points from
2 3 2 3 Xw
x0 r 11 r12 r13 tx 6 7 the data of the color sensors to a common image plane. To maxi-
6 7 6 76 Y w 7 Xc Yc mize the accuracy of the representations of the object geometries
Z c 4 y0 5 ¼ 4 r 21 r22 r23 ty 56 7; with x0 ¼ and y0 ¼
4 Zw 5 Zc Zc in the point cloud, a virtual pinhole camera is generated whose
1 r 31 r32 r33 tz
1 projection center corresponds to the depth sensor of the MHL2.
ð2Þ This ensures that the highest possible amount of acquired depth
points is transferred to the point cloud. For the same reason, the
After assigning the sensor images for the respective frame, the image sizes are chosen to be identical to those of the grayscale
depth points are transformed from the 2D image plane into 3D images of the depth camera and the principal point of the image
points in the coordinate system of the depth sensor. In the follow- is set to correspond to the center of the image. The depth points
ing, the 3D depth points are converted to the coordinate system of are projected onto the image plane at first. Afterwards, the respec-
the RGB camera. For this purpose, the depth images are first pro- tive color images are transformed onto the image plane of the pin-
jected onto the reference system of the left front grayscale camera. hole camera based on the previous assignment of the RGB and
Afterwards, the points are transformed from this coordinate sys- thermal image points to the depth points. In the following, the
tem into world coordinates. The reference point for defining a resulting images for the depth, RGB and thermal data are stored.
world coordinate system is determined automatically by the

Fig. 7. Illustration of the relations of image plane, camera coordinate system and world coordinate system in the pinhole camera model [22].

7
T. Kinnen, C. Blut, C. Effkemann et al.
Fig. 8. Schematic representation of the mapping method.
This entire process is repeated for each existing depth image. For
the generation of the colored point clouds, the stored image pairs
are finally superimposed. Because of the defined data structure,
the assignment of color values from the RGB and thermal data
can consequently be retrieved pixel by pixel for each depth point.
After the mapping, the depth points are transformed again into 3D
world coordinates and written into a point cloud file with their
two-color values.
4. Results
In the following, we first present our results on the evaluation
of the MHL2 in terms of its usability for the as-built acquisition.
Afterwards, we summarize our critical investigation of our method
for calibrating the MHL2 with the FLIR Boson ADK thermal camera
and its results. Lastly, we present our obtained results of generat-
ing colored point clouds resulting from sensor data fusion.
4.1. Evaluation if the MHL2 in terms of depth data acquisition
In order to evaluate the application of the MHL2 as a tool for the
as-built building acquisition, the point cloud quality and the
achieved accuracies of the ToF depth sensor have been investi-
gated. First, a representative test environment was selected. As
part of the energyTWIN2 project, a domestic hot water simulator
provides access to all components of a TBE system so that developed
methods for the optimization of the energy efficiency can be tested
and evaluated. The water simulator is located in a large-scale exper-
imental hall over three levels on a steel frame. The lowest and upper-
most levels each contain the main components of the TBE system,
while the middle level has two simulation rooms for bathrooms with
the corresponding connections and fittings. As a reference acquisi-
tion system, a terrestrial laser scanner (TLS) Riegl VZ-400 was used
in combination with an external RGB camera. According to the man-
ufacturer, the data acquired by the TLS VZ-400 has a geometric accu-
racy of a maximum deviation of five millimeters [26].
From the analysis of numerous acquisition series with the
MHL2, we have found that the risk of multiple reflections of the
emitted rays significantly increase at longer distances to the
objects to be scanned. Therefore, we recommend short scan dis-
2
energyTWIN is a research project that investigates new sensor-based and AI-
based methods for the digital, BIM-based commissioning of technical systems in
building structures and their energetic system optimization [25].
8
Energy & Buildings 288 (2023) 113020
tances to the objects and constantly slower movements of the
acquisition system in order to achieve higher accuracies of the
depth data. Especially in the field of TBE acquisition, the high
amount of metal surfaces in combination with punctual artificial
lighting poses a challenge. Therefore, the examination of the
MHL2 point clouds initially reveals a high scattering of points,
especially in the areas that represent the surfaces of objects. In
addition, it is evident from the point cloud analysis that this scat-
tering increases for objects that could only be captured from a
greater distance due to the surrounding structure, so that signifi-
cant details and structures are displayed more inadequately within
the point cloud. An example of this is illustrated in Fig. 9. The figure
presents the frontal and lateral view from the point cloud gener-
ated from depth images taken by the MHL2 of four jacketed pipes
with control elements and an electronic distribution box, which
stood under direct lighting and could only be acquired from a dis-
tance of more than half a meter. In the frontal section of the point
cloud, some points are discernable in the free space between the
four pipes, which is a result of the high number of reflections in
this area. Furthermore, the frontal and lateral sections indicate
almost imperceptible representation of the contours of the elec-
tronic distributor box and the four control elements in the point
cloud. Similarly, the cables located in this area are not represented
in the point cloud. Objects with a cross-sectional area smaller than
one centimeter are generally only recognizable in the point clouds
of the MHL2 if they were captured from a very short distance.
Opposite to this, the parts of the captured environment, which
were captured from a short distance and with a low level of occlu-
sion, are represented with clearer structures in the point cloud, as
shown in Fig. 10 by the example of a black pipe equipped with a
valve. However, the point cloud was manually processed by filter-
ing and removing the scatter points in the empty space in this case,
aiming to provide a distinct contrast. Nevertheless, the removed
points and the scatter are significantly less existent compared to
the example from Fig. 9, so that the shape of the pipe and its valve
do not have to be estimated, but are clearly recognizable which
enables the determination of geometries as well as the derivation
of topological relationships.
Like mobile laser scanners, the MHL2 has advantages compared
to a TLS of a time-efficient acquisition process and the ability to
acquire object geometry with little occlusion, which can simplify
the derivation of topological relationships. However, mobile acqui-
sition methods often exhibit significant systematic drift [27]. This
drift, in addition to the scan duration and the distance of the acqui-
T. Kinnen, C. Blut, C. Effkemann et al.
Fig. 9. Example of the effect of larger acquisition distances in combination with the problem of multiple reflections.
Fig. 10. Prepared output display for the acquisition of TBE components from short distance with the MHL2.
sition path, is fundamentally dependent on its extend on the IMU.
To investigate the drift of the MHL2, we processed a six-minute
scan with an acquisition path of 75 m and compared it to the
ground truth data from the TLS. Meanwhile, the MHL2 acquisition
was performed without visiting a location twice. The point clouds
of the MHL2 and the Riegl VZ-400 were registered only on the basis
of prominent control points in the immediate vicinity of the start-
ing point of the MHL2 acquisition. Using this approach, an average
drift of 75 cm was detected at the end of the MHL2 acquisition path
by measuring distances of identical points in both point clouds as
shown in Fig. 11. In terms of the length of the acquisition path and
the acquisition time, this corresponds to a drift of one centimeter
per meter and 0.2 cm per second, respectively.
To investigate the accuracy of the distance determination of the
MHL20 s depth sensor, a comparison method was chosen that
excludes the influence of drift. For this reason, we examined a sec-
tion of a wall plane taken immediately at the beginning of the
acquisition process with the MHL2 and from a distance of less than
two meters. In this case, the control points for matching the align-
ment and orientation of the MHL2 point cloud with the point cloud
of the TLS were set in close proximity to the area from which the
imaged wall section originated. Thus, a drift deviation can be
excluded and the detection accuracy of the depth sensor of the
MHL2 can be examined. As a result of this observation, a maximum
deviation of two centimeters can be observed in the point cloud of
the MHL2 in the area of the wall surface as shown in Fig. 12.
9
Energy & Buildings 288 (2023) 113020
4.2. Calibration of the MHL2 with the FLIR Boson ADK thermal camera
The foundation for any successful fusion of camera sensor data
consists in calibrating the systems as accurately as possible. For
this reason, we iterated the calibration process numerous times
for the MHL2 and FLIR Boson ADK camera systems. Thereby, the
number of recording sites differed in the individual implementa-
tions. Generally, the calibration quality increases with the number
of images taken from different positions and angles.
In order to analyze the consistency of the calibration parame-
ters determined by our method, we first investigated the devia-
tions of the certain parameters from different calibration
processes. Table 3 contains the specific intrinsic parameters of
the FLIR Boson ADK camera from four different calibrations with
their respective standard deviation. Generally, intrinsic camera
parameters are always interdependent [28]. For this reason, and
due to the fact that, on the one hand, the bulbs in the calibration
setup are of different depths at each feature point, and on the other
hand, the FLIR Boson ADK thermal imaging camera is equipped
with the function of an automatic focus, occasional variations of
the determined intrinsic values in our calibration are unavoidable.
Besides determining the intrinsic and extrinsic parameters, we
also identified the distortion factors within the calibration process
to correct each FLIR Boson ADK camera image prior to the mapping
process to the extent that the distortion minimally affects the
result. To illustrate the visual results of the mapping of an image
pair realized under the implementation of previously determined
T. Kinnen, C. Blut, C. Effkemann et al. Energy & Buildings 288 (2023) 113020

Fig. 11. Deviation of the point cloud of the MHL2 (yellow) and the point cloud of the VZ-400 (green) as a result of drifting of the pose along the acquisition path. (For
interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 3
Compilation of certain intrinsic camera parameters of the FLIR Boson ADK camera
from different calibration processes.

Calibration process Focal length Principal point Principal point

(X-Component) (Y-Component)
1 13.4524 0.5221 0.0784
+/- 0.0750 +/- 0.1678 +/- 0.1320
2 13.4186 0.0053 0.0113
+/- 0.0304 +/- 0.0249 +/- 0.0272
3 12.6655 0.0133 0.0081
+/- 0.0460 +/- 0.0263 +/- 0.0189
4 13.3668 0.0091 0.0517
+/- 0.0257 +/- 0.0313 +/- 0.0331

* All dimensions in [mm].

ginal deviations exist at the edges of the thermal image, which is

Fig. 12. Evaluation results of the accuracy of the MHL2 depth sensor (yellow)
versus a TLS (red). (For interpretation of the references to color in this figure legend,
the reader is referred to the web version of this article.)
calibration parameters is shown below in Fig. 13, that the image
coordinates of the FLIR Boson ADK camera and the RGB camera
of the MHL2 match particularly accurately in the central area
around the principal point of the thermal image. However, mar-
partly due to camera-specific factors.
Since the calibration board was photogrammetrically measured
in advance with an image composite of a high-resolution camera,
which provided the exact coordinates of the feature points, the
RGB image of the MHL2 is transformed into the same coordinate
system in the first step of the calibration. Subsequently, the devia-
tions of the image coordinates from the control points are calcu-
lated on the image plane. The result is the relative orientation of
the MHL2 as a transformation matrix. In the second step of the cal-
ibration process, the thermal image is transferred to the MHL2
coordinate system and the relative orientation of the FLIR Boson
ADK camera is determined as before by measuring the deviations
of the image coordinates of all control points of the thermal camera
with those of the RGB camera of the MHL2 and calculating the best
mathematical approximation to match them. This process is
repeated for each image pair of an acquisition location. By imagin-
ing the connection between the projection center, the image plane
and the control points as a ray, the totality of all rays is bundled in

10
T. Kinnen, C. Blut, C. Effkemann et al.
Fig. 13. Visual representation of the calibration results between the MHL2 and the FLIR Boson ADK thermal camera.
the projection center. Fig. 14 schematically shows the bundle
adjustment for the calibration of the sensor systems.
To classify and evaluate the calibration results, it is necessary to
consider the residuals in the image after bundle adjustment. How-
ever, since the determined deviations of individual image coordi-
nates are not constant within the analysis of an image pair and
the derived transformation matrix merely describes the best possi-
ble mathematical approximation, residuals occur. To evaluate the
calibration results, we analyzed these residuals, which can also
be referred to as the standard deviation of the image coordinates.
Therefore, we determined the standard deviations in the direction
of the two axes of the image plane separately for each RGB and
thermal image pair during the calibration process. From these
determined data, we subsequently determined the mean values
of the standard deviations in the direction of the two image axes
for each calibration process, in order to calculate the standard devi-
ation of the image coordinate. The results of this method are
shown in Table 4 for four different calibration processes.
According to the determined data, both sensor systems point
out a minor standard deviation of the image coordinates from
the bundle adjustment. For the FLIR Boson ADK thermal camera,
the maximum value amounted to less than two pixels for a pixel
size of 0.012 mm. For the MHL2, on the other hand, no maximum
value of more than one pixel was determined for a pixel size of
0.05 mm. Furthermore, the results show that the accuracy of the
calibration method improves simultaneously with the number of
image pairs.
However, the presented results are based on a calculation that
includes the specific relative orientations of the sensor systems
of each image pair of a calibration. But for the subsequent sensor
data fusion and generation of colored point clouds, we used the
mean relative orientation of all image pairs of a calibration process
11
Energy & Buildings 288 (2023) 113020
for the FLIR Boson ADK thermal camera. The use of the mean rela-
tive orientation results in fewer degrees of freedom for the system.
As a result, the residual gaps of the bundle adjustment and conse-
quently the standard deviations of the image coordinates increase.
To evaluate the effects of this, some of the degrees of freedom were
replaced with constants by assuming the known relative orienta-
tion of the MHL2 to be correct and using multiple image pairs from
a calibration process to determine the mean relative orientation of
the FLIR Boson ADK thermal camera. Subsequently, the thermal
images were again projected into the coordinate system of the
high-resolution camera and the deviations to the control point cen-
ters were measured. Based on these data, we were able to deter-
mine the standard deviations of the image coordinate similarly
to the previous method. Thereby, the result of a total of 608 mea-
surements was a standard deviation of the image coordinate of the
FLIR Boson ADK thermal camera of 7.246 pixels or 0.087 mm.
4.3. RGB- and thermographic colored point clouds
Generally, the results are significantly improved if the acquisi-
tion distance during the acquisition process is less than 2 m, as
shown in the evaluation of the MHL2 for depth data acquisition
(Section 3.1). Especially for the coloring results in the point cloud,
we recommend constant and slow head movements with the
acquisition system. Furthermore, in the comparison of different
acquisition processes realized conclusively that the starting point
of the acquisition is of crucial importance for the color representa-
tion of the thermal data due to the system configuration for the
acquisition of absolute color values. The finest coloring results
were obtained when the acquisition process began with the cap-
ture of a representative component of the TBE, which was charac-
terized by high heat or cold intensities relative to the other
T. Kinnen, C. Blut, C. Effkemann et al. Energy & Buildings 288 (2023) 113020

Fig. 14. Schematic illustration of the bundle adjustment process for the sensor calibration.

Table 4
Standard deviations of the image coordinates after bundle adjustment for images taken from a distance of between 1.50 and 2 m.

Calibration Number of Image Pairs Standard Deviation of the Image Standard Deviation of the Image
Coordinates (FLIR Boson ADK, Coordinates (MHL2,
pixel size = 0.012 mm) pixel size = 0.05 mm)
[mm] [Pixel] [mm] [Pixel]
1 4 0.0221 1.8403 0.0047 0.9477
2 9 0.0081 0.6741 0.0018 0.366
3 7 0.0079 0.6543 0.0047 0.9477
4 15 0.0065 0.5379 0.0015 0.2967

components of the systems. In compliance with this, we achieved
detailed color representations of the RGB and thermal camera data
in the resulting point clouds. An example of the results is shown in
Fig. 15 for an acquisition of a technical plant.
The figure shows that the representation of distinct structural
edges as well as the color representation is especially accurate
for large objects. In addition, even smaller color gradients such as
isolated heat regions on the walls or reflected heat radiation on
the neighboring objects are transferred to the 3D point cloud
unambiguously and identically to the original sensor data. How-
ever, particularly objects with a comparatively small cross-
section, such as valves or narrow pipes, occasionally reveal indis-
tinct color gradients along the edges of the structure due to scat-
tered color transitions between the various objects. Furthermore,
chromaticity values are also assigned to points in actually empty
space, which is due to the previously mentioned problem of multi-
ple reflections and the method of sensor data fusion on a two-
dimensional level. Nevertheless, the acquisition system and the
presented method enable the generation of colored 3D point
12
clouds, which represent a promising basis for the collection of geo-
metric information as well as for the derivation of topological and
semantic data.
5. Discussion
We developed a functional sensor system for the acquisition of
geometric, real color and thermal environmental data consisting of
the Microsoft MHL2 and the FLIR Boson ADK thermal camera,
implemented a method for the sensor data fusion for the genera-
tion of point clouds colored with RGB and thermal data and evalu-
ated the results. The use of this head-mounted sensor system for
as-built acquisition offers the advantage of a high degree of mobil-
ity, enabling low occlusion capture of objects. Moreover, the sys-
tem provides the advantage of a user-friendly interface, which
eliminates the need for specialized personnel on site. The evalua-
tion of the accuracy of the MHL2 depth camera revealed a maxi-
mum deviation of 2 cm. Therefore, the MHL2 is not able to
compete with high-priced sensor systems, but it represents a
T. Kinnen, C. Blut, C. Effkemann et al.
Fig. 15. Exemplary representation of the obtained colored point clouds of a TBE element.
promising and cost-effective alternative for use cases whose
requirements are compatible with the achievable accuracy of the
depth camera of the MHL2. However, the MHL2 shows a large drift
in the course of longer recordings and paths. For this reason, it is
not recommended to use the MHL2 for large-scale acquisitions
without splitting them into several scans. Furthermore, since the
presented acquisition system aims to collect data for the energetic
analysis of TBE components, which often have metallic surfaces
and are placed under artificial light, multiple reflections may occur.
Due to this, the point cloud can partially exhibit strong noise, but
this is a typical challenge that acquisition systems must deal with.
Nevertheless, we have found that although manual processing of
the point cloud may be required for some components, most rep-
resentations of the components in the point cloud show a sufficient
level of detail. Future work towards integrating an automated pro-
cess for point cloud processing, e.g., using neighborhood analysis to
identify points in empty space, could counteract this issue. For the
sensor data fusion of the depth, RGB and thermal camera, a custom
calibration setup was developed to determine the intrinsic and
extrinsic camera parameters of the FLIR Boson ADK thermal cam-
era in relation to the MHL2 system. By determining the standard
deviations of the image coordinates of the RGB camera of the
MHL2 and the FLIR Boson ADK thermal camera from the bundle
adjustment equation, the dispersion for both systems were shown
to vary within a spectrum of a low single-digit pixel number. Since
the calculation of these values includes the specific relative orien-
tation of each thermal image and not the averaged relative orien-
tation from all image pairs of a calibration process, these results
primarily serve to prove the basic functionality and to classify
13
Energy & Buildings 288 (2023) 113020
the realizable accuracy of the calibration setup. Considering the
averaged relative orientation for the determination of the standard
deviations of the image coordinates of the FLIR Boson ADK thermal
camera, the value of the scatter increases into the range of a low
two-digit pixel number. Nevertheless, the degree of dispersion is
to be classified as marginal. However, the calculation of the aver-
aged relative orientation is based on the assumption that the
known relative orientations of the MHL2 correspond to the ideal
values. Consequently, the relative orientation of the MHL2 needs
to be verified initially for a more accurate evaluation of the calibra-
tion results. More generally, the calibration procedure also requires
that the calibration parameters be determined prior to each new
acquisition process, since the intrinsic camera parameters cannot
be constant for the sensors in addition to the extrinsic camera
parameters, as in the case of a total station, for example. On the
basis of our calibration method, we successfully implemented an
automated generation of 3D point clouds colored with RGB and
thermal image data. The results of the colored representations in
the point clouds achieve a level of detail that serves for the recog-
nition of energetic saving potentials and topological correlations as
well as for the derivation of semantic object data in particular for
bigger object. The extent to which these subsequent analysis pro-
cedures can be implemented in an automated process offers much
potential for future investigation. For objects with particularly
small cross-sections, such as valves or narrow pipes, the geometric
and color representations are imperfect and therefore prospec-
tively unusable for further analysis procedures. However, the over-
all results are dependent on the speed of movement and the
distance to the objects during the acquisition process. The best
T. Kinnen, C. Blut, C. Effkemann et al.
results can therefore be achieved when the acquisition process is
constantly slow and the objects are acquired from a distance of
0.5 to 1 m. Furthermore, the method of the sensor data fusion pre-
sented in this paper is predicated on the availability of all the data
of the acquisition process. The reason for this is that our system
setup enables access to the different data only after the acquisition
process has been completed. Another interesting approach for
future investigations is therefore the modification of the system
setup for enabling a sensor data fusion between the data of the
MHL2 and the FLIR Boson ADK thermal camera in real-time. In
addition, an extended investigation regarding the possibility to
capture and store numerical temperature values as well as their
integration into the generated thermal point cloud represents a
promising approach for future work.
Data availability
The data that has been used is confidential.
Declaration of Competing Interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
Acknowledgments
The authors gratefully acknowledge the financial support of the
German Federal Ministry for Economic Affairs and Climate Action
in the project ‘‘EnOB: EnergyTWIN - Energiediagnosestecker Digi-
taler Zwilling. Neue sensorgestützte und KI-basierte Methoden
für die digitale, BIM-basierte Inbetriebnahme von technischen
Anlagen in Hochbauwerken und deren energetische Systemopti-
mierung” (reference number 03EN1026).
References
[1] R. Becker, C. Blut, C. Emunds, J. Frisch, D. Heidermann, T. Kinnen, A. Lenz, M.
Möller, Pauen, Nicolas, T. Rettig, D. Schlütter, Wenthe, Matthias, J.
Blankenbach, G. Bleimann-Gather, J. Fütterer, J. Jungedeitering, C. van Treeck,
BIM-assisted, automated processes for commissioning in building services
engineering, in: Proceedings of the 39th International Symposium on
Automation and Robotics in Construction, International Association for
Automation and Robotics in Construction (IAARC), 2022.
[2] P. de Wilde, W. Tian, Identification of key factors for uncertainty in the
prediction of the thermal performance of an office building under climate
change, Build. Simul. 2 (2009) 157–174, https://doi.org/10.1007/s12273-009-
9116-1.
[3] R. Becker, E. Lublasser, J. Martens, R. Wollenberg, H. Zhang, S. Brell-Cokcan, J.
Blankenbach, Enabling BIM for Property Management of Existing Buildings
Based on Automated As-is Capturing, in: Proceedings of the 36th International
Symposium on Automation and Robotics in Construction (ISARC), Banff, AB,
Canada, International Association for Automation and Robotics in Construction
(IAARC), 2019.
[4] F. Bosché, M. Ahmed, Y. Turkan, C.T. Haas, R. Haas, The value of integrating
Scan-to-BIM and Scan-vs-BIM techniques for construction monitoring using
laser scanning and BIM: The case of cylindrical MEP components, Autom.
Constr. 49 (2015) 201–213.
14
View publication stats
Energy & Buildings 288 (2023) 113020
[5] S. Dong, T. Höllerer, Real-time re-textured geometry modeling using microsoft
hololens, in: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces
(VR), 2018, pp. 231–237.
[6] M. Bern, P. Plassmann, Mesh Generation, in: Handbook of Computational
Geometry, Elsevier, 2000, pp. 291–332.
[7] M. Garon, P.-O. Boulet, J.-P. Doiron, L. Beaulieu, J.-F. Lalonde, Real-time high
resolution 3D data on the HoloLens, in: 2016 IEEE International Symposium on
Mixed and Augmented Reality (ISMAR-Adjunct), 2016, pp. 189–191.
[8] M. Weinmann, S. Wursthorn, M. Weinmann, P. Hübner, Efficient 3D Mapping
and Modelling of Indoor Scenes with the Microsoft HoloLens: A Survey, PFG-
Journal of Photogrammetry, Rem. Sens. Geoinformat. Sci. 89 (2021) 319–333.
[9] P. Hübner, K. Clintworth, Q. Liu, M. Weinmann, S. Wursthorn, Evaluation of
HoloLens tracking and depth sensing for indoor mapping applications, Sensors
20 (2020) 1021.
[10] G. Loporcaro, L. Bellamy, P. McKenzie, H. Riley, Evaluation of Microsoft
HoloLens augmented reality technology as a construction checking tool
(2019).
[11] K. Khoshelham, H. Tran, D. Acharya, INDOOR MAPPING EYEWEAR:
GEOMETRIC EVALUATION OF SPATIAL MAPPING CAPABILITY OF HOLOLENS,
International Archives of the Photogrammetry, Remote Sensing & Spatial
Information Sciences (2019).
[12] I. Soares, R.B. Sousa, M. Petry, A.P. Moreira, Accuracy and repeatability tests on
HoloLens 2 and HTC Vive, Multimodal Technol. Interact. 5 (2021) 47.
[13] S. Kubo, Y. Iwasawa, M. Suzuki, Y. Matsuo, UVTON: UV Mapping to Consider
the 3D Structure of a Human in Image-Based Virtual Try-On Network, in:
Proceedings of the IEEE/CVF International Conference on Computer Vision
(ICCV) Workshops, 2019.
[14] S. Teruggi, E. Grilli, F. Fassi, F. Remondino, 3D Surveying, Semantic Enrichment
and Virtual Acces Of Large Cultural Heritage, in: ICOMOS/ISPRS International
Scientific Committee on Heritage Documentation (CIPA) 28th CIPA
Symposium ‘‘Great Learning & Digital Emotion”, 2021, pp. 155–162.
[15] S. Fiedler, P. Grümpel, G. Karl, S. Knoblach, 3D-Punktwolke mit
thermografischer Information, zfv-Zeitschrift für Geodäsie, Geoinformation
und Landmanagement (2019) 144.
[16] S. Vidas, P. Moghadam, HeatWave: A handheld 3D thermography system for
energy auditing, Energ. Buildings 66 (2013) 445–460.
[17] D. Borrmann, A. Nüchter, M. Ðakulović, I. Maurović, I. Petrović, D. Osmanković,
J. Velagić, A mobile robot based system for fully automated thermal 3D
mapping, Adv. Eng. Inf. 28 (4) (2014) 425–440.
[18] Microsoft, HoloLens 2: Technische Daten und Funktionen – Microsoft
HoloLens 2, 2022. https://www.microsoft.com/de-de/d/hololens-2/
91pnzzznzwcp?activetab=pivot:%C3%BCbersichttab (accessed 10 September
2022).
[19] Microsoft, HoloLens 2 hardware, 2022. https://learn.microsoft.com/en-us/
hololens/hololens2-hardware (accessed 15 September 2022).
[20] D. Ungureanu, F. Bogo, S. Galliani, P. Sama, X. Duan, C. Meekhof, J. Stühmer, T.J.
Cashman, B. Tekin, J.L. Schönberger, others, Hololens 2 research mode as a tool
for computer vision research, arXiv preprint arXiv:2008.11239 (2020).
[21] Microsoft, HoloLens-Forschungsmodus - Mixed Reality, 2022. https://learn.
microsoft.com/de-de/windows/mixed-reality/develop/advanced-concepts/
research-mode (accessed 23 September 2022).
[22] D.J. Yeong, G. Velasco-Hernandez, J. Barry, J. Walsh, Sensor and sensor fusion
technology in autonomous vehicles: A review, Sensors 21 (2021) 2140.
[23] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern
Anal. Machine Intell. 22 (2000) 1330–1334, https://doi.org/10.1109/
34.888718.
[24] G. Bradski, A. Kaehler, Learning OpenCV: Computer vision with the OpenCV
library, O’Reilly Media Inc, 2008.
[25] J. Blankenbach, G. Bleimann-Gather, energyTWIN - accessed 5 October 2022
Project Website (2022). https://energytwin.org/.
[26] Riegl, RIEGL VZ-400 Datasheet.
[27] J. Bureick, S. Vogel, I. Neumann, D. Diener, H. Alkhatib, Georeferenzierung von
unmanned aerial systems über laserscannermessun-gen und 3d-
gebäudemodelle, DVW-Schriftenreihe 96 (2019) 63–74.
[28] C. Ricolfe-Viala, A. Esparza, The Influence of Autofocus Lenses in the Camera
Calibration Process, IEEE Trans. Instrum. Meas. 70 (2021) 1–15, https://doi.org/
10.1109/TIM.2021.3055793.