Connected Augmented Assembly

Cloud based Augmented Reality applications in architecture

Ayoub Lharchi1 , Mette Ramsgaard Thomsen2 , Martin Tamke3

1,2,3
Centre of Information Technology and Architecture (CITA)
1,2,3
{alha|mette.thomsen|martin.tamke}@kadk.dk

Current design practices rely on a set of computational tools to simulate and

optimize the design in regards to questions concerning architecture, engineering,
and construction. However, little progress has been made in tools related to the
design and execution of a building assembly. This paper aims to present an
integrated procedure that targets the assembly of complex structures. Two
challenges are identified and addressed: first, the necessity of a connected design
environment where multiple stakeholders can communicate, modify, and give
feedback on the assembly sequence. Second, the instructions for the assembly of
structures to untrained users. The suggested method is based on the Assembly
Information Modeling framework, which provides a general approach to
generate assembly information from CAD data and utilizes AEC cloud platforms
as a base for communication and Augmented Reality devices as a Human
Machine Interface. Ultimately, both cases are combined to constitute Connected
Augmented Assembly, a bidirectional approach to assembly design, review, and
execution.

Keywords: assembly sequence, augmented reality, assisted assembly, cloud aec,

assembly information modeling

INTRODUCTION sign and planning in architecture. The computa-

In recent years, the development of digital toolsets
for design and production enabled new ways of mak-
ing and increased the degree of automation in con-
struction (Bock 2015). Computational methods allow
mass customization approaches (Tamke and Thom-
sen 2009) and facilitate the fabrication and construc-
tion of complex structures. Despite this rapid in-
crease in digitalization in different fields of Archi-
tecture, Engineering and Construction (AEC) sector,
there is not much research about the assembly de-
tional research is generally focused on the early de-
sign phases and later concerns such as how the actual
architecture is assembled are not considered. The
translation of a 3D geometry from a building model
into a building component that can be assembled is
a significant issue (Sheil et al. 2020). Concepts and
tools which support a vertical integration of design
for assembly and production are missing.
To bridge this gap, we aimed at developing an
assembly-aware design approach that supports the

D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES - Volume 1 - eCAADe
38 | 179
 users in the early design as well as in the assem-
bly phase. This is done by proposing communica-
tion methods that specifically use an adequate as-
sembly language. We combined Augmented Reality
(AR) and AEC cloud-based services to allow users an
easy, effective communication and information shar-
ing, and at the same time assist them during the as-
sembly.
BACKGROUND
Assembly Information Modeling
This work builds upon previous work about assem-
bly considerations in the architecture realm. The re-
search is developed as an extension to the existing
framework “Assembly Information Modeling (AIM)”
(Lharchi et al. 2019). It is a digital framework that
is designed for applications in building processes. It
aims to include all the necessary data to describe pre-
cisely an assembly sequence such as the geometry,
assembly directions, sequence, and any site condi-
tions or obstacles. At the same time, it provides a
base to create data bridges to the different author-
ing tools used by the stakeholders. The flexibility of
the model allows a large panel of applications, such
as the creation of centralized design models, robotic
fabrication, and connected assembly and others (fig-
ure 1). The model specifications are freely available
online and cover most of the everyday needs in a typ-
ical architectural assembly planning process but can
be extended as well if needed.
180 | eCAADe 38 - D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES -
Volume 1
The assembly data can be stored in one central
model, called Assembly Digital Model (ADM). This
model can then be used for different applications
such as collaborative assembly design, decision mak-
ing and robotic assembly. This paper presents and
evaluates the use of AIM as the base for an online col-
laboration platform and for support of human assem-
bly using Augmented Reality (AR).
Augmented Reality in AEC
In parallel with the development of design tools and
construction techniques, the AEC industry tries to ex-
pand the visualization methods by which complex
construction information can be shared and assimi-
lated. Progressively increasing interest has been ob-
served both in research and industry towards two
3D visualization methods: AR and Virtual Reality
(VR) (Duston 2008). While AR techniques superim-
pose the virtual objects or information within the
real world, VR immerses the user in a completely vir-
tual environment that allows freedom of the phys-
ical space constraints but also limits the possibility
of interaction with real-world elements drastically.
Several studies have investigated how these tech-
nologies can be utilized in construction context and
demonstrated the potential of both technologies in
architectural design and structural design (Quinn et
al. 2018; Boud et al. 1999), urban planning (Broll et
al. 2004), interactive modeling (Felbrich et al. 2018),
etc.
While AR and VR share many similarities, AR
presents the advantage that the user is still aware of Figure 1
the environment, and depending on the used head- Different use cases
set, a hands-free experience. Therefore VR is more of an Assembly
adapted for the design phase and AR generally for Digital Model
on-site applications.
Research on the use of AR in assembly tasks has
focused on the training for assembly tasks for me-
chanical applications (Boud et al. 1999), and only lit-
tle on assembly within an architectural context. Re-
searchers from Fologram (Newnham et al. 2018) used
AR and developed a platform to create holographic
instructions to guide the users for the fabrication and
 assembly of metal pipes (Figure 2). Others consid-
ered a combination of AR and robotic fabrication (Ky-
janeka et al. 2019) in a setup where the user is de-
signing interactively, and the robot is responsible for
placing the pieces in the correct position. The parts
are then fixed by the user (figure 3).
Figure 2
Metal pipe
assembly using an
AR headset
(Newnham et al.
2018)
Figure 3
Human-Robot
collaboration with
AR (ICD University
of Stuttgart, 2018)
Although these projects present concrete case stud-
ies where AR has been leveraged to enhance the
assembly experience, the actual assembly instruc-
tions were generated post-design and require labo-
rious manual work. In most cases, the assembly in-
structions are limited to displaying the final position
within the structure. It is up to the user to figure
out on-the-fly how to achieve the intended assembly
step. This does not account for potential obstacles or
spatial maneuverability of the element (for example
weight or size) except when a robotic arm is used for
the assembly. However, the use of a robot is not al-
ways possible depending on the scale and resources.
Furthermore, larger projects require the use of in-
dustry standard software that differ from one stake-
D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES - Volume 1 - eCAADe
holder to another and are not necessarily adapted
to the AR design methods. Moreover, the person in
charge of designing the general assembly strategy,
the specific details and the worker doing the on-site
assembly is rarely the same person. The need for
a common platform that serves as an assembly au-
thoring tool, communication, and knowledge trans-
fer arises.
AEC Cloud based Platforms
Building Information Modeling (BIM) is a semanti-
cally rich and object-based digital representation of
a building, that facilitates the exchange and interop-
erability of building specifications between the archi-
tect, engineer and the rest of the stakeholders (East-
man et al. 2011). To extend these exchange capa-
bilities, numerous BIM software vendors introduced
cloud-based platforms to enable developers to cre-
ate bespoke applications for specific needs. Most of
these platforms are capable of translating native BIM
or IFC models into relational databases (Yan 2017)
and store them in the cloud for further analysis or
visualization: Depending on the platform, it is pos-
sible to query the data directly from the BIM model,
visualize the whole model or specific parts, perform
geometric operations and export/import the data in
different formats. Users can link the model to other
data management platforms to perform tasks such
as scheduling, project management, or quality assur-
ance. Table 1 gives a brief comparison between two
of the popular platforms and their capabilities.
One main advantage of the cloud is to lever-
age its computing and storage capabilities. Previ-
ous works demonstrated these usages, for example
it can be used to parallelize and to pre-process huge
amounts of sampling data (Zwierzycki et al. 2018)
or to store a large amount of data (Cristie and Joyce
2017) to keep track of changes in design documents.
While most of the platform allows an incremental
display of the elements, and connection to schedul-
ing tools such as BIM 360 [3], none of them provide a
native integration of 4D simulation that displays the
actual steps necessary for a successful assembly pro-
38 | 181
 Table 1
Comparison of two
AEC cloud
platforms

cedure. Based on the construction timeline, the ele-
ments just appear in the model and lack the intrinsic
information about the assembly.
METHODOLOGY
In order to test and evaluate our theoretical concepts,
the methodology used in this research consists of de-
veloping a set of related tools that were tested with
two study cases demonstrating some complex archi-
tectural features. In both cases, the models were con-
stituted of a small number of parts and were too vi-
sually similar and have ambiguous order, therefore it
requires excessive documentation to assemble.
Cloud Based Assembly Designer
The collaborative design for assembly is a platform
for communication, viewing and discussion of as-
sembly data. A web-based platform uses the ADM
and makes it available to the different stakeholders
with a powerful built-in 3D viewer. The requirement
was to create an integrated platform that would run
on most modern web browsers without the need of
any additional plug-ins. This would include mobile
devices such as smartphones and tablets for eventual
on-site usage. The presented platform utilizes the
Autodesk Forge cloud services. We used the Forge
Viewer API and Model Derivative API. The application
backend was written in ASP.NET using the .NET Core
framework; meanwhile, the front-end was mainly in
HTML, CSS and JavaScript.
One user can upload an ADM file that is pro-
cessed by the Autodesk Forge cloud, translated into
SVF format. Afterwards, the assembly information is
injected into the 3D model. The model is then made
available for all the users of the platform for editing.
It is possible to change the global order of the part
or within a component, to suggest alternative assem-
bly sequence, change fasteners types and add meta-
data (comments or notes) directly in the model (fig-
ure 4). Alternatively, the model is also loaded in the
3D viewer, with direct access to the whole model and
separated elements, advanced layers management
for large projects and assembly animation playback
(figure 5).
Figure 4
Web based
interface to view
the assembly model
structure
Figure 5
The advanced 3D
viewer included in
the Assembly
Designer

182 | eCAADe 38 - D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES -
Volume 1
 Augmented Assembly tions are instantly displayed (figure 7).
The Augmented Assembly (AA) aims to utilize the
Figure 6 information embedded in the ADM, and the tech-
A user experiencing nologies provided by AR devices to guide the users
augmented through an assembly process. The goal is to enable
assembly inexperienced users to assemble complex structures
without previous training (figure 6). AR devices are
particularly adapted for this purpose as they allow an
overlay of digital data onto the user’s view and align
the relevant information with the objects of interest
(Zollman et al. 2014). Moreover, some models are
equipped with microphones that permit communi-
cation features.
In this investigation, the AR setup was based on
the head-mounted device Microsoft HoloLens 1. The
Figure 7 device embeds autonomous self-tracking features,
The user view from which allows a correlation between the physical and
inside the AR device simulation world coordinate systems. Therefore, it is
possible to track the user in the space and display cor-
responding assembly information overlaid on the en-
vironment.
The software running the augmented assem-
bly is built using both C# and PureBasic following a
server/client model:
• The server is running on a computer and is a plu-
gin of the CAD software Rhinoceros 3D [1].
• The client runs on the HoloLens and is a Univer-
In all the usage modes, the assembly sequence is dis-
sal Windows Platform (UWP) application. It was
played to the user in sequential order. The assem-
developed using the Unity 3D [2] game engine.
bly is animated in a way that the user can distinguish
The AA presents three distinct usage modes: clearly the orientation of the element as well as the
necessary spatial manipulations to put it in the cor-
1. Interactive: The application is connected to the rect place.
server over Wi-Fi using the User Datagram Proto- The Human Machine Interface (HMI) to control
col (UDP). It receives a continuous data stream the AR device was designed to be simple and inte-
and updates the geometry, position and scale grate two of the Hololens input systems:
accordingly. In this scenario, the server converts
geometry data to mesh if necessary and serialize 1. Gaze and Gestures to select, manipulate the dif-
it into raw data before sending it. The applica- ferent elements, and to extract information from
tion is then converting it back to mesh and dis- the ADM file and display them directly on the
playing the geometry in the user’s field of view. augmented view.
2. Collaborative: The application is linked to the 2. Voice commands to assure a hands-free experi-
Autodesk Forge cloud, and the ADM is synced ence to control the sequence display (e.g. next,
and constantly updated. Changes and annota- previous, and pause).

D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES - Volume 1 - eCAADe
38 | 183
 Study cases
To test the suggested method, we developed a
demonstration construction system that was then
used in two similar, yet different scale cases . The join-
t/node allowed the intersection of three different lin-
ear elements at the same time (figure 8). It requires a
specific order to be assembled, and the orientation of
the parts is ambiguous on purpose. The goal was to
have a complex structure that would require exces-
sive documentation to assemble.
We combined the cloud-based assembly designer
with the augmented assembly interface in an integra-
tive approach (figure 9). We conducted two exper-
iments where untrained public visitors were invited
to assemble the structure. Initially only with the help
of 2D plans and final structure model, and afterwards
equipped with an AR headset directly linked to the
ADM on the cloud where significant ease of assem-
bly was observed.
RESULTS AND DISCUSSION
State of the art AR, cloud-based collaborative tech-
niques, and AIM were used in order to develop a ro-
bust approach to augmented assembly. The use of
Autodesk Forge cloud-based tools enabled the de-
velopment of web-based platforms that assist the
users with both processing of their raw data and ad-
vanced visualization. Notably, following the upload
of the ADM web, they can access it within the em-
bedded three-dimensional viewer in the browser and
184 | eCAADe 38 - D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES -
Volume 1
interact, comment, or adjust it. Direct export to the
AR headset is also possible for a seamless assisted as-
sembly.
While the AR part of the system was working
without issues during the development phase and
the lab conditions test, it was not the case dur-
ing tests in large, populated environments. The
HoloLens 1 limitations were quickly felt as the on-
board positioning system tended to lose its calibra-
tion. One possible solution to enhance the position-
ing reliability would be the integration of an exter-
Figure 8
nal localization system in parallel to correct any drifts
Study Case 2:
and sync the world coordinate system (Kyjaneka et
Tri-Direction node
al. 2019). Besides, the vocal commands did not work
system.
as expected because of the background noise. For-
tunately, this can be easily addressed using a sep-
arate microphone instead of the onboard one cur-
rently used.
Future work should focus on the implementation
of multi-user approaches where more than one user
can wear an AR headset in a collaborative setup.
CONCLUSION
In this research, We presented an integrative ap-
proach for the assembly of complex structures. We
emphasized the need for a connected design envi-
ronment where multiple stakeholders can communi-
cate on the assembly sequence and transfer intricate
knowledge among each other as well as to inexperi-
enced users. We showed potential usages of the As-
sembly Information Modeling framework leveraging
AR and AEC cloud solutions. The development of AR
technologies will arguably eliminate current techni-
cal limitations.
ACKNOWLEDGEMENTS
This project was undertaken as part of the Innochain
Early Training Network. This project has received
funding from the European Union’s Horizon 2020 re-
search and innovation programme under the Marie
Sklodowska-Curie Grant Agreement No. 642877. We
would also like to express our gratitude to the in-
dustrial partners of this research; Blumer Lehmann
Figure 9
Overview of the
connected
augmented
assembly approach.
and Design-To-Production, as well to Autodesk, espe-
cially the Forge development team.
Icons in the figures made by Freepik from
www.flaticon.com
REFERENCES
Amann, J, Preidel, C, Tauscher, E and Borrmann, A 2018,
’BIM Programming’, in Borrmann, A, König, M, Koch,
C and Beetz, J (eds) 2018, Building Information Mod-
eling: Technology Foundations and Industry Practice,
Springer International Publishing, Cham, pp. 217-
231
Bock, T 2015, ’The future of construction automation:
Technological disruption and the upcoming ubiq-
uity of robotics’, Automation in Construction, 59, pp.
113-121
Boud, AC, Haniff, DJ, Baber, C and Steiner, SJ 1999 ’Virtual
reality and augmented reality as a training tool for
assembly tasks’, 1999 IEEE International Conference
on Information Visualization (Cat. No. PR00210), pp.
32-36
Broll, Wolfgang, Lindt, Irma, Ohlenburg, Jan, Wittkäm-
per, Michael, Yuan, Chunrong, Novotny, Thomas,
Schieck, Ava Fatah, Mottram, Chiron and Stroth-
mann, Andreas 2004, ’A collaborative augmented
environment for architectural design and urban
planning’, Journal of Virtual Reality and Broadcasting,
1(1)
Cristie, V and Joyce, SC 2018 ’GHShot: 3D Design Ver-
sioning for Learning and Collaboration in the Web’,
D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES - Volume 1 - eCAADe
Extended Abstracts of the 2018 CHI Conference on Hu-
man Factors in Computing Systems, pp. 1-6
Dunston, PS 2008, ’Identification of application areas for
Augmented Reality in industrial construction based
on technology suitability’, Automation in Construc-
tion, 17(7), pp. 882-894
Eastman, CM, Eastman, C, Teicholz, P, Sacks, R and Liston,
K 2011, BIM handbook: A guide to building informa-
tion modeling for owners, managers, designers, engi-
neers and contractors, John Wiley \& Sons
Felbrich, B, Jahn, G, Newnham, C and Menges, A 2018
’Self-organizing maps for intuitive gesture-based
geometric modelling in augmented reality’, 2018
IEEE International Conference on Artificial Intelligence
and Virtual Reality (AIVR), pp. 61-67
Kyjaneka, O, Al Bahar, B, Vaseya, L, Wannemacherb, B
and Mengesa, A 2019 ’Implementation of an Aug-
mented Reality AR workflow for Human Robot Col-
laboration in Timber Prefabrication’, ISARC. Proceed-
ings of the International Symposium on Automation
and Robotics in Construction, pp. 1223-1230
Lharchi, A, Thomsen, MR and Tamke, M 2019, ’To-
wards Assembly Information Modeling (AIM)’, 2019
SimAUD, p. 51
Newnham, C, van den Berg, N and Beanland, M 2018,
’Making in Mixed Reality’, CUMINCAD
Quinn, G, Geleazzi, A, Schneider, F and Gengnagel, C
2018 ’StructVR Virtual Reality Structures’, Proceed-
ings of IASS Annual Symposia
Sheil, R, Thomsen, M, Tamke, M and Hanna, S (eds)
2020, Design Transactions: Rethinking Information for
a New Material Age, UCL Press
38 | 185
 Tamke, M and Thomsen, MR 2009 ’Digital wood craft’,
Joining Languages, Cultures and Visions, pp. 673-683
Yan, W 2017, ’WP-BIM: Web-based Parametric BIM To-
wards Online Collaborative Design and Optimiza-
tion’, SharingofComputableKnowledge!, p. 527
Zollmann, S, Hoppe, C, Kluckner, S, Poglitsch, C, Bischof,
H and Reitmayr, G 2014, ’Augmented reality for con-
struction site monitoring and documentation’, Pro-
ceedings of the IEEE, 102(2), pp. 137-154
Zwierzycki, M, Nicholas, P and Ramsgaard Thomsen, M
2018, ’Localised and Learnt Applications of Machine
Learning for Robotic Incremental Sheet Forming’,
in De Rycke, K, Gengnagel, C, Baverel, O, Burry, J,
Mueller, C, Nguyen, MM, Rahm, P and Thomsen, MR
(eds) 2018, Humanizing Digital Reality: Design Mod-
elling Symposium Paris 2017, Springer, Singapore,
pp. 373-382
[1] www.rhino3d.com/
[2] https://unity.com/
[3] https://www.autodesk.com/bim-360/

186 | eCAADe 38 - D1.T3.S1. DIGITAL PERCEPTION OF SPACE – CYBER-PHYSICAL SYSTEMS (VR, AR) – DESIGN STRATEGIES -
Volume 1