Evolutionary Intelligence

https://doi.org/10.1007/s12065-019-00314-6

SPECIAL ISSUE

The application of augmented reality and unity 3D in interaction

with intangible cultural heritage
Weibo Huang1 · Handun Xiang2 · Shaohui Li3

Received: 1 August 2019 / Revised: 10 October 2019 / Accepted: 24 October 2019

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Abstract
The current predicament of intangible cultural heritage is its vulnerability and rarity, and an expression that is difficult to
construct in the space-time dimension. It is difficult for visitors to really understand and touch these local intangible cultural
heritages. For this reason, the need to develop new methodologies for cultural heritage learning and art appreciation is always
present, and this is increasingly supported by digital technologies. The research in this paper is to apply the self-issuance
method to the folk intangible cultural heritage center. It is different from the traditional AR method of presenting crafts
directly through HMD. Instead, the use of low-cost technology, Augmented Reality technology and unity 3D technology
provide a new way of interaction. By using HMD, smartphone, Leap motion and software created in UNITY 3D, the appli-
cation of the augmented reality program is achieved. It allows users to interact with the intangible cultural heritage easily
and fully it in a virtual physical environment. Based on the principle of human natural interaction, it can be achieved by
structured light target image scanning and the design and experiment of photographic measurement. Then a local intangible
cultural heritage model database can be established to provide virtual prototype files. The results of the experimental show
that the data transmission is effective and the three-dimensional interaction between people and exhibits can be realized.
The research results can be applied to the interactive mode practice of the intangible cultural heritage center, and provide an
effective interactive mode for the intangible cultural heritage in the form of augmented reality interaction. In this way, visi-
tors can have a deep impression on the intangible cultural heritage and the culture can be spreaded widely all over the world.

Keywords  Augmented reality · Unity 3D · Virtual reality · Interaction · Intangible cultural heritage

1 Introduction
* Weibo Huang In the era of rapid development of science and technology,
172564946@qq.com
AR technology is an emerging digital technology with great
Handun Xiang potential of development and application. From industrial
893652867@qq.com
design to smart healthcare, from entertainment and travel
Shaohui Li to education, the application of AR technology is becom-
shevyyy@163.com
ing more and more extensive. Intangible cultural heritage
1
Network and Information Center, Experimental Teaching is not only an important part of national culture but also an
Center, Guangdong University of Foreign Studies, important manifestation of national spirit and national con-
Xiaoguwei, Panyu District, Guangzhou Higher Education notation. However, due to the evolution of social processes,
Mega Center, Guangzhou 510006, Guangdong Province,
China the survival and development environment of intangible cul-
2 tural heritage has become increasingly severe. Many excel-
School of Art, Guangdong University of Foreign Studies,
Xiaoguwei, Panyu District, Guangzhou Higher Education lent intangible cultural heritage resources are even facing the
Mega Center, Guangzhou 510006, Guangdong Province, dilemma of disappearing. Therefore, digital protection and
China application development work has become a critical way to
3
School of English Education, Guangdong University save intangible cultural heritage. The main shortcomings
of Foreign Studies, Xiaoguwei, Panyu District, Guangzhou of the folk intangible cultural heritage museum are that the
Higher Education Mega Center, Guangzhou 510006, close interaction mode of visitors to the intangible cultural
Guangdong Province, China

13
Vol.:(0123456789)

heritage is limited and the interactive mode is rigid [1]. It
is difficult for visitors to approach truly and understand the
intangible cultural heritage due to its fragility and rarity, or
its expression form which is difficult to construct in space
and time dimension [2]. Since the invention of digital com-
puters, more and more digital systems have been used. In
today’s world, the number of people accessing and under-
standing relevant information through online browsing has
increased dramatically [3]. Based on the above data, we can
see that smart terminals have become a huge scale founda-
tion to build the 3D augmented reality technology in the
intangible cultural heritage center [4]. Using smart phones to
connect the interactive mode of the folk intangible cultural
heritage center can not only attract people to experience it
and interact with the intangible cultural heritage directly,
but also can enhance the interest of the intangible cultural
heritage. In recent years, augmented reality (AR) technology
has been successfully applied in museums, cultural relics,
scientific exhibitions and tourist area. Some of these sites
use the AR application as a part of the exhibition [5], while
others use the technology as a guidance system [6].
Augmented reality (AR) technology bridges the gap
between the real world and the virtual world, taking the
3D intangible cultural heritage in the real interaction [7].
AR technology refers to the inclusion of virtual elements
in views of actual physical environments to create a mixed
reality in real time. It supplements and enhances the per-
ceptions humans gain through their senses in the real world
[8]. In this way, we can enhance the experience of the peo-
ple by interacting with intangible cultural heritage [9]. For
instance, in AR learning environments, motivational factors
related to attention and satisfaction are rated more highly
than for slide-based environments [10]. As far as we know,
so far, the research community has not tried to combine AR,
unity 3d, and human–computer interaction in the museum
environment. The main difference from the traditional AR
application in the cultural center is that the former focuses
on the field of visual performance, and our focus is on
human–computer interaction. People’s hands are used as
interesting 3D interactive bridge tools. In the process of
trying, we did not fully release the function of the handle,
but we were able to track the specific motions of the palm
and the fingers [11]. It means that we can grasp the strength
of the palms and grips, and make it more natural and agile.
2 Unity 3d technology
Unity 3D is a multi-platform game engine with built-in IDE
technology system. The software is mainly used in the pro-
duction of game and augmented reality technology, and the
interactive environment using graphic images is the primary
method. Unity 3D’s compiler can run on today’s mainstream
13
Evolutionary Intelligence
operating systems such as Window, MacOSX, and Android.
You can also use the plugin for Unity web player to pub-
lish web applications, and support web browsing on mobile
phones and computers [12].
Unity 3D supports mainstream 3D modeling softwares
such as 3Ds max, Maya, Blender and more. Its shader uses
the easy-to-use ShaderLab. The physical terrain editor is a
powerful feature of unity 3D and supports the creation of a
variety of complex terrain and planes. Unity 3D’s built-in
multiplayer engine will be a major multiplayer interaction
in later experiments. Some objects in the translation cul-
ture hall will simulate the physical feature attributes in real
space. Therefore, in the plug-in with physical effects in unity
3D, you can use a computer program to simulate Newton’s
mechanics, and set the quality, speed, friction air resistance
and other factors into regular independent variables and
dependent variables. Unity also has a sophisticated physi-
cal lighting system engine that can better simulate realistic
lighting effects and shadow texture. The built-in particle
simulation system can create special effects such as wind,
rain, explosion, dust, etc., so that the intangible cultural
heritage can be enhanced in an ancient physical condition
[13]. This provides the user with more realistic and realistic
experience. The setting of the resource server in Unity can
be multi-person collaborative development, which provides
a good solution for subsequent improvement and update.
3 The application requirements
of augmented reality and unity 3d
technology in folk intangible cultural
heritage pavilions
The folk intangible cultural heritage center is a permanent
existence in the local area. It reasonably organized the local
intangible cultural heritage, and established a type of non-
profit organization, a physical space, for acquisition, inherit-
ance, protection and exchange. There is a vague boundary
between the intangible cultural heritage center and the vir-
tual intangible cultural heritage center. Our definition of the
virtual intangible cultural heritage center is making good
use of digital to map the physical intangible cultural herit-
age. The user’s hands act as a medium to connect the virtual
world and the reality.
Natural interaction is a type of human–computer interac-
tion that uses the user’s natural movement behavior and digi-
tal programs to communicate. For an interactive interface,
we don’t need to display every action button just like in a
game. It is a recessive, uninterrupted, or natural interaction,
which makes it invisible through the user’s natural interac-
tion. In an ideal interaction, people interact using everyday
behaviors. We can realize it by a variety of contact sur-
faces or motion sensors, combining real intangible cultural
Evolutionary Intelligence

heritage with digital virtual intangible cultural heritage. In system gesture collection is similar. Add a LEAP sensor to
the process of constructing the 3D entity, users can use the the system to identify each bone feature of the gesture, and
position of fingers to calculate the digital information of the then map to the server database to achieve human–computer
object such as the color and curvature of the virtual incom- interaction.
plete [14]. For this kind of interaction, it can be a natural
human–computer one, because the user uses it just like using
tools naturally in reality. 4 Design of the virtual program system
The focus of the latest cultural library interaction research
is that the research is based on Symbian and Android smart- What we are making is a realistic virtual program where
phones. It uses a smartphone camera to capture sample users can see virtual copies superimposed on them. Then, by
video. Users can see live video on the screen. Then the sam- hand interactions, the user is completely immersed in virtual
ple video is further processed by an image processing task to composition space. In an immersive environment, they are
detect predefined markers. If a marker is detected, AR reg- able to see the world around reality. In terms of enhancing
istration is performed between the real world and the three- productivity, we use the low-cost HMD, the Google card-
dimensional world. AR registration is a method of aligning board, the programmable graphics scanner and the social
virtual objects in a real environment properly, which enables server to build it. It can also be synced to the same mobile
users to perceive virtual objects in the correct position and device, which can be used to track the movement of objects.
orientation. Then the generation of 3D computer graphics AR and virtual plugin are parts of this application even if
in real world video is completed. Users interact and control the object is not in the scope of the reference line of sight.
virtual objects through touch in touch enable phone or but- Generally, it does not need to display the full scene. How-
ton in non-touch phone. The focus of this technology is the ever, it needs to analyze the data of numerous sensors for
use of exchange patterns of specialized tool for interactive the implementation of tracking technology and 3D registra-
behaviors [15]. tion. As a result, there is a general need to run on specific
In augmented reality, the interaction mode of platforms and provide support for key technologies. The
human–machine is divided into spatial point interaction workflow is as follows:
mode, special tool exchange mode and command interac- The workflow of the work is roughly as shown in Fig. 1.
tion mode [16]. The way in which special tools interact is to
use some special tools that are easy to identify with simple 1. Get real-time scene information.
features as interactive devices. In the process of interaction, 2. Track and identify the information acquired by the sen-
these tools or actions are identified as commands for input- sor.
ting information. Most of the interactions in this folk intan- 3. Generate the virtual object.
gible cultural heritage center adopt a third type of command
interaction: initially by setting some actions and states as
running commands, such as contact, sliding, and pulling.
Two-dimensional spatial locations can also be used as input
to commands for such commands, such as human body lan-
guage gestures. By taking the position of the human body
language action in space, the relevant predetermined output
commands can be triggered.
Based on natural interaction, Shih proposed a user inter-
face design study to increase the willingness of the elderly
to use it. In the proposed content, the Kinect sensor is used
to retrieve the depth information of the elderly in motion,
so that the user interface of the operating system can be
used intuitively. On one hand, under the framework of the
system, use morphology to identify the characteristics of
the hand from the depth information obtained from the sen-
sor. The gesture is used to identify the user’s operational
behavior to achieve an interactive action, and collision
detection is used to check the effectiveness of the opera-
tion. On the other hand, by interpreting the structural model
(ISM), various design elements of the interactive interface
can be decomposed and implemented [17]. The principle of Fig. 1  The augmented reality system workflow

13

4. Imaging at the display end.
Complete Augmented Reality techniques are usually com-
posed of hardware systems and software systems. Hardware
systems include displays (ability to calculate continuously),
processors (abilities to display and collect), sensors (abil-
ity to sense the state of the target), interactions (ability of
human–computer interaction), and network modules (ability
of serial communication). The software system includes a
target recognition module, a registration and location infor-
mation determination module of the tracking technology, a
three-dimensional image imaging rendering module, and an
intelligent display module.
It is a general introduction to the traditional augmented
reality technology, and detailed innovative interaction meth-
ods will be described in system design.
4.1 System experiment for 3D model acquisition
In the experiment, a three-dimensional model of a system-
atic experiment is acquired and organized, which usually
using the object image scanning and photogrammetry of
structured light. For the scanning of structured light, laser
triangulation is used, which is a kind of line scanning meth-
ods. By projecting the light source onto the surface of the
object and then scanning on one side one-dimensionally,
we can obtain the position depth of the object. Different
structured light scanning algorithms require the SLM to be
able to produce one of the secondary or grayscale graphics,
or to be able to produce both graphics. The use of high-
contrast graphics helps maximize accuracy and durability
of the operation while handling changing object reflectance
and ambient light conditions. Therefore, the adverse effects
of the surrounding environment are alleviated during the
measurement. The camera images the object during each
line scan, as is shown in Fig. 2.
Then, through one-dimensional scanning on one side, the
depth information of the entire object is obtained.
Fig. 2  The front view scanning process and the top view scanning
process
13
Evolutionary Intelligence
When a normal digital camera performs photogrammetry,
the requirements for data collection are completed by pho-
tographing. The more computational methods used are the
Direct Linear Transformation Method and the Time-base-
line Parallax Method. Under certain conditions, the Light
Beam Method is used. The Time-baseline Parallax Method
is mainly used to measure the motion or deformation of an
object and to measure the two-dimensional plane deforma-
tion of an object, which has certain limitations. Therefore,
the Direct Linear Transformation Method is used here.
It has the following advantages in practical use.
1. The optical axes of the cameras do not need to intersect.
2. The camera position can be placed arbitrarily without
measurement.
3. Calculate the three-dimensional coordinates of the point
by taking two pictures of the same object from different
angles.
4. The accuracy of the three-dimensional coordinates of
the point can be greatly increased by using a plurality of
photographs of different angles of the same object that
ensure the coincidence ratio.
In the use of photogrammetry, since the imaging is per-
formed under an ideal state, the basic shape of the model can
be in a better state. When processing the image scanned by
computer, in the coordinate system of o0-uv space, although
the coordinate origin can represent any position, to better
standardize the operation process, try to use CCD array to
perform.
4.2 The construction of object image projection
To make the coordinate origin coincide with the frame label,
you need to locate the X axis and the Y axis in the frame
label first, as is shown in Fig. 3.
At this time, the origin of the coordinate and the XY
value of the positioning is the internal variance, which is
used to restore the photographic beam. The external variance
Fig. 3  The coordinate system of coordinate instrument and the image
plane coordinate system
Evolutionary Intelligence
of the three linear elements and the three angular elements
determines the position and orientation of the center of the
photograph.
In the photogrammetric system, to represent the point
position of each pixel in the picture sequence at the plane
position and the point position of the real position of the
space represented by the pixel, five coordinate systems
including the intermediate conversion coordinate system are
established. They are five kinds of coordinate systems such
as image plane coordinate system, image space coordinate
system, image space auxiliary coordinate system, photo-
grammetric coordinate system and object space coordinate
system. The theoretical basis of photogrammetry is the pre-
condition for photogrammetry of ordinary digital cameras.
This condition is based on the relative position of pixels in
the sequence of pictures and the topological relationship in
the real position of space. This topological relationship is a
function of the relative positional relationship between the
coordinate spaces and the focal length of the camera. In the
solution picture sequence, the equation of the real position
coordinate information of the pixel in space is to solve the
object space coordinates by using the known camera param-
eters and the image space coordinates.
The final step is to use the collinear equation to construct
the relationship between the image point and the projection.
⎡X⎤ ⎡ XA − XS ⎤
⎢ Y ⎥ = 1 ⎢ YA − YS ⎥ (1)
⎢ ⎥ 𝜆⎢ ⎥ (
⎣Z⎦ ⎣ ZA − ZS ⎦
⎡X⎤ ⎡ x ⎤ ⎛ a1 a2 a3 ⎞⎡ x ⎤ 4.3 Processing of the intangible cultural heritage
⎢ Y ⎥ = R⎢ y ⎥ = ⎜ b1 b2 b3 ⎟⎢ y ⎥ (2)
⎢ ⎥ ⎢ ⎥ ⎜ ⎟⎢ ⎥ model
⎣Z⎦ ⎣ −f ⎦ ⎝ c1 c2 c3 ⎠⎣ −f ⎦
The coordinate system is extracted from the image
sequence and the coordinate system is transformed, and
finally the real coordinate position of the object space is
calculated. The algorithm is the collinear equation algo-
rithm. Its essence is that under the absolute condition, the
object point, the shooting point and the image point are
on the same line. The biggest feature of the equation algo-
rithm is that it is more rigorous. It is necessary to calibrate
the camera’s parameters when calculating the real coordi-
nate position of the object point. In the imaging principle
of ordinary digital cameras, there are several important
parameters, such as light center, image point, focal length,
pixels, and object points. Let the lens optical center S have
its object space coordinate system coordinate (Xs, Ys, Zs);
any object space point M whose object space coordinate
is (X, Y, Z), its image on the image m The image space
coordinates and image space auxiliary coordinates are (x,
y, -f) and (Xm, Ym, Zm). According to the assumption that
the three points of S, m, and M are collinear, the following
formula can be obtained.
A collinear conditional equation can be obtained by
combining the above formula:
( ) ( ) ( )
a1 XA − XS + b1 YA − YS + c1 ZA − ZS
x = −f ( ) ( ) ( ) (3)
a3 XA − XS + b3 YA − YS + c3 ZA − ZS
( ) ( ) ( )
a2 XA − XS + b2 YA − YS + c2 ZA − ZS
y = −f ( ) ( ) ( ) (4)
a3 XA − XS + b3 YA − YS + c3 ZA − ZS
In fact, the digital camera of this calculation method
does not exist. When imaging is carried out in the actual
digital camera, the image plane is not strictly zero. Instead,
there is a certain distortion error, which is recorded as (x0,
y0) [5]. Other factors such as the distortion of the objec-
tive lens distortion of the lens of the digital camera, cause
the image points to deviate from the theoretical position
(x, y) on the image plane (Δx, Δy). So the final collinear
conditional equation for the actual image point is:
( ) ( ) ( )
a1 XA − XS + b1 YA − YS + c1 ZA − ZS
x − x0+ Δx = −f ( ) ( ) ( )
a3 XA − XS + b3 YA − YS + c3 ZA − ZS
(5)
) ( ) ( )
a2 XA − XS + b2 YA − YS + c2 ZA − ZS
y − y0+ Δy = −f ( ) ( ) ( )
a3 XA − XS + b3 YA − YS + c3 ZA − ZS
(6)
After passing the photogrammetry and laser triangulation,
these models are weakened to minimize vertex processing.
Such an operation will change the original shape of the
object, and the intangible cultural heritage models used
in vertex weakening store their color information in the
vertices. However, using the open source 3D modeling
software Blender can also reduce surface-related textures,
as shown in Fig. 4.
The key element of spatial interaction is how to project
a virtual object onto the display without errors. It is nec-
essary to restore the already collected image to a realistic
three-dimensional space, and determine the coordinate
point at which the virtual object is located in the space. In
most cases, markers are used to establish a spatial coordi-
nate system. For the innovation of human–computer inter-
action systems, traditional known tests cannot be used to
select the marking method. Therefore, after repeated set-
ting and experiment in the operation, it is found that the
13

Fig. 4  Vertex weakening of the 3D model after scanning
speed is the fastest when the critical point black and white
monochrome marking method attribute is reached.
Calculating an external image file with color information
and an initial 3D model normal map file is implemented.
Transfer to a low-resolution 3D model using a link and then
export the format supported by Unity 3D.
4.4 The hardware system
Current mobile devices are equipped with sensors, it is able
to act as a display module and processor module.
The display module is mainly used to provide the ability
of the intelligent terminal to display information. Generally,
there are cameras and displays. This is an important means
of realizing Augmented Reality technology.
The processor module is used to provide the computing
power of the technology for data and is a decisive factor
in the performance of the Internet terminal. This module
mainly contains the GPU and CPU. On mobile phones, it
has become common to join multi-core processors to ensure
high-speed computing requirements. At the same time,
under the conditions of rich applications, it can meet the
needs of handling more complex image tasks. Therefore, the
graphics display function of the GPU has been introduced
on most hardware, and the performance of the terminal has
been greatly improved.
4.5 The software system
The main mode of operation of this system is to establish
related data and connections through UNITY 3D software.
Other advantages of using unity 3D to implement this sys-
tem are the extensive user community and the writing of
good supporting documentation. To enable virtual objects
13
Evolutionary Intelligence
to be registered to the real space, Qualcomm Vuforia, 4, a
versatile AR development software that allows for various
image recognition methods. The image target can be any
image that can provide a variety of features. An important
parameter set in the Vuforia plugin is Extended Tracking,
which allows tracking camera movement even if the target
image is not in the camera view. AR plugins and virtual
components are all parts of the client application.
Pass the generated control information to the script. It
does not run in a loop where the task is completed, but
runs like a normal script. After the function is completed,
the controlled permissions will be re-controlled by Unity
3D. The function here is called an Event Function. Then in
Unity 3D, you need to replace the camera deletion that was
initialized in the Hierarchy View panel with the new AR
Camera. Import the created non-legacy model and adjust
the perspective of the model in the view panel. To ensure
that the virtual camera is combined with the real scene
and the virtual scene under the condition of using the AR
camera, it is necessary to ensure the state information in
the three-dimensional space is correct, and then edit the
multi-level development system with multiple develop-
ment environments comprehensively.
Mark the executed instance properties visually and
preview the property editor dynamically in advance. The
image engine uses Direct3D, OpenGL, and its own api.
The created resource is automatically imported, no action
is required, and new items are automatically updated as
the resource changes.
A total of four modules are involved in unity 3D. The
details of each section are described below.
1. Software system target recognition module
  Target recognition refers to finding and locating
objects in a picture in related scenes. Mark objects with
markers for accurate identification in complex system
environments. The object can be extracted and identi-
fied by taking the object’s color, physics, shape, etc.
to deeply lock the object. Finally, the virtual image is
mapped into the real scene.
2. Tracking technology registration and location informa-
tion determination module
  The algorithm for tracking technology registration is
to merge virtual objects onto real and accurate spatial
objects. By tracking the position and dynamics of the
terminal in the real scene in real time, and based on the
information, the coordinate points corresponding to the
corresponding virtual object are calculated in real time.
This is used to ensure that the virtual object and the real
object in the picture can be well-connected, which is an
important part of augmented reality technology.
3. 3D graphics rendering module
Evolutionary Intelligence
  Through modeling and algorithm processing of 3D
objects, 3D objects are superimposed in a real space envi-
ronment, and virtual visual display is performed to help
users better understand the real environment. When the
position of the mobile device changes in space, the virtual
graphics can automatically adapt to the perspective size
position relationship in the scene under the three-dimen-
sional graphics module algorithm. Regardless of the user’s
observation at any angle, the European theorem can be
followed and the object is consistent in the real scene.
4. Intelligent display module
  The purpose of using the smart display module is to
connect the imaging of the virtual image through the
head mounted display. To be accurate, the relationship
between the world axis coordinate system, the camera
coordinate system, and the openGL coordinate system
must be known. This module is a direct visual bridge
between the user and the device, and is an important
factor in determining whether the user can actually expe-
rience the virtual environment.
5 Generation of virtual object interaction
Interaction module is an important factor influencing
the user experience. In traditional museum interactions,
smart devices typically employ touch screen interaction
and voice interaction. Interact with virtual intangible
cultural heritage other than WIMP (window, icon, menu,
pointer), however, this is not necessarily a natural way of
interacting. The behavior of the hand can be divided into
two categories: the interaction of the hand and the natural
interaction of the hand. You need to build a data reposi-
tory that uses system identification for manual interaction.
The natural interaction of the hand, on the contrary, does
not require a fixed database to identify. This interaction
allows users to exist in the form of numbers, and they
Fig. 5  Sensor placement state
of the LEAP motion and open
angle control of the hand during
motion
can experience the reality of nature in the process of vir-
tualization. This has brought about tremendous changes
in human–computer interaction. By interacting with com-
puter-generated digital information through different chan-
nels, users can immerse themselves directly in the scene,
thus enhancing the user experience.
To build a simple system yourself, we can choose the
Android SDK. Sensor performance conditions may be poor
during subsequent testing, but can support smooth per-
formance. The rebuilded system retains the use of mobile
devices as a form of HMD, but it does not affect this pro-
cess. The interaction can still be done by hands and the Leap
motion sensor is directly connected. While being analyzed, it
interacts with the virtual proxy server, and the server termi-
nal of Unity that uses the proxy is connected to the network.
By placing the chart on the table, you can enable virtual
tracking. Top: The Orthogonal View Scene Panel. Medium:
The Front View Orthogonal Panel. Bottom: The Front Cam-
era View Game Panel.
The process of implementing the mobile location and the
virtual interaction on the server is to connect the sensor of
the lEAP motion to the system server. The user’s hands’
above the level up, as shown in Fig. 5.
It analyzes the movement of the hand. The sensor provides
tracking data in real time during the skeletal movement of the
hand. There are two ways to identify the posture and manipu-
lation of the hand. It tracks the closing of the palm between
the two extremes of the palm (value 0) and the palm (value
1). When the value of the closed palm is greater than 0.75, as
in Fig. 5, it can be determined that it is a grab state. When the
position of the Intangible Cultural Heritage is close to the palm,
the position and space properties match the palm. Meanwhile,
the object position and rotation are synchronized on the server.
You can create a scene for the user to hold the virtual intangible
cultural heritage on the hand. Then proceed to the next step.
The second is to set a setting to the standard, in which
the hand is pinched. The intangible cultural heritage
13

Fig. 6  The Unity 3D game engine snapshot
objects will follow the hand to move to the corresponding
position in the process. We can estimate the position of the
press from the average midpoint of the distance between
the median of the thumb and the middle finger. The posi-
tion of the virtual invisible point and the palm track is
rotated and synchronized, and the origin of the imma-
terial culture is initially defined in the 3D model area.
Then all objects in this area can have an expression. What
is referred to is the operation that matches the physical
properties of the object. This method largely distinguishes
the choice of objects directly. In the custom space, non-
remaining 3D models do not appear at the same time. The
intangible cultural heritage objects without the selected
vertices will not appear in 3D format.
In the five additional models, as is shown in Fig. 6, each
shape has a door frame trigger. We can replace the active
models one by one by triggering the displayed model trig-
gers. The theory seems simple: just connect the LEAP
motion sensor directly to the phone to achieve a natural
interaction. In the system’s composition, it needs to be
equipped with: a gyroscope, an accelerometer, a mobile
device, HMD for the mobile phone holder, the Google
cardboard, and the leap motion sensor. The image will also
change during this process. If you want to return the image
used, move the current object closer to the object you want
to select. And it will be replaced by the original graphic.
6 Construction of mobile workstation server
In the final design of the system, a laptop was handled as
a mobile workstation server. Usually the server is only
used to do 3D interactive mode for tracking data and
13
Evolutionary Intelligence
synchronizing devices. The 3D related content is not pro-
vided, only the Carl coordinates and some related vari-
ables are provided. In the configuration of the system, you
need to use the latest WIN10 system or medium-sized con-
figurations such as the IOSX 10.7+, the USB 2.0 port and
the Inter i7 processor.
Wireless Transmission System and Connection Set-
tings: The wireless communication technology of the
Internet is a supporting technology for mobile services.
In the Augmented Reality technology, a large amount of
corresponding data needs to be stored by the server, and
the processing of the data is also implemented by using a
remote server.
Another requirement in the system is wireless transmis-
sion and connection which is used to connect to the wire-
less router. As for the settings for the wireless router, we
can create a local network for the computer and client act-
ing as a server to connect to this network for information
exchange. The server is mainly responsible for tracking
the movement and the gesture of the hand, and selecting
the corresponding object, synchronously connecting the
client and sharing the interaction. Locking the variable
changed by the client is a basic task of the server. Under
this condition, multiple clients cannot change related vari-
ables at the same time. This mode is mainly used to select
the intangible cultural heritage artifacts. When the user
selects this mode, it will be sent to all clients through
the server port. This behavior occurs on the client, so all
data is synchronized. Suppose each client selects the next
object at the same time. If we don’t control the variables,
it will cause the system to crash.
7 Limitations
In this system, it is applicable not only to central HMD
mobile electronic devices. It also works for other devices
on the same server, and there are many social experience
interactions in this mode. However, with Unitiyd’s test
server, it is not certain whether it will be available in the
future.
In the cultural hall, the normal operation of the system
requires the proximity of light and real crafts. The selected
area needs to have consistent illumination to adequately
illuminate the craft without projection. The area with
backlight, that is, the area where the light is opposite to
the camera, because this will cause the image to darken,
the interactive system does not perform well in virtual
human–computer interaction. Therefore, it is hard to track
the interaction well under the backlight of the crafts in the
cultural hall under natural light conditions. Most cultural
centers have plenty of artificial lighting, however, because
Evolutionary Intelligence
of its unpredictability, it is less natural than in the condi-
tion of natural light. The authenticity of the Intangible
Cultural Heritage is also important. In the future experi-
ments, the tracking virtual program design will be further
improved.
8 Summary
The use of new technologies to protect the intangible cul-
tural heritage in private places is an inevitable trend of
the times. By using the system to construct realistic three-
dimensional models, combined with AR and the natural
interaction between people and numbers. The sensory expe-
rience that brings visitors real space, the presence of virtual
in AR allows visitors to feel real. You can use the sense of
touch to perceive them. This paper proposes a space sys-
tem in which the virtual digital intangible cultural heritage
interacts with nature, and the related production process is
given to the system. Visitors can easily interact with specific
intangible cultural heritages using low-cost technologies and
components. It provides a new display model for the local
Intangible Cultural Heritage by using mobile devices and
Leap motion in HMD cases, and software in Unity 3D. We
can easily bring a new way of display to the folk intangible
cultural heritage center, and also a new form of preservation.
Acknowledgements  This research was financially supported by the
Undergraduate Innovation Training Project of Guangdong University
of Foreign Studies in 2019, China (No. 201911846011).
Compliance with ethical standards  surgical navigation system based on artificial intelligence and
Conflict of interest  We all declare that we have no conflict of interest
in this paper.
References
1. Li C, Sun R, Dai Y, Cai S, Li Q, Li J (2017) Intelligent exhibition
platform of chinese ancient farming virtual scene based on Unity
3D. Trans Chin Soc Agric Eng 33:308–314
2. Mckenna M, Zeltzer D (1992) Three dimensional visual display
systems for virtual environments. MIT Press, Cambridge
3. Bolic M, Parri J, Wang W (2017) Pvfpga: paravirtualising an
FPGA-based hardware accelerator towards general purpose com-
puting. Int J High Perform Comput Netw 10(3):179
4. Rossetti V, Furfari F, Leporini B, Pelagatti S, Quarta A (2018)
Enabling access to cultural heritage for the visually impaired:
an interactive 3D model of a cultural site. Proc Comput Sci
130:383–391
5. Hatala M, Wakkary R (2005) Ontology-based user modeling in an
augmented audio reality system for museums. User Model User-
Adapted Interact 15(3–4):339–380
6. Chang KE, Chang CT, Hou HT, Sung YT, Chao HL, Lee CM
(2014) Development and behavioral pattern analysis of a mobile
guide system with augmented reality for painting appreciation
instruction in an art museum. Comput Educ 71:185–197
7. Han DID, Dieck MCT, Jung T (2019) Augmented Reality smart
glasses (arsg) visitor adoption in cultural tourism. Leisure Stud
4:1–16
8. Di Serio Á, Ibáñez MB, Kloos CD (2013) Impact of an augmented
reality system on students’ motivation for a visual art course.
Comput Educ 68:586–596
9. Chen A, Chen J, Min H (2019) A united framework with multi-
operator evolutionary algorithms and interior point method for
efficient single objective optimisation problem solving. Int J High
Perform Comput Netw 13(3):340–353
10. Mota JM, Ruiz-Rube I, Dodero JM (2018) Inmaculada Arnedillo-
Sánchez Augmented reality mobile app development for all. Com-
put Electr Eng 65:250–260
11. Tran TX, Younis A, Pompili D (2017) Understanding the compu-
tational requirements of virtualized baseband units using a pro-
grammable cloud radio access network testbed. In: IEEE interna-
tional conference on autonomic computing, IEEE
12. Yang YL, Ma WJ, Zhao KZh, Dong QK (2017) Communication-
aware task scheduling algorithm for heterogeneous computing. Int
J High Perform Comput Netw 10(4/5):298–309
13. Abdullah NAS, Rusli NIA, Ibrahim MF (2015) Mobile game size
estimation: COSMIC FSM rules, UML mapping model and Unity
3D game engine. Open Systems
14. Shepherd R, Peele B, Murray BM, Barreiros J, Luebke D (2017)
Stretchable transducers for kinesthetic interactions in virtual real-
ity. In: ACM SIGGRAPH 2017 emerging technologies, ACM
15. Liu Y, Tang P (2019) The prospect for the application of the
augmented reality. In: 2018 IEEE international conference on
artificial intelligence and virtual reality (AIVR)
16. Polvi J, Taketomi T, Moteki A, Yoshitake T, Fukuoka T, Yama-
moto G et  al (2018) Handheld guides in inspection tasks:
augmented reality versus picture. IEEE Trans Vis Comput
Graph 24(7):2118–2128
17. Hsiao SW, Lee CH, Yang MH, Chen RQ (2017) User interface
based on natural interaction design for seniors. Comput Human
Behav 75:147–159
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
13