Demonstration
21–25, 2022, Virtual Event, Tempe, AZ, USA
Personalized Information Retrieval for Touristic Attractions in
Augmented Reality
Felix Yang, Saikishore Kalloori, Ribin Chalumattu and Markus Gross
Media Technology Center
ETH Zürich, Switzerland
fyang@student.ethz.ch,ssaikishore@ethz.ch,cribin@inf.ethz.ch,grossm@inf.ethz.ch
ABSTRACT
The rapid advances and increasing accessibility of augmented re-
ality (AR) in recent years opened up many new possibilities to
incorporate AR into our daily lives. A very interesting area for
AR is tourism where one can enhance attractions with virtual ele-
ments and provide tourists with additional information about the
places they are visiting. In this paper, we present our prototype,
an AR application that augments various points of interest (POIs)
by showing images and facts about each POI. We also developed
a simple recommender system that ensures the facts are selected
based on user preferences, thus creating a unique and personalized
experience for each user. Furthermore, we also conducted a live user
study to assess the usability of our prototype and the usefulness of
our personalization system.
CCS CONCEPTS
• Human-centered computing → Mixed / augmented reality.
KEYWORDS
Augmented Reality, Tourism, Personalization
ACM Reference Format:
Felix Yang, Saikishore Kalloori, Ribin Chalumattu and Markus Gross. 2022.
Personalized Information Retrieval for Touristic Attractions in Augmented
Reality. In Proceedings of the Fifteenth ACM International Conference on Web
Search and Data Mining (WSDM ’22), February 21–25, 2022, Tempe, AZ, USA.
ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3488560.3502194
1 INTRODUCTION
Augmented reality enhances the perception of reality by superim-
posing virtual content on top of the view of the real world. Using
sensory and visual information, an illusion can be created where
virtual objects and information seem to reside in the real world
[1]. The increasing computing power of smartphones and the rapid
development of AR technologies in the past years have enabled
more and more stable, immersive and accessible AR experiences
for consumers.
The motivation of our work is to demonstrate the potential of AR
in the tourism sector by creating an enhanced AR experience where
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
WSDM ’22, February 21–25, 2022, Tempe, AZ, USA
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9132-0/22/02. . . $15.00
https://doi.org/10.1145/3488560.3502194
1613
WSDM ’22, Feb.
users can visit points of interest in the city of Zurich, Switzerland
and learn some interesting facts about the places they are visiting.
A major goal of this project is to create a unique experience for each
user by showing only relevant and personalized content based on
the preferences of the user. By developing a simple recommender
system, we are able to provide users with facts which coincide with
their interests, thus preventing an overload of information that is
irrelevant to the users.
2 RELATED WORK
Augmented reality has been showing significant promise in many
different domains such as medicine, construction, manufacturing,
advertising or gaming [8]. It has also become increasingly popular
within the tourism industry to improve the tourists experience
and to reach a wider audience for attractions. Not only is there a
possibility to provide tourists with valuable information, but the
seamless integration of virtual content into the real environment
also enables an appealing and engaging way to present multimedia
content to users [6]. By considering user interests, one can further
improve user experiences using personalization methods [6].
Augmenting an attraction with digital content requires precise
techniques to locate the device within the environment containing
the POI. Using GPS and compass, a rough estimate of the device’s
position and rotation can be obtained [10]. However, those esti-
mates are not sufficient to place content with high accuracy due
to GPS errors. Vision-based localization approaches provide much
better solutions for such purposes [9]. Marker-based methods are
very popular to estimate the pose of the camera relative to a QR
code or any 2D image/planar object with enough visual features.
However, markers are less suitable to detect and track arbitrary
3D structures such as statues which may look completely different
depending on the viewpoint. For that, more sophisticated meth-
ods are required such as machine learning-based approaches with
object detection networks [4] or approaches where the currently
observed environment is matched against 3D point clouds of previ-
ously scanned environments. In our work, we used a localization
method based on point clouds.
MobiAR [7] is an AR platform which provides tourists with
information and multimedia content of the places they are visit-
ing. When using the app, icons associated with different POIs are
overlaid on top of the camera view according to the distance and
direction of each POI. When clicking on an icon, users are able to
browse multimedia content related to the corresponding POI.
3 DESCRIPTION OF OUR PROTOTYPE
In order to use our prototype, new users first have to set up their
user profile after starting the app for the first time. During the
Demonstration
(a) Reconstructed point cloud (b) Reconstructed textured mesh
(d) AR view with icons
Figure 1: Screenshots of the reconstructions and the AR experience at the POI
registration process, users first have to provide their name, age and
gender. After that, they can define their preferences by choosing
which of the following six areas they are interested in: politics,
history, economy, art, religion and technology. Finally, they have to
fill out the Five-Item Personality Inventory (FIPI) questionnaire [3]
to define their personality traits. We use age, gender and personality
traits to provide a personalized user experience [5].
When the registration is finished, the system is ready for the
AR experience. Users can then visit points of interest in the city
of Zurich that are supported by our application, log in with their
profile and aim their phone at the POI they are visiting. When
the POI gets recognized by the app, two icons will appear in the
AR view next to the POI (see figure 1d). Clicking on the image
icon will show an image and clicking on the information icon will
reveal facts (see figure 1e) about the POI which are chosen by the
personalization system based on the user preferences.
3.1 Object Detection and Tracking
We used the Immersal software development kit for Unity to detect
and track points of interest. Immersal SDK is an AR solution which
lets you spatially map real-world locations and then augment them
with digital content. Contrary to some other AR SDKs, Immersal
SDK works for very large spaces and is even scalable to entire cities.
As a first step, we made a selection of interesting POIs in the
city of Zurich that we wanted to include in our application. For
our prototype, we chose 3 buildings and 4 famous statues that
were not too far from each other. We then went to Zurich and
1614
WSDM ’22, Feb. 21–25, 2022, Virtual Event, Tempe, AZ, USA
(c) Setup of the AR experience in Unity
(e) AR view after clicking on the icons
used the mapper app by Immersal to map all the POIs by taking
numerous pictures of each POI from multiple viewpoints. Each set
of images was then uploaded to the Immersal Cloud Service which
then constructed a 3D point cloud (see figure 1a) and a textured
mesh (see figure 1b) of each POI by finding distinct visual features
and matching those features from the set of images. Additionally,
each constructed point cloud was tagged with the GPS coordinates
of the corresponding POI such that it is possible to narrow down
nearby POIs when using the app and thus making the process of
finding out which POI the user is currently looking at more scalable.
We then used the textured meshes inside the Unity editor as a point
of reference to define the position, rotation and scale of each piece
of virtual content at each point of interest (see figure 1c). Without
those meshes, it would have been very difficult to guess where to
place each piece of content.
When using the app, the current GPS coordinates of the device
are regularly used to find out which POIs are nearby to the user.
Every two seconds, the current camera frame is captured and sent
to the Immersal Cloud Service. This serivce then analyzes the sub-
mitted image and tries to match it with a point cloud of one of
the nearby POIs. If there is a match, the cloud service returns a
projection matrix which is used to extract the current position and
orientation of the device. This information is then combined with
the device’s local SLAM (Simultaneous Localization and Mapping)
system to create the illusion that the virtual content is attached to
the physical world. We noticed that by using the Immersal SDK,
we were able to detect and track our POIs very reliably, even under
vastly different lighting conditions.
Demonstration
3.2 Personalization System
When our application recognizes a POI, it sends a request with the
ID of the POI and the ID of the current user to our server. Using
these two IDs, our recommender system fetches the corresponding
user data and all existing facts about the POI from our database.
Previous research has shown that representing a POI by a fixed
number of 𝑛 Boolean features is useful in Tourism [5]. In the first
step, the system constructs a feature vector 𝑢 = (𝑢 1 , . . . , 𝑢 6 )𝑇 for the
(𝑖) (𝑖)
user and feature vectors 𝑥 (𝑖) = (𝑥 1 , . . . , 𝑥 6 )𝑇 for each fact 𝑖. 𝑢 is
a 6-dimensional boolean vector and its entries describe which of
the six categories (politics, history, economy, art, religion and tech-
nology) the user is interested in. For example, if 𝑢 = (0, 1, 1, 0, 0, 0)𝑇 ,
this means that the user is interested in history and economy. 𝑥 (𝑖)
(𝑖)
is a 6-dimensional integral vector where each entry 𝑥 𝑗 ∈ {0, 1, 2}
describes if and how much fact 𝑖 belongs to category 𝑗. For ex-
ample, if 𝑥 (𝑖) = (2, 1, 0, 0, 0, 0)𝑇 , this means that fact 𝑖 is mainly
about politics, but also a partly about history. Of course, one could
have for example used arbitrary weights between 0 and 1. However,
deciding how much a fact actually belongs to a certain category
is quite subjective in many cases. By restricting ourselves to only
using the three integers 0, 1 and 2, we were able to annotate each
fact in a much simpler and more consistent manner.
Using 𝑢 and 𝑥 (𝑖) , we can model the user’s utility function 𝑓 as:
6 •
(𝑖)
Õ
𝑓 (𝑖) = 𝑢𝑘 𝑥 𝑘 (1)
𝑘=1
That is, for each fact 𝑖, we calculate its utility score 𝑓 (𝑖) and then sort
all non-zero scores in a descending order to create a ranking for the
user. The more the categories of a fact coincide with the preferences
of the user, the higher its score will be. The recommender system
then returns the top facts of the ranking to the mobile app.
We noticed that in some cases, the system may not find enough
facts for the user. For example, if the user is only interested in
economy but there are not any facts about economy for the POI
that the user is currently visiting. In these special cases, the system
will return facts based on the preferences of similar users. For that,
it first performs a user-based nearest neighbor search based on age,
gender and personality traits in order to find similar users to the
current target user. Then, it constructs a new feature vector for the
user by averaging the feature vectors of those similar users (i.e., by
using similar users’ vector 𝑢). This new feature vector is then used
to calculate new utility scores for each fact and the system returns
the missing facts based on the new ranking.
Fact Gathering: To gather all facts, we used publicly accessible
APIs to retrieve data about the attractions we wanted to include in
our application and also collected few facts manually. Our database
overall has a more or less balanced distribution over all categories.
Additionally, we also developed a simple annotation interface which
allowed us to define the categories for each fact and to upload all
facts to our database in a quick and automated manner.
Implementation: We developed our mobile application1 using
the Unity engine, a commonly used development platform to create
video games and 3D applications. Our recommender system was
implemented in Python as a server application using the Flask web
framework and we stored all the facts and user data with AWS. We
1 Source code: https://github.com/MTC-ETH/ar-personalized-facts-recommendation
1615
WSDM ’22, Feb. 21–25, 2022, Virtual Event, Tempe, AZ, USA
also share a demo video2 of our AR application presenting users
with personalised facts retrieval.
4 USER STUDY
We conducted a live user study to test and evaluate the usability of
our prototype and the usefulness of our personalization system. In
total, 15 people participated in our experiments. They were aged
between 20 to 35 and the majority were either undergraduates,
research students or working employees. The participants were
thus among the typical users of tourism apps.
For each user, we made a selection of 2-3 points of interest to
visit in order to attain a reasonable duration for the study. After
creating their user profiles, participants were asked to visit the POIs
we assigned to them. At each POI, users were able to experience
our virtual information in AR (see figure 1d and 1e). In order to test
the usefulness of the personalization system, we also included a
randomized system as the baseline model. The randomized system
randomly picks facts about the POIs without taking the preferences
of the users into account. Each user tested the prototype with both
of the models: our personalization system (described in section 3.2)
and the randomized system. Next, to evaluate the perceived useful-
ness of our personalization system, users filled out a questionnaire
containing the following statements after visiting each POI:
The suggested facts were relevant. (PersStmt1)
• I liked the facts suggested by the system. (PersStmt2)
• The system suggested too many bad facts. (PersStmt3)
• The personalized facts fit my preference. (PersStmt4)
• The personalized facts fit my preference better than the
randomized facts. (PersStmt5)
At the end of the study, participants also filled out a questionnaire
to assess the usability of our prototype. For that, we used the System
Usability Scale (SUS) [2], a simple and well established tool for
measuring usability. The 10 SUS statements were:
• I think that I would like to use this system frequently.
• I found the system unnecessarily complex.
• I thought the system was easy to use.
• I think that I would need the support of a technical person
to be able to use this system.
• I found the various functions in the system were well inte-
grated.
• I thought there was too much inconsistency in this system.
• I would imagine that most people would learn to use this
system very quickly.
• I found the system very cumbersome to use.
• I felt very confident using the system.
• I needed to learn a lot of things before I could get going with
this system.
Furthermore, we also included two statements about the appli-
cation potential of our prototype:
• The application is useful to learn more about attractions.
(AppStmt1)
• I would be interested in using such an application for other
attractions in Zurich or other cities. (AppStmt2)
2 video:https://github.com/MTC-ETH/ar-personalized-facts-
recommendation/tree/master/Demovideo
Demonstration WSDM ’22, Feb. 21–25, 2022, Virtual Event, Tempe, AZ, USA

Neither The majority of the participants generally liked the facts shown
Strongly Dis- to them and thought that the shown facts were relevant. About
disagree Agree Strongly
t
en

dis- agree 73% of the participants also agreed that our system did not show
nor agree (%) Agree
me

agree (%)
at

(%) (%) too many bad facts about the POIs. Our personalization system
St

(%)
was able to successfully accommodate the user preferences in most
PersStmt1 0 0 15 41 44
cases (79%). Around 64% of all users affirmed the usefulness of our
PersStmt2 0 6 15 38 41
personalization system and were shown facts which clearly fit their
PersStmt3 32 41 21 6 0
preferences better than the randomly picked facts. However, some
PersStmt4 0 9 12 44 35
users (36%) felt that there was not an obvious difference between
PersStmt5 9 9 18 29 35
the personalized and the randomized fact retrieval. This is due to
Table 1: Results of the personalization system
the fact that some personalized facts occasionally also appeared as a
Neither randomized fact or some randomized facts also happened to fit the
Strongly Dis- preferences of the users. Furthermore, a few users were interested
disagree Agree Strongly
t
en

dis- agree in a lot of categories and therefore, most facts accommodated their
nor agree (%) Agree
em

agree (%) preferences, even when picked at random.

at

(%) (%)
St

(%)
AppStmt1 0 0 0 40 60
AppStmt2 7 0 13 47 33
Table 2: Results of the application potential
Each item in our questionnaire had to be replied on a five points
Likert scale from "strongly disagree" to "strongy agree".
5 EXPERIMENTAL RESULTS
System Usability: For scoring a SUS we followed the standard pro-
cedure for calculating the overall SUS score, which ranges between
0 and 100 [2]. The response to each item of the SUS questionnaire
section is associated with a score ranging from 0 (strongle disagree)
to 4 (strongle agree). The score contribution of each positively
phrased statement is the score of the response minus 1. For neg-
atively phrased statements, the score contribution is 5 minus the
score of the response. The total sum of all score contributions is
then multiplied by 2.5 to obtain the overall SUS score. In our exper-
iments, we used the widely known and popular benchmark [2] of
500 studies score 67 as benchmark for our system’s usability [5].
Overall, our prototype obtained a SUS score of 87.3 which is
significantly (𝑝 < 0.05) higher than the benchmark SUS score 67.
The lowest SUS score given by any participant was 75 which is also
above the benchmark SUS score. The high SUS scores given were
in accordance with our expectations because our application has
a very simple interface and is quite straightforward to use - users
only need to aim their smartphones at the POIs and click the icons
when they appear. Indeed, none of the users had difficulties to use
the app nor found the prototype too complex.
However, a couple of users found some functions of our system
not well integrated such as the missing option to zoom into the
facts when standing further away from the POI. Occasionally, some
inconsistencies were found when the localization algorithm of Im-
mersal did not manage to properly match the camera image with
the point cloud of the POI. In those situations, the virtual content
was not positioned correctly next to the POI. In our future work,
we plan to explore more accurate localization algorithms and add
additional features into our prototype.
Personalization System: For the evaluation of our personaliza-
tion system, we averaged the responses over all users and all points
of interest that were visited. Table 1 shows the responses given by
all the participants (in percentage) to the questionnaire statements.
Application Potential: All the participants agreed that our pro-
totype was useful to learn more about the attractions they were
visiting (AppStmt1). Observing table 2, we note that 12 users (80%)
were also interested in using such an application for other attrac-
tions in Zurich or other cities.
6 CONCLUSION
In this paper, we have presented our prototype, a mobile AR appli-
cation which provides users with facts about the attractions they
are visiting while taking their preferences into account. Our user
study showed that the prototype has a very high usability and that
our personalization system works adequately in most situations.
However, we have barely scratched the surface of what is possible
with augmented reality in the tourism sector. In future work, we
plan to achieve an even more compelling and entertaining user
experience by incorporating videos, audio or animated 3D models
and navigation support to suggested places. We also plan to increase
the utility of the prototype by providing personalized suggestions
for shops, restaurants and accommodations.
REFERENCES
[1] Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., Blair, M.: Recent advances
in augmented reality. ieee computer graphics and applications (2001)
[2] Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system
usability scale. International Journal of Human–Computer Interaction 24(6),
574–594 (2008)
[3] Gosling, S.D., Rentfrow, P.J., Swann, W.B.: A very brief measure of the big-five
personality domains. Journal of Research in Personality 37(6), 504–528 (2003)
[4] Hu, M., Weng, D., Chen, F., Wang, Y.: Object detecting augmented reality system.
In: 2020 IEEE 20th International Conference on Communication Technology
(ICCT). pp. 1432–1438 (2020)
[5] Kalloori, S., Ricci, F., Gennari, R.: Eliciting pairwise preferences in recommender
systems. In: Proceedings of the 12th ACM Conference on Recommender Systems.
pp. 329–337 (2018)
[6] Kounavis, C.D., Kasimati, A.E., Zamani, E.D.: Enhancing the tourism experi-
ence through mobile augmented reality: Challenges and prospects. International
Journal of Engineering Business Management 4, 10 (2012)
[7] Marimon, D., Sarasua, C., Carrasco, P., Álvarez, R., Montesa, J., Adamek, T.,
Romero, I., Ortega, M., Gascó, P.: Mobiar: Tourist experiences through mobile
augmented reality. Information and communication technologies in tourism (01
2010)
[8] White, J., Schmidt, D., Golparvar-Fard, M.: Applications of augmented reality.
Proceedings of the IEEE 102, 120–123 (02 2014)
[9] Wolf, J., Burgard, W., Burkhardt, H.: Robust vision-based localization by combin-
ing an image-retrieval system with monte carlo localization. IEEE transactions
on robotics 21(2), 208–216 (2005)
[10] Yovcheva, Z., Buhalis, D., Gatzidis, C.: Smartphone augmented reality applications
for tourism. E-review of tourism research (ertr) 10(2), 63–66 (2012)

1616