Mobile Application for the Recommendation of Furniture and
Appliances Through Augmented Reality to Improve The User
Experience in The Online Shopping Process
Ricardo A. Guevara Jordy J. Rojas
School of Systems and Computer School of Systems and Computer
Engineering, Peruvian University of Engineering, Peruvian University of
Applied Sciences, Peru Applied Sciences, Peru
u201618078@upc.edu.pe u201512406@upc.edu.pe
ABSTRACT
Due to the increase in online shopping, companies are looking
to reinvent themselves and improve their ecommerce platforms.
However, some users still do not dare to make an online purchase
mainly because they cannot physically view the product. As a result
of this, innovative technologies have started to emerge. Therefore,
this project seeks to improve the customer’s user experience, using
augmented reality and colour recognition to recommend products
in a mobile application. For the development of the application,
ARCore was used as a software development kit (SDK) to display
products and take measures with augmented reality. In addition,
a colour recommendation was implemented using the Palette API
(Application Programming Interface) for colour recognition and
an algorithm based on colour theory to generate the combinations.
The usability of the application was evaluated by a group of users,
who completed a form that was based on Nielsen’s heuristic prin-
ciples. The main conclusions of the analysis of the responses are
that they would make purchases using the application without hav-
ing to physically see the product, users understood the purpose of
using augmented reality and colour recognition to obtain recom-
mendations. In addition, they were able to use all the functionalities
implemented in the application without the need for a user man-
ual, which highlights the minimalist design and ease of use of the
application.
CCS CONCEPTS
• Applied computing → Electronic commerce; Online shopping.
KEYWORDS
E-business, E-commerce, online shopping, augmented reality
ACM Reference Format:
Ricardo A. Guevara, Jordy J. Rojas, and David G. Quevedo. 2022. Mobile
Application for the Recommendation of Furniture and Appliances Through
Augmented Reality to Improve The User Experience in The Online Shopping
Process. In 2022 3rd International Conference on Internet and E-Business (ICIEB
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
ICIEB 2022, June 15–17, 2022, Madrid, Spain
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9732-2/22/06. . . $15.00
https://doi.org/10.1145/3545897.3545898
1
David G. Quevedo
School of Systems and Computer
Engineering, Peruvian University of
Applied Sciences, Peru
pcsidque@upc.edu.pe
2022), June 15–17, 2022, Madrid, Spain. ACM, New York, NY, USA, 6 pages.
https://doi.org/10.1145/3545897.3545898
1 INTRODUCTION
In recent years, the constant advance of technology has marked
a change in the way humans perform their daily tasks and what
tools are used to accomplish them. A clear example of this is the
way products are currently bought and sold [1]. Despite the signifi-
cant increase in online shoppers, not all product categories have
increased their sales. During social isolation, the products that have
been most in demand are health products, followed by technologi-
cal products [2]. This is because there is still great insecurity on the
part of the buyer. According to Mastercard, of the people who have
not yet bought virtually, 31% do not dare to do so because they do
not see the product in person, 29% simply distrust, 15% think that
their card data will be stolen, 15% consider that shipping costs are
high and 13% believe that the order will not arrive [3]. Based on
this problem, the use of augmented reality (AR) is proposed, a tech-
nology that has not yet been fully exploited, so that users have a
tool that allows them to find the product they need, through recom-
mendations, and can really visualize what the products would look
like, without leaving home. One of the limitations is compatibility
with the devices since, in addition to the fact that the solution is
only compatible with the Android operating system, augmented
reality is only available for devices that have version 7.0 or later.
This problem has not been solved because there is no application
that satisfies the need to acquire products in a remote way that
offers a different user experience with high security and reliability.
In this paper a solution is proposed that will allow users to view
products through augmented reality, obtain recommendations by
size, measuring the available space, also with augmented reality
and obtain recommendations by colour, generating combinations
based on colours recognized from a photograph. The application
was developed for Android devices with Google’s ARCore software
development kit (SDK) and Palette API in the Kotlin programming
language. The application is developed for Android devices with
version 7.0 (Nougat) onwards that support the Google Play service
for AR. In addition, Firebase services are used such as: Cloud Fire-
store, for the non-relational database; Storage, for the storage of
3D models and images of the products and Authentication, for the
login management.
Our contributions are the following:
• 1. Being able to visualize the products to buy with augmented
reality so that the user has a better perspective of the product.
ICIEB 2022, June 15–17, 2022, Madrid, Spain
• 2. To be able to recommend a product based on the colours of
an environment that will be obtained through a photograph
taken by the camera of the mobile device. This is in order to
recommend a product that combines and is in harmony with
the environment in which you want to place the product.
• 3. To be able to recommend a product based on dimensions
such as height, width and length. Which will be measured
from the same application. This is in order to recommend
products that fit in the space that the user has available.
2 CONTEXT
2.1 Augmented Reality [4]
According to Nowacki and Woda, AR combines the real world with
the virtual world, and can be captured through a lens. For example,
this technology is used in the solution to allow the user to view the
products from their mobile device.
2.2 ARCore [5]
It is Google’s platform for the development of applications with Aug-
mented Reality, it allows the device to recognize the environment
and, based on that information gathering, create new experiences
interacting with the user. ARCore has three important characteris-
tics:
• Motion tracking.
• Environmental understanding.
• Light estimation.
ARCore can detect normal, smooth or vertical surfaces, like
tables or dividers, and makes them accessible to applications such
as plans. ARCore can also decide the limits of each plane and make
this data accessible to the application. For example, it is used in
the solution for taking measurements of the available space using
environmental understanding.
2.3 Palette API [6]
It is a compatibility library that allows extracting the highlighted
colours of an image, to make the application more attractive. It
can detect 6 main colours in an image and is generally used to
change the colours of the application’s design components such
as headings or subtitles. For example, it is used in the solution
to recognize the colours of a photograph taken by the user, and
based on that, generate combinations. From the aforementioned
definitions, it is understood that it is possible to provide the user
with a better shopping experience using technologies such as AR
and colour recognition, which not only help to visualise the product
before buying it but also help the user to decide what product is the
one that can best be adapted to the space that they have available.
2.4 Nielsen’s heuristic principles [7]
Heuristics were created in 1990 by Jakob Nielsen. These 10 "laws"
guarantee that all tasks and operations carried out in a software are
executed in an efficient and straightforward way, thus creating a
more intuitive and user-friendly interface. The heuristic principles
are the following:
• 1. Visibility of system status
• 2. Match between system and the real world
2
Ricardo Alfredo Aparicio et al.
• 3. User control and freedom
• 4. Consistency and standards
• 5. Error prevention
• 6. Recognition rather than recall
• 7. Flexibility and efficiency of use
• 8. Aesthetic and minimalist design
• 9. Help users recognize, diagnose, and recover from errors
• 10. Help and documentation
3 MAIN CONTRIBUTION
As mentioned above, ARCore was used to offer the user the ability to
measure the different spaces that they have available when wanting
to purchase a product and based on this, recommend a product that
fits the previously measured space. In addition, the user can choose
to obtain a recommendation by colour, which consists of taking
a photograph of the environment he wants to place the product
in, to carry out this functionality it was considered to recognize
the main colours of the image, combinations are generated and,
finally, matching products recommended. Finally, the user can view
the products with AR to know how the product will look after it
is purchased. For the visualization of the products, two functions
were generated for the loading and positioning of the product with
AR, loading the 3D models from Firebase Storage. Please see Fig. 1
and 2.
On the other hand, for the colour recommendation functionality,
first the user is allowed to open the device’s camera application and
take a picture, from this image, 6 main colours are obtained, defined
by the Palette API as: VibrantColor, DarkVibrantColor, LightVi-
brantColor, MutedColor, LightMutedColor, and DarkMutedColor.
Once recognized, colour combinations such as: Complementary,
Split Complementary, Triadic and Analogous are generated (See
Fig. 2). Based on these results, all products that combine with the
environment are recommended. Finally, for the recommendation
by size, ARCore is used since it allows obtaining the location of a
point within a surface. To measure the length and width, the user is
allowed to locate 2 points within the surface (See Fig. 3), from the
position of these points, the distance between them is calculated
and finally the measurement obtained is changed to centimetres
(See Fig. 4). To perform the height measurement, the user is allowed
to locate a point on the surface, once located, he is allowed to man-
ually increase or decrease the height until the desired measurement
is found. It is important to note that the measurements obtained are
not exact, they have a margin of error of ± 2cm and, on the other
hand, when loading 3D models from Firebase, there is a delay of 2.5
seconds when positioning the model to be viewed with augmented
reality.
4 RELATED WORKS
4.1 The Use of Marker-Based Augmented
Reality in Space Measurement [8]
The authors seek to find an algorithm to perform measurements
using AR with markers. The application was developed with Unity
and the OpenCV library. It begins with the functionality of recog-
nizing the markers with the device’s camera, once this is done, the
system calculates the angle of the walls and the distance between
them. If the points are perpendicular to each other, another 2 points
Mobile Application for the Recommendation of Furniture and Appliances Through Augmented Reality to Improve The User
Experience in The Online Shopping Process
Figure 1: Model display functions
must be captured, and they must be combined with the points al-
ready obtained previously to obtain the distance between the walls.
The authors obtained high precision measurements. Compared to
our solution, ARCore allows us to dispense with the use of book-
marks, which is optimal for applications where usability and user
experience must prevail, since only the mobile device is needed to
start using the application.
4.2 Using Augmented Reality to Measure
Vertical Surfaces [9]
In this paper, the authors seek to obtain vertical surface measure-
ments using ARKit and Unity. The application allows the user to
locate 3 points on the plane, from this, lines are drawn between the
points generating a polygon, and the Shoelace formula is used to
calculate the area. The authors obtained results with an imprecision
of 6.2%. Compared to our solution, we obtained a margin of error
of ± 2cm and the height measurement was implemented, which is
necessary for the user to be able to fully measure the space they
have available.
3
ICIEB 2022, June 15–17, 2022, Madrid, Spain
4.3 Iterative prototyping of urban cobuilder:
Tracking methods and user interface of an
outdoor mobile augmented reality tool for
co-designing [10]
The authors developed an application for phones where the user
can work collaboratively in the construction of urban designs us-
ing augmented reality with markers. In the first simulation, they
worked with photographs of buildings located in strategic places.
Each building represented a marker, however this proved insuffi-
cient due to the general instability of recognition of a photographed
building. Later, functions such as GPS, compass and gyroscope were
used for motion detection and alignment. It was finally determined
that the distance between the user and the building posed a prob-
lem for marker detection. Our solution, by using augmented reality
without markers, gives the user the possibility of being free to use
the application in any environment and space, so that the distance
does not mean a problem for its correct use.
4.4 AR Furniture: Integrating Augmented
Reality Technology to Enhance Interior
Design using Marker and Markerless
tracking [11]
In this paper, the authors seek to visualize a product in a real
environment from their homes using AR with and without markers.
The developed application consists of 4 stages which are: The input
of the video from the camera to send them to the other processes,
ICIEB 2022, June 15–17, 2022, Madrid, Spain
Figure 2: Viewing 3D Models
the detection of the markers, the modelling of the positioning of
the objects where the image frame can be superimposed and the
interaction interface. As results it was obtained that when markers
are used, the distance between them and the camera should not be
greater than 60 cm. On the other hand, when it comes to the solution
without markers, it works better on light surfaces and the lighting
should be between 100 and 300 lux. Compared to our solution,
ARCore automatically regulates lighting and can detect several
types of surfaces, which allows the user to use the application
wherever they want without the need to search for an environment
with specific lighting.
4
Ricardo Alfredo Aparicio et al.
Figure 3: Recognition and generation of colour combinations
5 EXPERIMENTS
This project was developed using the agile Scrum framework, which
consisted of 9 Sprints of 3 weeks in each one. In each Sprint, func-
tional tests were performed on each user story and the documenta-
tion of the main artifacts. The result was a certificate from ITService,
an internal company of the university that provides support and
support with certifications and software quality tests.
5.1 Experimental protocol
The user satisfaction tests were carried out with a survey that was
based on the 10 heuristic principles of Nilsen. The surveyed users
had Android devices that supported augmented reality. The first
thing that was done was to install the APK of the application. Once
the correct installation was validated, the main characteristics that
our proposal offers were indicated so that the user can put it to
use on their device and then can answer the survey. In addition to
these heuristic principles, which are based on usability and user
experience, users were asked if they would be willing to use the ap-
plication’s functionalities to purchase a product without physically
seeing it.
5.2 Theoretical evaluation
The interview experiment was conducted with 25 participants who
had augmented reality compatible devices and had Android 7.0
Mobile Application for the Recommendation of Furniture and Appliances Through Augmented Reality to Improve The User
Experience in The Online Shopping Process ICIEB 2022, June 15–17, 2022, Madrid, Spain

Figure 4: Calculate distance between 2 points

Figure 5: Distance calculated between 2 points

5
ICIEB 2022, June 15–17, 2022, Madrid, Spain
or onwards. Once users fully interacted with the application and
tested all the implemented functionalities, they were asked to fill
out a form. From the data obtained, it was concluded that 98% of the
users indicated that it was possible to use the application without
the need for a manual or instructions. On the other hand, users
indicate that they find the design of the application attractive and
that they can use it by applying the previous knowledge acquired
in other shopping applications. Users also note that the app was
able to recover from surface recognition errors and take photos
for colour recommendation. Finally, all users indicated that the
measurement was accurate, this means that the margin of error
was not perceived by the users, which made a pleasant impression
on them.
5.3 Evaluation of the experimental comparison
When analysing the responses of our interviewees for each heuris-
tic, we reached the following conclusions:
• 1. The technologies used for the development of the project
allow to meet the quality attributes of availability, perfor-
mance and usability.
• 2. Users understand the purpose of using augmented reality
for taking measurements and displaying products. In addi-
tion, they deduce how to use the application without the
need for a manual or guide.
• 3. Users find the design and flow of the application attrac-
tive, however, some interviewees indicate that it could be
improved.
• 4. The margin of error in taking measurements did not mean
an inconvenience for users when interacting with the appli-
cation.
• 5. The small delay when loading the products to be viewed
with augmented reality was not highlighted as a defect in
the interviews. Users are willing to purchase a product using
the functionalities of our solution, without having to have
previously seen the product.
6 CONCLUSIONS AND PERSPECTIVES
The project consisted of the development of a mobile application
with augmented reality for Android devices, which allows users to
have a better online shopping experience. With the application, the
user will obtain product recommendations according to their needs,
such as a recommendation by colour of a room or a recommenda-
tion by size in a specific space. Once the user has found the product
that suits their needs, the application allows them to visualise it
with augmented reality. Google’s ARCore SDK was used for this,
as this is the tool that recognises the largest surfaces. Compared
to other applications, our solution gives the user the freedom to
use the application at any time and in any space, as it makes use of
AR without markers. Moreover, it can not only be used to preview
products, but also serves as an automatic sales guide or advisor that
helps the user to find the right product according to their needs.
To carry out the experiment, surveys were conducted with users
who had already installed and interacted with our solution. As a
result of these surveys, it was found that users could use the appli-
cation without impediments and without the need to read a user
manual or instruction manual. Furthermore, they understand the
Ricardo Alfredo Aparicio et al.
purpose of using AR. On the other hand, the margin of error in the
measurement and the delay in visualising the objects with AR did
not mean discomfort for the user. Finally, all users agreed that they
would purchase products using the implemented functionalities.
As a future plan to improve the application, we could test other
technologies that are compatible with more devices, our solution is
limited to certain versions of Android, which could result in other
users not being able to use it. On the other hand, we could imple-
ment the functionality to visualise and position several products at
the same time, so that users can visualise a complete environment
and make the purchase of several products at the same time.
REFERENCES
[1] Forbes. 2018. New Technologies That Will Change How Con-
sumers Shop In Store. (January 2018). Retrieved May 10, 2021 from
https://www.forbes.com/sites/michelleevans1/2018/01/18/new-technologies-
that-will-change-how-consumers-shop-in-store/?sh=311c8f735fc9
[2] ECommerceNews. 2020. Comercio electrónico: El 47% de peruanos realiza sus
compras de manera online en la nueva normalidad. (August 2020). Retrieved
May 10, 2021 from https://www.ecommercenews.pe/comercioelectronico/2020/
peruanos-compras-online.html
[3] Mastercard. 2020. Electronic Commerce and online services as an alter-
native for a society in the process of digitization. Retrieved May 11, 2021
from https://www.mastercard.com/news/latin-america/es/perspectivas/blog-
posts/blog-es/2020/marzo/el-comercio-electronico-y-los-servicios-en-linea-
como-alternativa-para-una-sociedad-en-proceso-de-digitalizacion/
[4] Intel. Virtual Reality vs. Augmented Reality vs. Mixed Reality. Retrieved May
10, 2021 from https://www.intel.com/content/www/us/en/tech-tips-and-tricks/
virtual-reality-vs-augmented-reality.html
[5] ARCore. 2021. Overview of ARCore and supported development environments
(November 2021). Retrieved May 10, 2021 from https://developers.google.com/ar/
discover
[6] Developers. 2020. Selecting Colors with the Palette API. (January 2020). Re-
trieved May 10, 2021 from https://developer.android.com/training/material/
palette-colors
[7] NN/g Nielsen Norman Group. 2020. 10 Usability Heuristics for User Interface
Design. (November 2020). Retrieved May 10, 2021 from https://www.nngroup.
com/articles/ten-usability-heuristics/
[8] Salin Boonbrahm, Poonpong Boonbrahm, and Charlee Kaewrat. 2020. The Use
of Marker-Based Augmented Reality in Space Measurement, Procedia Manufac-
turing. 337-343. https://doi.org/10.1016/j.promfg.2020.02.081.
[9] Bergquist Robin and Stenbeck Nicholas. 2018. Using Augmented Reality to Mea-
sure Vertical Surfaces (Dissertation). Retrieved from http://urn.kb.se/resolve?
urn=urn:nbn:se:liu:diva-148868
[10] Hyekyung Imottesjo, Liane Thuvander, Monica Billger, Peter Wallberg, Gustav
Bodell, Jaan-Henrik Kain, and Stig Anton Nielsen. 2020. Iterative Prototyping
of Urban CoBuilder: Tracking Methods and User Interface of an Outdoor Mo-
bile Augmented Reality Tool for Co-Designing. Multimodal Technologies and
Interaction 4, 2: 26. https://doi.org/10.3390/mti4020026
[11] Waraporn Viyanon, Thanadon Songsuittipong, Phattarika Piyapaisarn, and Suwa-
nun Sudchid. 2017. AR Furniture: Integrating Augmented Reality Technology
to Enhance Interior Design using Marker and Markerless tracking. In Proceed-
ings of the 2nd International Conference on Intelligent Information Processing
(IIP’17). Association for Computing Machinery, New York, NY, USA, Article 32,
1–7. https://doi.org/10.1145/3144789.3144825