Multimedia Tools and Applications

https://doi.org/10.1007/s11042-020-10148-5
1173: INTERACTION IN IMMERSIVE EXPERIENCES

User experience design for a smart-mirror-based

personalized training system

Hye Sun Park1 · Gun A. Lee2 · Byung-Kuk Seo1 · Mark Billinghurst2

Received: 1 May 2020 / Revised: 16 October 2020 / Accepted: 23 October 2020 /

© Springer Science+Business Media, LLC, part of Springer Nature 2020

Abstract
This paper describes the user experience (UX) design for a smart-mirror-based personalized
training system which aims to help people live a healthy life. A number of researchers and
companies have developed fitness systems that use a virtual coach which shows the user
with actions they should perform. However such systems can be difficult to accurately fol-
low the virtual guide’s motions and there are also limitations in the feedback provided to
inform users of their correct body posture. This is because most systems are designed for
users to simply watch and follow a character’s motions (poses) from a third person perspec-
tive. In our smart mirror-based system, users are able to follow the exercise-postures of a
virtual professional trainer shown in a first person viewpoint and receive coaching through a
real-time motion correction. This is based on a predefined database of the trainer’s postures
gained from motion-capture technology, and it is personalized to the user’s body 3D model
acquired through an instant one-time scanning process. In this paper, we report on the UX
design of our system, mainly focusing on understandable visualization, intuitive interaction,
attractive information representation and easily acceptable user scenarios. Through a series
of user studies, we analyze and discuss user friendliness, information comprehension, and
user satisfaction as they relate to our design. In addition, we also assess the similarity and
effectiveness of the proposed system compared to traditional personalized training (PT) at
a gym. Based on the implications, we discuss future research directions for improving the
user experience of the smart-mirror-based PT system.

Keywords User experience (UX) design · Personalized training system · Mirror

visualization · User centered interaction · UX design guideline · Smart mirror

 Hye Sun Park

hspark78@etri.re.kr

Extended author information available on the last page of the article.

 Multimedia Tools and Applications

1 Introduction

People want to live a healthy life full of beauty, and to achieve this, many choose to
exercise with different types of activities. When engaged in fitness activities, it is com-
mon to learn how to exercise from an expert using a variety of methods (e.g., face-to-face
training sessions, or watching online videos). This is helpful in order to learn correct
postures or skills for exercise. Recently, people have increased their desire to practice cus-
tomized exercises suitable for them through the use of personal training (PT). Figure 1
shows a typical example of PT in a common gym. In this type of PT session, a trainer teaches
the proper posture and movement according to the trainee’s body condition, the purpose of
the exercise, and the overall health status of the trainee. If a trainee can repeatedly receive PT
from their preferred trainer at their preferred time, the exercise will likely be more effective
and the trainee will likely have a higher level of satisfaction. However, receiving train-
ing from a popular personal trainer is not always possible, and sometimes a trainee has to
wait for their trainer, or find another trainer who understands their personal circumstances.
Moreover, private lessons are very expensive. Accordingly, most people cannot utilize PT
due to the cost or they are unable to exercise at a time convenient for them. On the other
hand, when people exercise on their own, they typically lack proper guidance. To increase
the effect of an exercise, each action should be performed with a perfect posture. In the
worst-case scenario, incorrect actions or poses can be counterproductive or can even cause
injury [26].
Recently, combinations of technologies in various fields, such as advanced motion sen-
sors, wearable devices, wireless sensor networks, and AI technology have been used to
make traditional environments more intelligent. With the development of these technolo-
gies, fitness systems are also become smarter and offering a wider variety of options [33,
41]. Such smart fitness services are usually developed as mobile or game applications that
are designed such that users simply watch and follow a character’s motions (poses) from
videos or animations [23, 24, 29]. However, with these types of fitness services, it can be
difficult to follow professional motions accurately because the user must follow the pos-
tures from a third-person perspective. There may also be limitations with regard to feedback
to correct the posture, such as showing one-sided simple shapes of the user motion [23], or
providing simple vibration feedback to the user [24].
With the advancement of Internet of Things (IoT) technology, objects frequently encoun-
tered in everyday life have become intelligent. For example, smart mirrors have been
actively developed and applied to intelligent health systems, including fitness systems [2,

Fig. 1 Typical scenes of fitness personal training in a gym

Multimedia Tools and Applications

3, 6, 18, 20, 30]. Recent smart-mirror-based fitness systems can show videos or animations
of an expert’s actions to be viewed and followed from the user’s first-person perspective. In
this case, the users can see themselves reflected in the mirror [5, 10, 34]. Such smart mir-
ror systems are considered to make it easy for the user to understand the expert’s motions
compared to other methods. Despite such an advantage, there are still persistent com-
plications, such as checking and providing feedback about whether the user’s posture is
correct.
To address this issue, we applied user experience (UX) design methodologies to develop
a personalized immersive trainer for a smart PT system referred to here as the Per-
sonalized Smart Mirror Trainer. In order to realize a mirror-based intelligent service
beyond the use of typical mirrors for viewing oneself, the user’s appearance reflected
in the mirror and the information provided to the user must be displayed at the same
time. Accordingly, a suitable interface that fits the purpose and scope of the service
is needed. Therefore, the following two UX design approaches were required: a visu-
alization method by which the system presents the provided information to the user,
and an interaction method by which the user handles the provided information on the
system.
In this paper, we discuss the UX design approach used here to develop a smart-mirror-
based PT service. To achieve this, we first extract the user’s precise motion based on a
vision sensor embedded in a large display system using an actual translucent mirror. Next,
we provide customized exercise activities to the user based on the user’s status information.
We designed a service that enables a user to feel as though a trainer is with the user and
checks the user’s posture to provide feedback on correct postures. This is done by the sys-
tem analyzing the user’s motion information in real-time, and by visualizing digital guide
information overlaid onto the optical reflection of the real user’s body (this is referred to as
Apart yet Together schema in this paper).
To demonstrate that the design is suitable for a smart PT system service, we conducted
a set of user studies on both the visualization method, and an intuitive gesture interac-
tion method, which allows the user to operate the fitness system most conveniently. We
analyzed the results and discussed usability and user satisfaction issues of the mirror PT
system developed based on the proposed UX design approach. The results indicated that our
mirror PT system with real time feedback and intuitive information visualization can help
users gain an effect equivalent to that of an actual trainer in a gym by instantly informing
them of their exercise posture by means of intuitive information visualization. The results
also indicated that the system is easily controlled through interactive hand gestures that
are defined based on gesture classification/clustering techniques determined from a user
study.
The main novel contributions of this work are:
• An eligible UX design for a mirror-based smart PT system: 1) a novel schema, Apart
yet Together with a customized virtual trainer who corrects postures via real-time visu-
alization based on the user’s own body fitting information, and 2) an intuitive user
interface (UI) and interaction based on a fluent UX scenario that is easy to understand
and use.
• Findings from a user study, evaluating the impact of the UX design approach to visu-
alization, interaction, the UI design and the user scenarios for smart-mirror-based PT
system.
• Discussions on the research implications and UX design guidelines for future mirror
based smart PT systems.
 Multimedia Tools and Applications

The remaining parts of this paper are organized as follows. Section 2 describes works
related to our approach. Section 3 and the sub-sections therein detail our UX design and
Section 4 evaluates the UX design and its application. Section 5 discusses the overall results,
and the conclusions are the presented in Section 6.

2 Related works

2.1 Fitness applications & smart mirrors

Several fitness programs that combine 3D motion and computer graphics technologies
have been developed as game applications. Since IoT technology is rapidly developing and
being applied to health applications, many researchers have also proposed fitness systems
combining IoT with machine learning technologies [25, 41] (Fig. 2).
A typical example of fitness game application is the Xbox Nike Training System using
Kinect technology [23]. This recognizes the user’s motion with a Kinect sensor on an Xbox
360 game console and uses it as input into a fitness program. Based on an analysis of
the user’s movements, the system delivers a personalized program for achieving various
exercise goals. The system is also able to record the history of the user’s movement, and can
capture and measure the user’s poses. As the system adds game elements, users can exercise
in a more entertaining manner while making progress toward their goal.
The Xbox Nike Training program creates 3D shape-based avatars of users from the sen-
sor by recognizing the user’s motions. The avatar displayed on the screen moves according
to the user’s motion, and the user can exercise by following the pose of a virtual trainer char-
acter. This does not measure the exact 3D pose of the user, but instead it assesses the user’s
pose as an overall shape generated based on the depth sensor image. Such motion game-
based fitness programs can induce users to exercise more enjoyably. However, because these
systems emphasize game elements rather than providing personalized training guide infor-
mation, they are not suitable for use as a steady healthcare service. Users who have tried
these systems want a system that can measure and analyze their body data in more details
to provide proper information to them. This is the goal of the study.
The main difference between the proposed system and conventional motion game-based
fitness systems is in how the user’s pose and motions are shown to the user. Most motion
game-based fitness applications use virtual avatars (either a 3D character or a video of the

Fig. 2 Examples of smart systems for a healthy life: (left) Fitness video game [23], (Middle) Mobile app.
[41], (right) Smart mirror reflecting health [20]
Multimedia Tools and Applications

user) as a representation of the user. For example, the Xbox Nike Training program creates
3D shape-based avatars of users based on information received by its motion-tracking depth
imaging sensor. The avatar displayed on the screen moves according to the user’s motions,
and the user can exercise by following the poses demonstrated by a virtual trainer character.
However, this approach does not measure the exact 3D pose of the user. Instead, it assesses
the user’s pose as an overall shape generated based on the depth sensor image.
While not for sports training, there have also been prior investigations of using motion
tracking and visualizing virtual avatars for motion training for those interested in ballet
dancing [19, 35] and playing a stringed instrument [22]. In comparison, our system uses a
‘mirror display’ that allows users to watch their own body reflected in the mirror. Allowing
users to observe their own body motions and postures gives much more details and accuracy,
hence mirrors are widely utilized as a standard in gyms and physical training centers.
Mirror displays can be developed by either a physical half-mirror backed by a display or
a virtual mirror consisting of a camera and display pair. Such displays have been used in
various applications, such as to show virtual clothing [2, 7, 12, 32], virtual shoes [8, 21], for
interactive entertainment [37] or to provide personalized medicine [20]. Most of these prior
works focused on visualization of virtual objects on mirror displays, but not on applying
them to physical training. The YouMove system [1] uses a virtual mirror setup for a generic
motion training application and is somewhat similar to our research, though it does not
include features for the managing and guiding of physical exercise.
From prior research and development, we can identify advances in motion tracking tech-
nology, the requirements of PT systems (e.g., accurate motion guidance, feedback about
incorrect postures), and the requirements when applying smart mirrors to PT systems con-
sidering their unique features (e.g., the user’s body physically reflected and overlaid on
the display). There are especially few investigations of visualization methods that show an
expert’s motion information on a smart mirror for PT applications. PT is not merely for fun
or excitement through exercise but requires accurate motion to be effective. Therefore, it is
important to investigate visualization methods that help participants understand an expert’s
motions clearly and intuitively.

2.2 Intuitive interaction for smart information systems

The target end-users of smart information systems are members of the general public new
to exercise, hence, it is crucial to provide an intuitive and easy-to-use UI. Such a UI design
should meet the requirements of easy learnability, effective use, easy recall of usage, and
enjoyableness from the user’s perspective. To achieve an intuitive user interface, many
researchers have explored the use of speech, gesture, and even behavior recognition tech-
nologies [13, 15, 28] (Fig. 3). In our work, we focus on investigating a hand gesture
recognition for interacting with the smart mirror system.
Kurtenbach et al. defined a gesture as a motion of the body that contains information
[14]. Therefore, to deliver meaningful gestures, all gestures used for human computer inter-
action must be defined such that the gestures are easily understood. There are two main
challenges when defining gestures. First, a gesture definition must not be too specific or too
broad. When a definition is too specific, it has a higher risk of becoming obsolete as UI tech-
nologies change over time. On the other hand, if it is too broad, it may become irrelevant
in other use cases. Another challenge is that gestures should allow users easily to learn and
remember them, or even guess their meaning. Moreover, these conventions must be based
not on culturally determined rules but rather on those determined by technology. For exam-
ple, we know how to tap or flick on a smartphone touch screen not because these interactions
 Multimedia Tools and Applications

Fig. 3 Hand gesture-based interaction: Natural multimodal interactions with a large screen [13]

are cultural conventions, but rather because they are natural cross-cultural user interface
conventions [38]. When defining gestures, these issues should be properly addressed.
To overcome these challenges, researchers have used a user-participatory design pro-
cess for table-top computing [40] and Augmented Reality (AR) [27] interfaces, and in our
research we apply these advances to the mirror display-based smart PT system. We catego-
rized and clustered various gestures defined by and collected from potential end users with
various background variables, including culture, age, and gender. We then selected the most
appropriate gestures for our system based on the clusters formed.

3 UX design

We designed the UX for a smart-mirror-based PT system focusing on personalized training

visualization, user-centered interaction with a smart mirror display, information presentation
based on consistency, and an easily acceptable user scenario. In the following sub-sections,
we describe the design and methodology of each factor in detail.

3.1 Personalized training visualization

In a smart-mirror-based PT system, visual feedback is an important feature to ensure that

the user can easily understand and follow the exercises without a trainer’s coaching. To
have an experience comparable to receiving PT in an actual gym, a user requires effective
visual feedback to as readily to correct his or her own posture in real time. The visual
feedback should also inform the user of accurate postures according to the individual body
characteristics, similar to an actual real trainer instructs the user’s movements.
Multimedia Tools and Applications

3.1.1 Design methodology

For effective personalized visualization, we adopt the aforementioned Apart yet Together
schema, which means that a user (trainee) can feel as though they are together with a trainer
while exercising. This is done by providing correct postures in real-time through visualizing
the user’s own 3D body avatar in the mirror display. Previous training systems using mirror
displays [3, 5, 6, 10, 34] provided a video or animation of a trainer in the middle of the
mirror, typically on a smaller scale. This, however, involves the limitation of split attention,
as the user must watch and pay attention to both the video of the trainer and their reflection
in the mirror.
As shown in Fig. 4, our schema visualizes the trainer’s poses in the body-contour of
the user’s personalized life-size avatar. Therefore, the user can see the visual guides of the
poses together with his or her own body reflected in the mirror, allowing the user intuitively
to recognize the mismatch between them as they exercise. Moreover, the system pauses
the visual guide when the user’s pose is inconsistent with the visualized pose, providing
blinking feedback until the user performs it correctly. This is similar to an actual trainer who
pauses their training and points out incorrect postures to ensure that the trainee follows the
motion correctly.

3.1.2 Implementation

The Apart yet Together schema is applied to both offline and online steps. In the offline
step, two standard human-body 3D models with controllable skeletons (with 100 joints) are
created based on a man and a woman’s bodies, aged in their 30s. To link each standard model
with a trainer’s body motions, a professional trainer’s body motion data during 20 exercises
is initially acquired using 360-degree whole-body motion-capture equipment. This is done
with an angular motion error of less than one degree. A skeleton is then generated for each

Fig. 4 Basic concept of our Apart yet Together schema

 Multimedia Tools and Applications

motion data, enabling each standard model to be animated with the trainer’s motions in the
online step.
The standard human-body 3D models and their contours does not perfectly match to
those of the actual users, making it difficult for users intuitively to perceive whether their
poses are correct or not. Therefore, in the online step a standard human-body 3D model
chosen by the user is adapted to the user’s personalized avatar using live measurements
from the user’s actual body. To capture the measurements, the user’s frontal T-posture is
instantaneously captured by a Kinect camera attached to the mirror system, and its body
information is obtained on the basis of lengths and circumferences of 46 body parts (six
lengths of the height, shoulder height, neck height, shoulder width, arm length, leg length;
40 circumferences of 12 parts of the body consisting of seven parts of each arm, seven parts
of each leg), including the joint structure. Given the user’s personalized avatar, its body-
contour is finally extracted and visualized to guide the user to follow it. In our system, this
step of avatar personalization is performed online within a few seconds (five seconds on
average) before the user starts exercising.
During the exercise, the user can move to match his or her reflection in the mirror to the
visualized contour while seeing both types of information in the mirror at the same time. If
the user does not create the correct posture for a certain period of time, the animated contour
is paused and blinks to provide feedback to the user. The correctness of the user’s pose is
evaluated and determined by analyzing the matching error between the user’s body shape
and the personalized avatar’s contour. When the user repeatedly fails to correct the posture,
the system recommends and visualizes the user in an alternative posture that can give the
same exercise effect. The overall workflow of these steps is shown in Fig. 5.
Our system operates identically to the conventional PT. The system sets the basic pos-
tures for exercise in order. Each posture is repeatedly performed based on the exercise
effectiveness. If the user is unable to follow the posture well, it substitutes another exercise-
posture with an identical exercise effect, or move on to the next posture. The set of the basic
posture, its substitution, and the next posture are pre-defined based on the offline processing
of the motion data from a professional trainer.

3.2 User-centered interaction with a smart mirror display

The smart-mirror-based PT system needs to support natural user interaction so that anyone
can immediately use it as if interacting with a trainer in an actual gym. To enable this, we
designed user-friendly and intuitive gestures by investigating the most commonly accepted
gestures through a user study. We also applied adaptive UX principles to support natural
interactions that are customized to the user.

3.2.1 User-friendly and intuitive gestures

We followed the end-user participatory design procedure outlined in Fig. 6. The design pro-
cess started with a user study to define the intuitive hand or body gestures for interacting
with the smart mirror display. Following the structure of a guessability study [16, 39], the
participants defined and proposed user-friendly and intuitive gestures for the given tasks.
To do this, the participants stood in front of a large screen mirror display and they watched
an animated user interface describing the interactive task to be done through gesture inter-
actions. The users are then asked to define the hand or body gestures to achieve the given
task, and to explain and act out the defined gesture while being recorded. By analyzing the
Multimedia Tools and Applications

collected user-defined gesture motions from this user study, we identified the types of ges-
tures that have high consensus among the participants. The user motion recording database
was then analyzed to model the gesture motions according to the gesture types. Finally, the
gesture models and motion analysis were used for gesture recognition and validation.
In our study, we analyzed the gesture motions collected from 16 participants (9 male, 7
female) with various cultural backgrounds, after which we identified intuitive gestures for
five interactive tasks (pointing, selecting, dragging, scrolling, dichotomous options) by mea-
suring the degree of consensus among the participants. Here, the consensus was measured
based on the agreement score, denoted as A, calculated as the squared sum of the proportion
of each category [16, 39]. A value of A higher than 0.5 indicates a good consensus among
the participants.
Figure 7 shows a summary of the results. The results indicated that for the given tasks,
there are types of gestures with high levels of agreement upon among participants, including
a hovering gesture for pointing and a swiping gesture for scrolling tasks. However, there

Fig. 5 Overall workflow for personalized training visualization

Fig. 6 User participatory design procedure for gestures

 Multimedia Tools and Applications

Fig. 7 Results of the user study to define common gestures for given tasks (pointing, selecting, dragging,
scrolling, dichotomous options; Here, the A indicates the user’s agreement score)

were also tasks in which the level of agreement was insufficient to define a common gesture
for the given task. For instance, the participants were divided into groups demonstrating
tapping or grabbing gestures as triggering methods for selecting and dragging tasks.
To overcome this problem, we propose the support of more than one type of gesture,
and allowing users to select any of the major gestures for tasks. For example, for the task
of choosing from several options, we found that the participants used both deictic and
metaphoric gestures to indicate one of the options provided. These results suggested that
metaphoric gestures can be supported as shortcuts (much like keyboard shortcuts in a 2D
menu system) to indicate a chosen item instead of using deictic gestures to accurately point
to an item. Once the gestures to support in the system were identified, we further analyzed
the motion data of the selected gestures to identify distinctive features of the motion (for
example, speed and acceleration vectors) and defined a model for each gesture based on a
finite state machine. We note that while the result showed tapping being most common, our
additional testing with the optical mirror showed that users prefer the wait gesture. Hence,
we reflected this in the final design.

3.2.2 User-adaptive interaction

In addition to defining user-friendly and intuitive gestures, we also designed user-adaptive

interactions. We envision that systems should operate by adaptively changing the location
and size of the UI elements and arranging them differently according to the user information.
In addition, the system could collect and estimate user information such as the user’s body
position and size or the user’s profile, and adapt the content layout on the display screen
based on this information.
Based on this notion, we designed the following three principles for adaptive UX. First,
the smart mirror system estimates the user’s dominant hand for automatic positioning of
Multimedia Tools and Applications

Fig. 8 User-adaptive interaction: a when the user raises the left hand first as the dominant hand, the control
menu is displayed on the left side, and vice versa for right-handed users, b based on the user’s body measure-
ment, the icon layout changes for (left) a shorter user and (right) a taller user, c when the user needs to stretch
or bend down to reach a distant object (left), the user can remain in a comfortable pose with a hand-extending
interaction (right)

the information or UI elements (e.g., menu buttons) with which the user must interact.
Second, the system measures the user’s physical attributes (e.g., height and arm length), and
automatically controls the size, location, and spacing of the presented information according
to these measurements for greater visual convenience. Third, if the user must select the
information located in a position that is difficult to reach with hand gestures (such as the
edges, top, or bottom of the screen), the system enables the user to make selection without
having to reach the position physically based on estimating the user’s arm motion.
In Fig. 8, we describe how the UI layout is presented adaptively to the user with regard to
the three principles described above. Figure 8a shows how the layout of the control buttons
on the screen changes depending on the dominant hand of the user. Depending on the hand
initially raised by the user, the smart mirror system determines whether the user is right- or
left-handed. Based on this information, the system adaptively places the control menu UI
to make it easier for the user to control it with their dominant hand. As shown in Fig. 8b,
if a taller person operates the system, the menu items are widely distributed throughout the
mirror display. On the other hand, if a shorter person operates the system, the menu items are
placed closer to the user’s body. To achieve this, the system uses the positional information
of each body part, as detected and tracked by the motion sensor.
 Multimedia Tools and Applications

While the system can arrange the UI elements and place them within reach of the user,
there are certain cases in which some of the UI elements need to be placed farther away.
Example, include situations in which, there are many options from which to choose, or
when accessing content placed on the periphery of the screen, such as an advertisement.
In such cases, the users must stretch further to select those buttons that are out of normal
reach. This may result in poor usability and physical discomfort, as shown on the left side
of Fig. 8c. To overcome this type of limitation, we designed a hand-extending interaction
method by which the user’s hand extends when they stretch farther to reach a distant object,
as shown on the right side of Fig. 8c. This method is based on observations from the user
study described earlier in this section, in which we found that some participants when using
a hovering gesture did not bother to match the positions exactly on the screen location, but
instead simply reached as far as they could. This approach resembles the Go-Go interaction
technique [31] in VR where the length of the user’s virtual hand is extended when trying to
reach a distant object.

3.3 Information presentation based on consistency

To enhance usability and ease of understanding of the smart-mirror-based PT system, it

required both an intuitive and attractive design of the UI elements, such as visual icons.
For this purpose, we designed an information presentation scheme which follows the con-
sistency principle and corresponding important attributes associated with UI elements,
familiarity and consistency [11].
Familiarity is an attribute that allows a user easily to access the system and acquire what
they need in a typical way. To achieve this, we designed visual icons that clearly represent
the meaning and function based on feedback from potential users. As shown in Fig. 9, the
icon for Return to the First Screen has the shape of a home, commonly adopted in web and
mobile interfaces. The Exit icon has the form of an emergency exit, while the Setting icon
resembles a cogwheel, all common in mobile interfaces. The Exercise icon has a barbell
shape which is familiar and easily recognized by those who undertake exercise and PT. In
addition, the motion count, time and speed features are displayed along with intuitive icons
(e.g., timer, hourglass) to help users gain a better understanding of each function.
Next, consistency is an attribute that organizes the commands, menus, and UI elements
in a consistent style so that users will not be confused with contradictory information when
using the system. In this regard, we designed the size, color and style of icons to be consis-
tent throughout the system. The shapes and sizes of the icons are such that they are not only
visible when the user is approximately 1 m away from the system, but are also large enough
such that the user’s hand gestures can be used to select them. All visual icons were designed
as circular buttons with a diameter of about 12 cm. The outline of each icon was displayed

Fig. 9 Examples of the designed icons for a smart mirror based PT system: (from left to right) Return to the
First Screen, Exit, Setting, Exercise, Time Remaining, Time Count
Multimedia Tools and Applications

in white, a color which is noticeable on the mirror display, and the selection process is visu-
alized by means of a circle that is filled. Time-related icons (e.g., count, time, speed) were
also displayed with a horizontal progressive bar next to each icon.

3.4 User scenario

We designed an easily acceptable user scenario based on the previously described person-
alized visualization schema, user-centered interaction, and attractive information presenta-
tion. An ideal system scenario should improve usability, and one of the ways to increase
usability is to optimize the user’s task load. Based on research by Tullis and Alber [36],
the user’s task load can be optimized when the system fully meets the four requirements
of learnability, effectiveness, efficiency, and memorability. Therefore, in this section, we
describe how the proposed user scenario based on a UX design meets these requirements in
the user scenario.
First, the proposed UX-design-based user scenario establishes good learnability by pro-
viding users with intuitive PT programs, rather than having users learn them through their

Fig. 10 The proposed UX design based user scenario of the Personalized Smart Mirror Trainer
 Multimedia Tools and Applications

own efforts. To enable users to operate the system intuitively and conveniently, we defined
the gestures by which they will interact with the system through a user study (as described in
Section 3.2). As shown in step3 (Body Measurement) in Fig. 10, we designed the user sce-
nario to visualize the simple frontal T-posture as a body pose contour on the mirror display
so that the user can follow it immediately (design methodology in Section 3.1).
Next, the visualized training posture is occasionally maintained in certain cases to
improve the exercise effect. We provided feedback so that the proposed visualization
schema-based training posture could be maintained for a reasonable period of time, deter-
mined based on the complexity of the pose. When the user follows the pose guide shown
on the service as the correct pose, the system checks the time taken by the user to make
the correct pose and evaluates if it is within the time limit calculated based on Fitts’ law
[9] to decide whether the user successfully performs or not. This approach enhances the
effectiveness of our system.
Finally, we configured the relevant icons (such as the number of exercises, and when
to conduct a movement) so that they are selectable, viewable, and accessible as quickly
and easily as possible. In the user scenario, we also adopted an adaptive task flow.
When a user starts the PT service, the system identifies whether the user is a first-time
or returning user (Step 1 (Member Check) in Fig. 10), and for returning users the sys-
tem can use previous measurement data and skip the following processes (Step 2 (User
Information) and Step 3 (Body Measurement) in Fig. 10). As shown in Step 4 (Mirror
PT Process) in Fig. 10, our system initially displays the trainer’s correct posture motion
video. Then, it shows the posture of the trainer with the contour of the user’s personalized
avatar and provides feedback if it is incorrect, similar to what a trainer does in an actual
gym.
Moreover, the system visualizes the professional trainer’s body motion through the ani-
mated user’s personalized avatar, which is created based on the user’s body measurements.
In this way, users can easily position themselves to fit within the presented body outline of
their own. As such, we designed the user scenario so that users can easily operate the smart-
mirror-based PT system based on their own experience, understand the process, and quickly
follow the process, thus satisfying both the efficiency and memorability requirements of the
system.

4 Evaluation and analysis

In this section, we report on evaluation and analysis of our UX design and its application
to smart-mirror-based PT system, referred to here as the Personalized Smart Mirror Trainer.
As shown in Fig. 11, the applied system used a large display (about 70 inch) covered with a
half-silvered mirror to present digital information overlaid onto the user’s physical environ-
ment reflected in the mirror. As the user is standing in front of the display, using a motion
recognition sensor, the system recognizes the user’s hand gestures for interaction and cap-
tures the user’s body shape and motion to measure the degree of fitness and to provide
training guide information. The training guide information is provided based on how well
the user’s exercise motions match the professional trainer’s motions stored in the database.
Here, an exercise is defined as a sequence of body postures while the exercise-test refers
to the process of matching (comparing) the user’s body posture sequence to that of a
professional trainer.
Multimedia Tools and Applications

4.1 Evaluation of user-centered interaction

We evaluated the user satisfaction of the proposed gesture-based UI (described in

Section 3.2.1) and user-adaptive interaction (described in Section 3.2.2), in order to analyze
whether the mirror system control method is appropriate. We also measured the task com-
pletion time to evaluate the efficiency and user satisfaction in relation to the dominant hand
routine in the user-adaptive interaction design.
In this user study, the participants tested two types of mirror systems: one with our UX
design and the other without (the baseline condition). The experiment used a within-subject
design, in which the order of the experimental conditions was counterbalanced to prevent a
carryover effect. A total of 30 participants (20 male and 10 female) with an average age of
33.2 years participated in the study.
The study focused on assessing our user-centered interaction design features of the smart-
mirror-based PT system. The first feature (F1) is the gesture-based UI to evaluate the level
of user satisfaction in terms of ease of system operation. The second feature (F2) is the
user-adaptive interaction to evaluate user satisfaction for both icon layout changes based on
the user’s body measurement and the hand-extending interaction. Focusing on each of the
design features, the participants assessed the two types of mirror systems through a ques-
tionnaire designed to measure user satisfaction based on the system usability scale (SUS)
[4].

4.1.1 User satisfaction

We conducted an experiment focusing on user satisfaction with F1 and F2 on the mirror

systems either with or without our user-centered interaction design. In addition to analyzing
the aggregated SUS score (ranging from 0 to 100), we classified the ratings into three levels
of satisfaction based on [40]: dissatisfaction (poor or worse), neutral (ok), and satisfaction
(good or better) scores.

Fig. 11 Examples of the Personalized Smart Mirror Trainer

 Multimedia Tools and Applications

Table 1 Results of a user satisfaction assessment measured with SUS

Feature Type N Mean (SD) Min–Max Dis Neu Sat

N % N % N %

F1 w/o 30 44.60 (11.40) 27.5–75.0 14 47 14 47 2 7

F1 w/ 30 55.20 (12.97) 35.0–92.5 5 17 16 53 9 30
F2 w/o 30 44.20 (11.00) 27.5–72.5 16 53 12 40 2 7
F2 w/ 30 57.40 (11.40) 37.5–92.5 3 10 18 60 9 30

Type: with or without the proposed UX design, N: number of participants

Dis: dissatisfaction, Neu: neutral, Sat: satisfaction

The results are summarized in Table 1. The experimental results show that both F1 and
F2 improved the overall user satisfaction (Mean (SD), Min–Max in Table 1) when applying
the proposed user-centered interaction design, compared to the system without our design.
A Wilcoxon signed rank test revealed that the participants rated the system with our user-
centered interaction design significantly higher for both F1 (Z = −3.49, p = 0.0005)
and F2 (Z = −4.29, p < 0.0001). In particular, four times more participants reported
that they were satisfied regarding their experience with our user-centered interaction design
compared to the baseline condition, an increment from 7% to 30% for both F1 and F2. The
number of participants who showed dissatisfaction also decreased from 47% to 17% for
F1, and from 53% to 10% for F2. The difference between the conditions in terms of the
distribution of ratings was statistically significant according to Chi-square tests for both F1
(χ 2 (2) = 8.85, p = 0.012, Cramer’s V = 0.3841) and F2 (χ 2 (2) = 14.55, p = 0.0007,
Cramer’s V = 0.4924).
Overall, the results indicated that both of our user-centered interaction design features
(user friendly and intuitive gestures, and user-adaptive interaction) had a positive impact on
the user satisfaction as measured through a subjective questionnaire.

4.1.2 Task completion time

For a further evaluation of the benefits of our user-adaptive interaction (F2), we also inves-
tigated how the task completion time was influenced by our interaction design. To highlight
the benefit of presenting the UI on the side of the user’s dominant hand, we conducted
an experiment comparing the task completion time between the system types in which the
user had to select an icon either on their dominant hand side or on the opposite side. The
participants were asked to stand in front of the screen and to provide input to the system
using the same user information by selecting an icon shown on either their left or right
side.
The system presented the correct option on the participant’s dominant hand side for
one question, and presented it on the opposite side for the next question. The order of the
correct option being on the dominant hand side was counterbalanced. For example, for a
right-handed female participant in their 20s, the system initially asked the participant to
choose her age group, with the correct option (20s) placed on the right-hand side and the
other option (for example, 30s) placed on the opposite side. Next, when asking about her
gender, the system showed the correct option (female) on the left side, which is the oppo-
site side of her dominant hand. The hypothesis of this experiment was that the participants
Multimedia Tools and Applications

Fig. 12 Results of the task completion time (in seconds)

would take less time to select the option on their dominant hand side, hence the adap-
tive UI design of changing the layout according to the user’s dominant hand would be
supported.
The results, as shown in Fig. 12, indicated that the participants were able to select the icon
more quickly when it was on the same side as their dominant hand. The difference in the
task completion time between the conditions was 0.6 seconds on average. Although a paired
t-test found that this difference is not statistically significant given the variance (t (29) =
−1.64, p = 0.11), we found a mild trend according to which 83% of the participants took
less time to select the icon when placed on their dominant hand side. Five of the partici-
pants (subjects 6, 13, 15, 19, and 20), as highlighted in yellow in Fig. 12, took less time to
select an icon on the opposite side of their dominant hand. These cases pertained to those
using both hands, and they selected the icon using the hand on the side where the icon was
presented.

4.2 Evaluation of the applied PT system

In a follow-up study, we compared the Personalized Smart Mirror Trainer to PT in the field
with actual experts (trainers, trainees), in order to verify how similar the UX of our system
is to the actual PT, and to determine its advantages and disadvantages. In order to focus on
assessing the UX design based on the visualization schema in our system, we conducted a
user survey based on a video demonstration to reach out to actual potential end-users such
as trainees or trainers. The survey mostly focused on asking users how easy the system was
to understand and about the ease in following the instructions and user interfaces visualized
in the system.

Fig. 13 Results from the rating questions on the user survey

 Multimedia Tools and Applications

Fifty potential end-users (22 Male, 28 Female) participated in the survey, most of whom
were in their 40s (62%) while the rest were mostly in their 30s (16%) or 20s (14%). The
majority of the participants (62%) had previous experience with receiving personal training,
mostly less than 5 years. The majority of the participants (68%, n = 34) also had never
before seen a smart-mirror-based PT system. All of these who had seen or used such a
system before (n = 16) mentioned that the proposed system was easy enough to follow by
themselves or with little help from other users.
Figure 13 shows the results of the user survey. A comprehensive analysis of these replies
is given below.

Q1. Similar to traditional PT

When asked to rate how similar our system is to traditional PT programs that involve
trainers on a 11-point rating scale (0: very low similarity—10: very high similarity), the
participants reported that the training process is sufficiently similar, with the most of them
(64%) rating it as significantly higher than the mid-point value of 5 (Median = 7, IQR = 5–
8) according to a One-sample Wilcoxon Signed Rank test (Z = −4.165, p < .001). When
asked to provide the reasoning behind their rating, most people noted that the visualiza-
tion of the pose guide information with personalized body measurements, and the following
visualized pose were similar to those in traditional PT. However, they also expressed
concerns about the lack of detailed pose corrections provided by the trainer.
Regarding the user’s body measurement step before training, most of the participants felt
that the time taken for measurement (15 seconds) was reasonable (72%; χ 2 (2) = 33.76,
p < .001), and that the measuring posture shown on the mirror was comfortable (96%;
χ 2 (1) = 40.5, p < .001). When the participants were asked whether it was easy or difficult
to follow the postures, most of them (94%) reported that they were easy to follow (χ 2 (1) =
36.98, p < .001).

Q2. Easy to understand visualization

After the measurement, the system showed the training postures on the mirror screen,
visualized with the outline of the user’s personalized avatar. The participants were asked
how easy it was to understand the training pose with the provided visualization, rating this
on an 11-point scale (0: very hard—10: very easy). Nearly three quarters (74%) of the
participants reported that it was easy to understand rating this as significantly higher than the
mid-point value (Median = 7, IQR = 5–9, Z = −5.272, p < .001). When the participants
were asked about their reasoning, they mostly reported that the visualization was easy and
intuitive to understand. However, they also noted that it would be better to show a side view
of certain postures that were difficult to understand when viewing them only as a body
outline from the front.

Q3. Easy to follow posture

Regarding the visual feedback provided when the user was not following the visualized
posture correctly, the participants mostly perceived it as being useful to correct their pos-
tures, with most of them (82%) rating this as significantly higher (Z = −5.443, p < .001)
than the mid-point on an 11-point rating scale (0: very difficult—11: very easy; Median =
8, IQR = 6.75–9). When asked about their reasoning, most of the participants mentioned
that it was easy and intuitive to notice which part of the body was in an incorrect pose.
However, understanding details about how to correct the pose was unclear in certain cases.
Multimedia Tools and Applications

When asked about the visualization and design of the icons used in the interface, the
participants were split with regard to how easy they were to understand. About half of the
participants (n = 23) reported that all of the icons were easy to understand and intuitive,
while other half (n = 27) reported that most of the icons were easy to understand while
some could be improved, although none of them stated that they were difficult to understand
(χ 2 (2) = 25.48, p < .001). For example, the icon designed to resemble fitness equipment
was perceived as difficult to recognize and thus in needing improvement.
When the participants were about what were the strengths and weaknesses of the smart
PT system, most of them reported that it was easy to exercise and follow the training posture
alone without help from an actual trainer. The participants especially valued the system as
being easily accessible without the restriction of having to meet a trainer at a designated
time and place. On the other hand, the participants also expressed concern about losing
motivation or lacking detailed guidance with regard to correcting postures of the type that a
trainer could provide through personal feedback.
The participants also provided suggestions about how to improve the system. Some of
the feedback included visualizing postures from different viewpoints (e.g., side view), pro-
viding detailed feedback on how to correct postures, adding a remote conferencing function
to connect with a remote trainer when needed, and supporting a social networking feature
between users to encourage each other. One participant suggested, “It would be nice to have
a video call with a trainer for confirmation.” Another participant stated, “It would be nice to
make a call to ask any questions.” Being able to talk to the trainer while exercising was also
considered to be useful.
In addition, we asked similar questions to a group of professional trainers (n = 5, 1
Male, 4 Female). Their feedback on the visualization and user interfaces was similar to
that from the trainees, perceiving the visualization as being easy to understand and easy to
follow. Overall, the trainers commented that the system provides a similar process to tradi-
tional PT, and that it may be useful to those who want to exercise on their own. However,
they mentioned that there was a lack of detailed feedback on how to correct postures and
motivate the trainees. Suggested improvements included the provision of detailed sched-
ules and goals for exercise, recommendations of appropriate exercises, and the provisions
of warnings about common mistakes to prevent the trainees from performing with incorrect
postures.

5 Discussion

Recently, many people have shown an increased levels of interest in health, and there is a
trend towards increased interest in self-home care. To apply a customized fitness service
to individual users, a current healthcare field, we designed an exercise system based on a
user’s personalized avatar that reflects expert’s 3D motion. To do this, we used a method
that extracted 3D motion by an expert, created a personalized avatar for each user’s body
size, and reflected the expert’s 3D motion in the form of the created avatar.
However, most other IT-based fitness systems currently developed have limitations in
how well they accurately score whether the user actually properly performs the exercise
or holds the posture. In order to resolve this issue, we focused on personalization and
researched an eligible UX design for a mirror-based-smart PT system that can increase the
efficiency of exercise effects as well as the usability of the system and ultimately prevent
false self-care.
 Multimedia Tools and Applications

Our UX design has a novel schema, termed Apart yet Together, with a customized virtual
trainer which corrects the user’s postures via real-time visualization based on the user’s own
body fitting information. The proposed interface is an intuitive-gesture based interaction
that supports a fluent UX scenario that is easy to understand. From a user study conducted
here, we evaluated the impact of the UX design approach on visualization, interaction, UI
design and user scenarios for the smart-mirror-based PT system. In this section, we discuss
the implications as the UX design guidelines for future mirror-based smart PT systems.
Overall, the results of our evaluations indicated that the users had satisfactory experi-
ences while using the smart mirror personalized training system. Our UX design appeared
to be effective in terms of visualizing pose contour fittings on the measured user avatar
and providing an intuitive gesture interface. Our design approach that considered the Smart
Mirror Personalized Trainer was constructive, as most of the feedback from participants
was positive on the topics of both usability and satisfaction when comparing the similarities
between our mirror PT system and the actual PT for potential users who are real trainees or
trainers who engage in practicing PT.
The results above arose because we carefully observed how trainers and trainees engage
in personal training in an actual gym and incorporated our findings into our UX design. In
an actual gym, most people want to exercise while checking their posture, and to do this they
stand in front of a mirror to exercise. Therefore, we assumed that the future PT system will
be designed as a mirror type of system. We then proposed a design method that considers
the provision of personalized service to the users to perform PT with such a mirror system
and that considers how users can conveniently operate the system without much effort.
To do this, we focused on two aspects: the characteristics of the mirror (i.e., reflection of
the user’s appearance) and the most important factor in movement being the correct posture.
We expressed the correct pose represented by the user’s body contour model which reflects
the user’s body size as measured through a simple 3D scan without requiring much effort
by the user, making it easy to understand and follow. In order to naturally operate the mirror
system while exercising, we designed the system to operate based on simple and intuitive
gestures without cumbersome movements. In addition, we intuitively designed the UI button
layout and the shapes of the buttons. Therefore, our user experience design for the Smart
Mirror Personalized Trainer can be used as an example UX design guide for future mirror
systems.
However, it is necessary to reinforce adaptive contour information based on user feed-
back. This can be done by visualizing the side contour of the user, in cases involving poses
where it is difficult to determine the correct pose from the front view (e.g., postures that
involve body parts overlapping when viewed from the front). There is also a need to extend
our system to support a wider range of users. Our system was designed with reference to
the common bodies of healthy adults who attend a fitness center. For body measurements
and postures out of this scope, such as for children or for elderly users, different standard
3D models may need to be generated, and the system size (height and width) as well as the
Kinect position may also need to be adjusted accordingly.

6 Conclusion

In this paper, we presented a novel UX design for smart-mirror- based PT systems and
described details of its methodology based on the following features: personalized training
visualization, user-centered interaction with a smart mirror, information presentation based
Multimedia Tools and Applications

on consistency, and an easily acceptable user scenario. We also evaluated and analyzed our
UX design and its applied PT system through a series of user studies and surveys. The
results showed that our approach improved overall user satisfaction and usability levels.
Based on feedback collected from the user studies, we are planning to develop and
improve the smart-mirror-based PT system further to support the capturing and visualization
of postures from additional viewpoints, as doing so may help users improve their accuracy
when they attempt to follow complicated poses. In addition, we are experimenting with 3D
visualization display technology [17] to overcome the limitations of 2D displays, so that
the posture visualization can be more accurately overlaid onto the user’s body as reflected
in the mirror. Another potential future investigation would be applying the methodology to
other applications such as physical therapy or dance motion training. We expect that these
future investigations will help to improve both user’s performance capabilities and their
satisfaction levels even further.

Acknowledgements This work was supported by Institute of Information & Communications Technology
Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.15501-14-1016, Instant
3D object based Join & Joy content technology supporting simultaneous participation of users in remote
places and enabling realistic experience and No.2017-0-01849, Development of Core Technology for Real-
Time Image Composition in Unstructured In-outdoor Environment). We thank all those who helped setting
up and running the user study. Special thanks should be given to Dr. Ho Won Kim, Dr. Kyu Sung Jo, Dr. Tae
Joon Kim and Mrs. Ki Nam Kim for support of 3D motion tracking and 3D modeling generation.

References

1. Anderson F, Grossman T, Matejka J, Fitzmaurice G (2013) Youmove: enhancing movement training

with an augmented reality mirror. In: Proceedings of the 26th annual ACM symposium on user interface
software and technology, pp 311–320
2. Beck M, Crié D (2018) I virtually try it... I want it! Virtual fitting room: a tool to increase on-line and
off-line exploratory behavior, patronage and purchase intentions. J Retail Consum Serv 40:279–286
3. Besserer D, Bäurle J, Nikic A, Honold F, Schüssel F, Weber M (2016) Fitmirror: a smart mirror for pos-
itive affect in everyday user morning routines. In: Proceedings of the workshop on multimodal analyses
enabling artificial agents in human-machine interaction, pp 48–55
4. Brooke J (1996) Sus: a quick and dirty usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClel-
land I (eds) Usability evaluation in industry, lecture notes in computer science. Taylor and Francis,
London, pp 189–194
5. CBS This Morning (2018) This smart mirror puts a personal trainer in your reflection. https://www.
youtube.com/watch?v=nSmTTZcpVGg. Accessed 2020
6. Colantonio S, Coppini G, Germanese D, Giorgi D, Magrini M, Marraccini P, Martinelli M, Morales MA,
Pascali MA, Raccichini G, Righi M, Salvetti O (2015) A smart mirror to promote a healthy lifestyle.
Biosyst Eng 138:33–43
7. Ehara J, Saito H (2006) Texture overlay for virtual clothing based on pca of silhouettes. In: Proceedings
of the 5th IEEE and ACM international symposium on mixed and augmented reality, pp 139–142
8. Eisert P, Fechteler P, Rurainsky J (2008) 3-d tracking of shoes for virtual mirror applications. In:
Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–6
9. Fitts PM (1954) The information capacity of the human motor system in controlling the amplitude of
movement. J Exp Psychol 47(6):381–391
10. Hunter College Center for Health Technology (2019) https://nychealthtech.org/mirror-the-newest-trend-
in-technolo gy-based-home-fitness/. Accessed 2020
11. Interface design rules, lecture hci, lecture-2 informatics, University of Edinburgh, 23rd (October 2007).
https://www.interaction-design.org/literature. Accessed 2020
12. Kinect for windows retail clothing scenario (2013) https://www.youtube.com/watch?v=Mr71jrkzWq8.
Accessed 2020
 Multimedia Tools and Applications

13. Kaiser E, Olwal A, McGee D, Benko H, Corradini A, Li X, Cohen P, Feiner S (2003) Mutual dis-
ambiguation of 3d multimodal interaction in augmented and virtual reality. In: Proceedings of the 5th
international conference on multimodal interfaces, pp 12–19
14. Kurtenbach G, Hulteen E (1990) Gestures in human-computer communications. Addison-Wesley
Longman Publishing Co. Inc, Reading, pp 309–317
15. Lee M, Billinghurst M, Baek W, Green R, Woo W (2013) A usability study of multimodal input in an
augmented reality environment. Virtual Real 17(4):293–305
16. Lee GA, Wong J, Park HS, Choi JS, Park CJ, Billinghurst M (2015) User defined gestures for augmented
virtual mirrors: a guessability study. In: Proceedings of the 33rd annual ACM conference extended
abstracts on human factors in computing systems, pp 959–964
17. Lee GA, Park HS, Billinghurst M (2019) Optical-reflection type 3d augmented reality mirrors. In:
Proceedings of the 25th ACM symposium on virtual reality software and technology, pp 1–2
18. Mango Mirror (2018) https://www.mangomirror.com. Accessed 2020
19. Marquardt Z, Beira JA, Em N, Paiva I, Kox S (2012) Super mirror: a kinect interface for ballet dancers.
In: Proceedings of the CHI ’12 extended abstracts on human factors in computing systems, pp 1619–1624
20. Miotto R, Danieletto M, Scelza JR, Kidd BA, Dudley JT (2018) Reflecting healtH: SMART Mirrors for
personalized medicine. NPJ Digit Med 1:62
21. Mottura S, Greci L, Travaini E, Viganò G, Sacco M (2007) Magicmirror & footglove: a new system for
the customized shoe try-on. In: The future of product development, pp 441–450
22. Ng KC, Weyde T, Larkin O, Neubarth K, Koerselman T, Ong B (2007) 3d augmented mirror: a multi-
modal interface for string instrument learning and teaching with gesture support. In: Proceedings of the
9th international conference on multimodal interfaces, pp 339–345
23. Nike+ Training (2012) https://news.nike.com/news/introducing-nike-kinect-training. Accessed 2020
24. Nintendo Wii Fit (2008) https://www.youtube.com/watch?v=-Taruqvk30E. Accessed 2020
25. PerfectGym (2018) Iot trends: how will they affect the future of gyms? https://www.perfectgym.com/en/
blog/business/iot-trends-how-will-they-affect-future-gyms. Accessed 2020
26. PhysioRoom (2018) Can poor posture make you more susceptible to injuries? https://www.physioroom.
com/info/poor-posture-susceptible-injuries/. Accessed 2020
27. Piumsomboon T, Clark A, Billinghurst M, Cockburn A (2013) User-defined gestures for augmented
reality. In: Proceedings of the CHI ’13 extended abstracts on human factors in computing systems,
pp 955–960
28. Piumsomboon T, Altimira D, Kim H, Clark A, Lee G, Billinghurst M (2014) Grasp-shell vs gesture-
speech: a comparison of direct and indirect natural interaction techniques in augmented reality. In:
Proceedings of the IEEE international symposium on mixed and augmented reality, pp 73–82
29. PlayStation®Move (2010) https://www.youtube.com/watch?v=7Wuvia1-Mw8. Accessed 2020
30. Poh MZ, McDuff D, Picard R (2011) A medical mirror for non-contact health monitoring. In:
Proceedings of ACM SIGGRAPH 2011 emerging technologies
31. Poupyrev I, Billinghurst M, Weghorst S, Ichikawa T (1996) The go-go interaction technique: non-linear
mapping for direct manipulation in vr. In: Proceedings of the 9th annual ACM symposium on user
interface software and technology, pp 79–80
32. Saakes D, Yeo HS, Noh ST, Han G, Woo W (2016) Mirror mirror: an on-body t-shirt design system. In:
Proceedings of the 2016 CHI conference on human factors in computing systems, pp 6058–6063
33. Thompson WR (2016) Worldwide survey of fitness trends for 2017. ACSM’s Health Fitness J 20(6):8–
17
34. Tonal (2018) Intelligent home gym. https://www.tonal.com/ Accessed 2020
35. Trajkova M, Ferati M (2015) Usability evaluation of kinect-based system for ballet movements. In:
Proceedings, Part II, of the 4th international conference on design, user experience, and usability: users
and interactions, vol 9187, pp 464–472
36. Tullis T, Albert B (2013) Measuring the user experience: collecting, analyzing, and presenting usability
metrics, 2nd edn. Elsevier Inc, Amsterdam
37. Vera L, Gimeno J, Coma I, Fernández M (2011) Augmented mirror: interactive augmented reality system
based on kinect. In: Human-computer interaction—INTERACT 2011, pp 483–486
38. Webb J, Ashley J (2012) Beginning kinect programming with the microsoft kinect SDK, 1 edn. Apress
39. Wobbrock JO, Aung HH, Rothrock B, Myers BA (2005) Maximizing the guessability of symbolic input.
In: Proceedings of the CHI ’05 extended abstracts on human factors in computing systems, pp 1869–1872
40. Wobbrock JO, Morris MR, Wilson AD (2009) User-defined gestures for surface computing. In:
Proceedings of the SIGCHI conference on human factors in computing systems, pp 1083–1092
41. Yong B, Xu Z, Wang X, Cheng L, Li X, Wu X, Zhou Q (2018) Iot-based intelligent fitness system. J
Parallel Distrib Comput 118:14–21
Multimedia Tools and Applications

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps
and institutional affiliations.

Affiliations

Hye Sun Park1 · Gun A. Lee2 · Byung-Kuk Seo1 · Mark Billinghurst2

Gun A. Lee
gun.lee@unisa.edu.au
Byung-Kuk Seo
byungkuk.seo@etri.re.kr
Mark Billinghurst
mark.billinghurst@unisa.edu.au
1 Electronics and Telecommunications Research Institute, Daejeon, Korea
2 University of South Australia, Adelaide, Australia