2021 The 21st International Conference on Control, Automation and Systems (ICCAS 2021)

Ramada Plaza Hotel, Jeju, Korea, Oct. 12~15, 2021

Implementation of Rehabilitation Platform based on Augmented Reality

Technology
Nguyen Truong Thinh, Nguyen Anh Quoc, Nguyen Vo Tam Toan and Tran The Luc
Department of Mechatronics, Ho Chi Minh City University of Technology and Education, Vietnam
Email: thinhnt@hcmute.edu.vn, {anhquoc11358, nguyenvotamtoan, tranthelucbcd}@gmail.com

Abstract: Stroke is now one of the leading causes of death and disability in both motor and cognitive functions. In
addition, rehabilitation for stroke survivors also faces many physical and human difficulties. To address this issue, we
studied the use of Augmented Reality (AR) in rehabilitation using a system called RARS. This system creates limbs
rehabilitation exercises in the form of games using an AR interface. The goal of the RARS is to increase patient’s positive
emotions, which motivates them to enjoy rehabilitation exercises. As a result, recovery efficiency will be improved, and
the burden on physiotherapists will be reduced. The RARS is evaluated through a research about the effectiveness of this
system on rehabilitation for patients after stroke (n=10). Reported results showed that the RARS produced significant
improvement in the patient’s indicators of functional status. From that, this system shows that it not only creates great
benefits for personnel and economic but also bring about huge potential for future growth.
2021 21st International Conference on Control, Automation and Systems (ICCAS) | 978-89-93215-21-2/21/$31.00 ©202110.23919/ICCAS52745.2021.9650059

Keywords: stroke, rehabilitation, Augmented Reality.

1. INTRODUCTION compatible with the patient’s limb movement.

Complete loss or partial loss of limb motion function
is one of the conditions that affect a lot in a daily life of
the patient. The case of loss of limb motion function is
mainly hemiparesis, which is a condition in which one
side of the body is weakened, the right or left hemiplegic
pain depends on the brain area damaged during the stroke
or other causes. Damage to the left brain will cause the
right hemiparesis and vice versa. The paralyzed side will
have weaker motion than the other side or may not even
move. The main cause of hemiparesis is cerebral
hemorrhage or hemorrhagic stroke, cerebrovascular
diseases that interrupt the transfer of blood to the brain
causing ischemia leading to stroke. Injuries and brain
damage are also the causes of hemiparesis. This
syndrome can easily make patients feel depressed, lose
confidence in life, easily accept to live with that state and
no longer have more motivation to perform rehabilitation
exercises. Therefore, by designing and manufacturing a
device which can make users feel interesting, fun, and
active during rehabilitation is the optimal solution to this
problem. One solution proposed by Burke and his
colleagues is AR Games for Limb Stroke Rehabilitation
[1]. In this research, AR has been proved to bring about
interesting and rewarding outcomes for upper limb stroke
rehabilitation. AR technology opens up the possibility of
high-quality, user-friendly rehabilitation and help patient
feel comfortable during training sessions and quickly
recover. It also reduces the pressure on hospitals, as a
result the lack of nurses and doctors to guide the patients
will no longer be a difficult problem.
An AR rehabilitation system through gaming (RARS)
has been researched and completed for the above
practical benefits. On an AR interactive platform, the
system includes hardware and software for limb motor
rehabilitation. The patients use the physical system to
practice movement, which is displayed on the screen to
give the impression that they are playing a game. Sensors
in the system provide real-time feedback parameters
Furthermore, the movement of the character in the game
is simulated based on the sensor parameters to be
compatible with the patient in the present, increasing the
realism of the game. When the patient performs wrong
movements or applies inappropriate force, the system
will warn and give some advice to the patient.
A rehabilitation study is being conducted to assess the
current system in two groups: RARS and traditional
physical therapy rehabilitation. Patients who have
mobility issues within three months of a stroke are
eligible to participate as volunteers. Patients need early
access to stroke rehabilitation because major
improvements only occur in the first few months after a
stroke [2]. In this paper, the recovery results between the
two groups will be analyzed and discussed, thereby
evaluating the effectiveness of the RARS.
2. REHABILITATION AND AR
According to a research [3], defined emotion was
defined as a form of biological manifestation related to
the nervous system due to different physiological and
neurological changes. It has a major effect on people’s
moods, temperaments, attitudes, and inspiration.
Negative feelings can make one feel uneasy, like sadness,
rage, anxiety [4]. Positive feelings, on the other hand,
may be made up of a variety of distinct pleasant-valenced
emotions like pleasure, pride, contentment, or affection,
or a more generalized state of positivity [4]. Positive
emotions help us take in more information, remember
more, moreover these give us the better ability to learn
and build our skills.
Immersion can be defined as the degree to which the
AR system affects the senses of users. Specifically, it is
the concentration of users and objects and scenes in a
virtual environment, which makes their senses lose their
attention to the real world. At that time, they only focus
on observing, receiving information, and interacting with

978-89-93215-21-2/21/$31.00 ⓒICROS 1926

Authorized licensed use limited to: University of Ulsan. Downloaded on January 03,2022 at 00:16:56 UTC from IEEE Xplore. Restrictions apply.
the virtual environment. This immersion level is private clinic or community as well as home. In addition
evaluated [5] depending on the following factors: to the AR method, there are other methods of using more
Scalability: the amount of information received from the expensive treatment aids such as robotic systems.
virtual world (image, sound, physical force); Match: The However, AR systems showed the ability to provide a
change of the scene in the virtual environment more intuitive experience because it is based on a game
corresponds to the movement of the head; Ambiance: The platform with a virtual world that is simulated like our
complete level of the virtual environment, which allows surroundings. In a research about an Augmented Reality-
the user to observe a 360º perspective; Vividness: The based rehabilitation system was developed to restore
image and sound quality of AR system devices; movable hands [9], hand-holding sensation and hand-eye
Interoperability: The ability of virtual objects to respond coordination met the requirements. Therefore, the
to user actions. Immersion is an important part of virtual application of AR technology in rehabilitation shows the
environment building systems, including AR and ability to save costs but still bring high efficiency to
adjusting the immersion level of the AR system will patients.
directly affect the user’s experience in a virtual
environment.
The concept of presence is described as the measure 3. INTEGRATING AR INTO PLATFORM
for the number and fidelity of available sensory input and
output channels. To be more precise, personal presence is The RARS was developed to support patients base on
a measure of the degree to which the person feels like he the above methods (Fig. 1). It includes a mechanical part
or she is part of the virtual environment [6]. Presence can with rotating mechanism on which sensors are mounted:
be divided into two categories: Cognitive Presence and torque sensors, encoders, and servo motors to support
Perceptual Presence. Cognitive Presence is a situation in force when the patients have signs of fatigue; a display
which the users can have a mental feeling that they are in interacts with the patient as a virtual assistant; a camera
virtual worlds while they are in real worlds, and is crucial have used to collect images of the patient which is data
for any kind of immersive experience. Perceptual for image processing.
presence is a situation in which the user can feel their
existence in an unreal world throughout their senses. For
an AR system to have a presence, that system needs to
achieve both cognitive presence and perceptual presence.
This requires AR systems to provide users with a
seamless, low-latency experience so that interactions can
happen instantly, like in the real world.
According to a document of Zhou, Duh and
Billinghurst, AR is a technology which allows computer
generated virtual imagery to exactly overlay physical
objects in real time. They are performed through a variety
of senses, including visual, auditory, haptic,
somatosensory, and olfactory [7]. AR can be defined
according to Hsin-Kai Wu and his partners, it as a system
that responds to three basic features: a combination of
real and virtual worlds, real-time interaction, and
accurate 3D registration of virtual and real objects [8].
Constructive virtual information helps to create virtual
objects that are introduced into the real world, and
provides an experience that combines the real world and Fig. 1. The AR rehabilitation system through gaming
the virtual world to the user. As virtual information
The RARS interacts with patients through an interface
changes, the perception of the users about the world
as shown in Fig. 2. The interface includes: an AR view, a
around it also changes. Hence, AR is used to improve the
real environment and enhance the experience through the human body model, an actual view and some necessary
senses of users. parameters for the user such as rotation speed of the limbs,
Rehabilitation is defined as solutions to improve accuracy of the training process, power-assisted motor
and training time. The AR view is a Unity-based software
function and minimize the difficulty in daily activities
that creates a scenario depicting a bicycle on the road.
such as moving, eating, ... Rehabilitation helps patients
When the user exercises limbs, the AR view also moves
reduce dependence on supporting devices or help from
others, which helps patients increase their ability to at a speed proportional to the user training speed. The
participate in social activities. The commonly used human body model shows the main muscle bundles of the
method of rehabilitation is to perform the functions of the body. As the user exercises, the device gathers data from
the sensors and uses it to process and generate outcomes
part over and over again, from easy to difficult levels.
for the muscle bundles being exercised, which is shown
These rehabilitation therapies can be offered in many
on the interface by changing the color of active muscle
facilities, from an inpatient or outpatient hospital to a
bundles to light red. To detect motion of the patient’s

1927
Authorized licensed use limited to: University of Ulsan. Downloaded on January 03,2022 at 00:16:56 UTC from IEEE Xplore. Restrictions apply.
entire joints, a camera was employed. Image processing
using the open source MediaPipe package was used to
detect key points on the frame. The location and angle of
the patient’s joints will be obtained as a result of this. The
algorithm then compares the erroneous poses to a
standard data set to assess thema and then sends a
notification to the patient to swiftly alter his or her
posture.
world. The program is intended to help patients practice
rehabilitation activities so the suggested prototype is a 3D
bike riding game in which players must reach
checkpoints in order to gain bonus points. Another good
idea is the Rowing game, Obstacle bike ride game (Fig.
3), in which players must ride to various locations to
collect prizes.

Fig. 3. Several simulating views on screen

Fig. 2. The interface of the RARS The mini games based on the principle diagram in Fig.
4. Where: Varm and Vfeet are the angular velocity of top
AR integrated rehabilitation system created by shaft and bottom shaft; Ph1 and Ph0 are the current vector
building Unity 3D-based games with a variety of scenes. and the previous vector which indicate the patient’s
It provides the necessary 3D environments and tools to direction of sight. The number of encoder pulses can
make simulation more intuitive. We have created a high- easily be used to quantify the velocity and direction of
resolution model of the environment [10] and the human the axes during rehearsal. However, synchronizing the
models. The perspective includes 2 types: first-person initial position of the real pedal with the initial position
and third-person. Patients who use rehabilitation devices of the pedal in the mini game is challenging. This has a
would take on the part of the game’s lead character to major impact on the patient’s sense of immersion and
complete some activities. If the patient performs the appearance so the image processing approach was
game’s mission, they will also perform the rehabilitation selected as the best alternative. The camera location on
activity that corresponds to it. Real-world patient the machine remains unchanged, allowing the threshold
movements are collected by the system through camera algorithm to calculate the real pedal position.
and sensors and then the program analyzes and simulates
the movement of patients from these results. The gestures False
of the patient heads need careful consideration because
False Camera
Update
Update
they are a crucial movement that defines the patient’s Initial
State
Start
True Velocity
Position
Varm> 0 or True
Vfeet > 0
position
of arm
position
vision in the augmented environment. Immersion and and feet
of player

appearance are profoundly influenced by the accurate False

Encoder

and seamless movement of shifting the vision. In addition, Stop

Update
Rotate True
camera in Ph1 Ph0
the development of simulated objects in the augmented viewport
game
environment should be taken into account. Objects True
False

should be the correct scale, place, form, and react easily,

much as in the real world, so that the patient has no
trouble adjusting to a new environment. 3D human
simulation is another part of the device interface to which
we apply AR. The colour of the muscle bundles on the
model corresponds to their energy intake. The findings
shown in this manner will assist patients in observing
their training success in real time. Mini Game 3D is a
Unity-based game with a high degree of interaction
between users and the augmented universe. All objects in
the augmented universe have parameters like density,
weight, hardness, momentum, acceleration, and so on,
ensuring that they can interact physically. They offer
input that is almost similar to what you will get in the real
Fig. 4. Algorithm flowchart for mini games
4. EXPERIMENTS AND DISCUSSIONS
4.1 Protocol
The trial lasted 11 weeks was split into two stages:
training and practicing (Fig. 5). During the first week of
the study, patients were participated the training stage.
Three sessions per week with 10-minute per session were
used to teach patients about the device, training
procedures, and system familiarization. Then, they were

1928
Authorized licensed use limited to: University of Ulsan. Downloaded on January 03,2022 at 00:16:56 UTC from IEEE Xplore. Restrictions apply.
put through the practicing stage, which was also the hospital and the consent of the patient as well as their
study’s main focus. For ten weeks, patients were trained relatives.
five times per week for 20 minutes each. In total, during
the whole experiment, these patients were assessed 3 4.3 Results
times: Base, After and Final. The results of the experimental process are shown in
Table 2 (group A) and Table 3 (group B).
Baseline Evaluation Final
evaluation after training evaluation Table 2. Assessment results of group A

ID Assessment Base After Final

FMA/100 31 35 63
Training Practicing FIM/91 32 35 72
1
Barthel/100 35 40 70
Sas/5 - - 3
1 week 10 weeks FMA/100 46 49 76
Fig. 5. The stages of the study FIM/91 41 45 70
2
Barthel/100 50 55 75
The patients who took part in the study were Sas/5 - - 4
randomly split into two groups to practice rehabilitation: FMA/100 23 28 53
A and B (Table 1). The patients in group B used the 3
FIM/91 45 47 65
RARS to perform the arm and leg rotation exercise. Barthel/100 40 40 65
Group A, on the other hand, did this exercise too, but Sas/5 - - 3
practiced without the help of AR. During the study, these FMA/100 39 45 79
patients were evaluated using four main scales: FIM FIM/91 28 32 62
4
(Functional Independence Measure) [11], the Barthel Barthel/100 35 40 70
Index [12] and the Fugl-Meyer Assessment Test (FMA) Sas/5 - - 3
[13] for the upper and lower extremities of paretic side. FMA/100 53 56 88
Furthermore, level satisfaction of the patients about the FIM/91 32 32 67
study at the end of the experiment (Sas) was also assessed 5
Barthel/100 50 50 80
on a scale of 1-5 which 1-very dissatisfied and 5- Sas/5 - - 3
extremely satisfied.
Table 3. Assessment results of group B
Table 1. Patient information
ID Assessment Base After Final
Patient Age Gender Days Paretic Group FMA/100 53 63 94
ID post-stroke side FIM/91 52 55 82
1 45 M 15 Left A 6
Barthel/100 55 60 90
2 52 F 10 Left A Sas/5 - - 4
3 55 M 33 Right A FMA/100 30 33 76
4 44 F 18 Left A FIM/91 21 29 85
5 50 M 44 Right A 7
Barthel/100 30 45 95
6 53 F 26 Left B Sas/5 - - 5
7 37 M 20 Right B FMA/100 53 58 83
8 58 F 35 Left B FIM/91 45 47 75
9 50 M 21 Right B 8
Barthel/100 50 50 75
10 63 M 17 Right B Sas/5 - - 3
FMA/100 33 48 92
4.2 Participants FIM/91 25 32 62
Twelve volunteers with hemiparesis as a result of a 9
Barthel/100 30 40 90
stroke who were being treated in a hospital in Vietnam
Sas/5 - - 5
participated in a study to assess and analyze the system.
FMA/100 23 29 88
The patients had natural vision, sensory nerves and high
FIM/91 22 27 87
consciousness (tested by the Mini Mental State 10
Examination [14]). In these patients, 2 patients dropped Barthel/100 35 45 95
the study so we just consider 10 patients who finished the Sas/5 - - 5
trial completely. The information of these patients was
shown in Table 1. An average age of all participants is For an intuitive and fair assessment for all patients, we
50.7 years and an average time following a stroke is 23.9 only focused on the percentage improvement (Fig. 6):
days. This study was carried out with the approval of the The Base evaluation was conventionally 0%, the After

1929
Authorized licensed use limited to: University of Ulsan. Downloaded on January 03,2022 at 00:16:56 UTC from IEEE Xplore. Restrictions apply.
and the Final evaluation were calculated by these (49.7% and 49%) after 11 weeks of exercise, compared
formulas: to patients in group A (just 34.7% and 30%).
Rehabilitation exercises helped patients improve not only
( After  Base) *100 (1) their limb function, but also their communication,
After _ improve 
Total cognition, movement, and living skills. In this practice,
( Final  Base) *100 AR has been shown the ability to accelerate the recovery
Final _ improve  (2)
Total process and balance life.
In the evaluation, we extracted data of 3 patients from
group B: 7th, 9th and 10th who were paralyzed on the right
Mean FMA side. Experiment results in Fig. 7 showed the average
torque exerted on the right arms and legs of these patients
Improvement (%)

48.2
which were received from the torque sensors. Their
average torque of arm and leg were only 6 and 31.7 in the
33.4 first week, but they improved significantly by the last
7.8 week, reaching 26.7 in the arm and 74.7 in the leg.
Although the patient’s levels of increase varied
0 depending on their condition and motivation to exercise,
4.2
they all showed that the RARS had a positive impact
Base After Final during exercise.
Group A Group B

Mean FIM Right arm

49.7 40
Improvement (%)

Torque (Kg.cm)
30
34.7 20
10
5.5
0 0
2.9
1 4 7 11
Base After Final Weeks
Group A Group B
7th patient 9th patient 10th patient
Mean Barthel
Right leg
49
Improvement (%)

100
Torque (Kg.cm)

8
0
3 0
Base After
Group A Group B
Fig. 6. The average percentage of improvement
Despite the not large size of the group, there is a trend
that group B, who practiced with AR’s help, improved
more than group A. Using the FMA scale of motor
functioning in the upper and lower extremities, we
discovered that rehabilitation practicing with limb
rotation exercises helped all 10 patients improve their
limb function. The improvements in limb function
appeared within the first week, when the patients were
just getting used to the system. Finally, group B finished
the study with a higher average improvement than group
A (48.2% versus 33.4%).
Meanwhile, we saw a positive signal on the FIM and
Barthel scales, which measure functional independence.
Patients in group B improved their function significantly
80
60
30
40
20
Final 1 4 7 11
Weeks
7th patient 9th patient 10th patient
Fig. 7. The average torque of 3 patients
The findings also showed that when using traditional
physical equipment, the patient’s morale was a bit low
(average 3.2 on the Sas scale). This can be explained by
the fact that motorized equipment training is often very
boring and does not change day to day, resulting in low
motivation to practice, despite the fact that training
results improved but were still at a low level limit. Group
B, on the other hand, showed satisfaction when practicing
with the RARS, with an average satisfaction score is 4.4,
with up to three patients (7th, 9th and 10th) being
completely satisfied. This demonstrates that AR had a
very positive impact on the practice spirit of patients.
Every day, the games in the patient’s exercise were

1930
Authorized licensed use limited to: University of Ulsan. Downloaded on January 03,2022 at 00:16:56 UTC from IEEE Xplore. Restrictions apply.
changed according to the patient’s wishes, assisting them
in creating a sense of self-control in practice and as a
result, increasing motivation to practice. Furthermore,
practicing with AR games allowed them to forget about
their current body’s pain by immersing them in a virtual
world, making the exercise as simple and gentle as
playing a game.
5. CONCLUSIONS
The RARS is an AR-integrated system whose primary
purpose is to assist patients with rehabilitation exercises
with the ability to provide patient with restorative
exercises through interactive mini-games in real time
between user and system. It increases exercise motivation
and experiences an AR world where they can perform
sport activities while their current health condition in real
life remains disallowed. The presence of feedbacks such
as scoring and exercise duration are additional features to
prolong the patient’s interest in exercise. In addition, we
demonstrated our development at Thu Duc General
Hospital with a trial with 10 patients. The RARS was
evaluated by patients as more comfortable and effective
than previous traditional methods, thereby reducing
rehabilitation training time. Patients can do their own
low-cost rehabilitation exercises at home without the
need for a regular doctor’s presence. However, their
exercise is still monitored by the system and notified to
the doctor when necessary. This helps to reduce the
overload for patients and physiotherapists in medical
facilities. In the future, we will develop and improve the
existing weaknesses so that this system soon becomes an
important solution in the field of rehabilitation.
ACKNOWLEDGEMENT
The authors wish to thank HCMC University of
Technology and Education which funded and facilities
supported to complete this project. The authors also
express gratitude to the Department of Traditional
Medicine at Thu Duc General Hospital for assisting the
team throughout the research at the hospital, as well as
the volunteers who took part in the experiment about the
RARS.
REFERENCES
[1] J. W. Burke, M. D. J. McNeill, D. K. Charles, P. J.
Morrow, J. H. Crosbie and S. M. McDonough,
"Augmented Reality Games for Upper-Limb
Stroke Rehabilitation," in Proc. 2010 Second
International Conference on Games and Virtual
Worlds for Serious Applications, pp. 0075–0078,
2010.
[2] S. H. Kreisel, H. Bazner, and M. G. Hennerici,
"Pathophysiology of Stroke Rehabilitation:
Temporal Aspects of Neuro-Functional Recovery,"
Cerebrovascular Diseases, vol. 21, no. 1, pp. 6–17,
1931
Authorized licensed use limited to: University of Ulsan. Downloaded on January 03,2022 at 00:16:56 UTC from IEEE Xplore. Restrictions apply.
2006.
[3] Cacioppo, JT &Gardner, “Emotion”, Annual
Review of Psychology, vol. 50, no. 1, pp. 191–214,
1999.
[4] Soskin, D. P., Carl, J. R., Alpert, J., & Fava, M.,
“Antidepressant Effects on Emotional
Temperament: Toward a Biobehavioral Research
Paradigm for Major Depressive Disorder”, in
Proc. CNS Neuroscience & Therapeutics, pp.
0441–0451, 2012.
[5] J. Jerald, The VR book: Human-Centered Design
for Virtual Reality, California, USA, 2015.
[6] Martijn J. Schuemie, Peter van der Straaten, Merel
Krijn, and Charles A.P.G. van der Mast, “Research
on Presence in Virtual Reality: A Survey”,
CyberPsychology & Behavior, vol. 4, no. 2, pp.
183–201, 2004.
[7] Feng Zhou, H. B. Duh and M. Billinghurst, "Trends
in Augmented Reality Tracking, Interaction and
Display: A Review of Ten Years of ISMAR," in
Proc. 2008 7th IEEE/ACM International
Symposium on Mixed and Augmented Reality, pp.
0193–0202, 2008.
[8] Hsin-Kai Wu a, Silvia Wen-Yu Lee, Hsin-Yi Chang,
Jyh-Chong Liang, “Current status, opportunities
and challenges of augmented reality in education,”
Computer & Education, vol. 62, no. 1, pp. 41–49,
2013.
[9] D. Zhang, Y. Shen, S. K. Ong and A. Y. C. Nee, "An
Affordable Augmented Reality Based
Rehabilitation System for Hand Motions," 2010
International Conference on Cyberworlds, pp.
0346–0353, 2010.
[10] M. S. Cameirao, S. Bermudez i Badia, E. D. Oller
and P. F. M. J. Verschure, "Using a Multi-Task
Adaptive VR System for Upper Limb
Rehabilitation in the Acute Phase of Stroke," 2008
Virtual Rehabilitation, pp. 0002–0007, 2008.
[11] R. A. Keith, C. V. Granger, B. B. Hamilton, and F.
S. Sherwin, "The Functional Independence
Measure: A New Tool for Rehabilitation,"
Advances in Clinical Rehabilitation, vol. 1, pp. 6–
18, 1987.
[12] F. I. Mahoney and D. W. Barthel, "Functional
Evaluation: The Barthel Index," Maryland State
Medical Journal, vol. 14, pp. 61–65, 1965.
[13] A. R. Fugl-Meyer, L. Jaasko, I. Leyman, S. Olsson,
and S. Steglind, "The Post-Stroke Hemiplegic
Patient. 1. A Method for Evaluation of Physical
Performance," Scandinavian Journal of
Rehabilitation Medicine, vol. 7, no. 1, pp. 13–31,
1975.
[14] M. F. Folstein, S. E. Folstein, and P. R. McHugh,
""Mini-mental state". A Practical Method for
Grading the Cognitive State of Patients for the
Clinician," Journal of Psychiatric Research, vol. 12,
no. 3, pp. 189–198, 1975.