Cognitive AR based solution for prevalent post-stroke

communication deficits

ABSTRACT
About 800,000 to 1,000,000 people in India are affected by Aphasia annually. Aphasia is an
impairment of language, which majorly affects the production or comprehension of speech and
the ability to read or write. Speech and language therapy is often the suggested course of
treatment for Aphasia carried out by a speech & language therapist. For any language therapy to
be more effective and successful over the long term, rapid therapeutic actions are essential. To
provide the same, there is an approach with great potential which focuses on treatment that is
incorporated in the form of everyday activities through Augmented Reality. Augmented Reality
applications offer a solution with enormous potential to meet the requirements of those who need
help with speech therapy and comprehension. By interacting and practicing with their own
belongings at home, patients may engage in an extremely natural workout scenario. This
facilitates a better learning curve for an individual. This paper discusses the concept of using
Augmented Reality to introduce a new set of options in the realm of offering Aphasia therapy
using mobile applications, therefore increasing their communication and understanding abilities
while giving a calming experience.

INTRODUCTION
Around a third of people who have a stroke will experience aphasia. Stroke remains the main
cause for Aphasia, an impairment of language, which majorly affects the production or
comprehension of speech and the ability to read or write. Aphasia following a stroke continues to
be a significant handicap in 20% - 40% of stroke survivors worldwide, and it can range from
11% - 40% in the Indian context [1]. The Standard diagnostic aphasia tests like the Boston
Diagnostic Aphasia Evaluation and the Western Aphasia Battery, require provision of
specialized speech and language services, are time consuming and may be a challenge for wider
use in low-resource countries like India. These methods require qualified personnel to carry out
the screening test and analyze the test results to find aphasia in the context of an acute stroke.
Thus, for the low resource settings, there is a need for easy to use, bedside aphasia screening
tools that have greater efficacy in early diagnosis of aphasia.

In this work, we propose an AR Cognitive Screening and therapeutic intervention for Aphasia
designed application based upon the Frenchay Aphasia Screening Test (FAST). The novelty in
the proposed solution is lying into its specific scenario of multi-lingual and multi-cultural
support for Indian environment. The languages will be supported by the proposed research will
be English and Hindi. Having incubated start-up to take the research outcome to market, we have
taken on board some start-ups with some planned activities like Product finalization, Payment
gateway, Re-creating & Packaging, Social Media Awareness, Publicity & Marketing, and Field
Management.

This paper is organized in different sections:

Section1: Grounding for proposed work on Aphasia.
Section2: Talks about Background in India.
Section3: Talks about Rehabilitation process involved for prevalent post stroke communication
deficits.

Background
1. Grounding for proposed work on Aphasia
Stroke is one of the leading causes of death and disability in India [5]. The estimated adjusted
prevalence rate of stroke range is 84-262/100,000 in rural and 334-424/100,000 in urban
areas [5]. The incidence rate is 119-145/100,000 based on the recent population-based
studies. Stroke rehabilitation is not well developed in India due to lack of personnel.
Organized rehabilitation services are available in the country, but they are mainly in private
hospitals of the cities. Stroke has been defined by the World Health Organisation as ‘a
clinical syndrome consisting of rapidly developing clinical signs of focal (or global in case of
coma) disturbance of cerebral function lasting more than 24 hours or leading to death with no
apparent cause other than a vascular origin’ (Hatano, 1976). Aphasia is a communication
disability caused by damage to the language centres of the brain.

Post-stroke aphasia persists as a major disability in 20%-40% of stroke survivors across the
globe and ranges from 11% to 40% in the Indian context. Screening and diagnosis of
language impairment in the Indian context, given the cultural, linguistic and educational
diversity necessitates the availability of tools that are adapted to these populations. We aim to
adapt and validate the Frenchay Aphasia Screening Test (FAST) to the Indian context.
Standard diagnostic aphasia tests like the Boston Diagnostic Aphasia Evaluation and the
Western Aphasia Battery, require provision of specialized speech and language services, are
time consuming and may be a challenge for wider use in low-resource countries like India.
Commonly used screening tests for aphasia include Frenchay Aphasia Screening Test
(FAST), Mississippi Aphasia Screening Test (MAST) and Acute Aphasia Screening
Protocol.
 Aphasia can persist as a disability in 21 percent to 38 percent of stroke
survivors, with around 43 new cases being reported each year per 100,000
populations. There are around two million people in India living with the condition.
 Aphasia is a disturbance that impairs a person's formulation, expression and
comprehension of language as well as reading and writing abilities. Persons experiencing
Aphasia are unable to formulate/ convert non-verbal mental representations of a thought
into a grammatically organized sentence or a phrase and vice versa.
 The language disorder may disturb the syntax, lexicon as well as morphology, all
contributing to the meaningfulness of communication
 Aphasia is a condition that essentially disrupts the brain’s filing system – where the
people affected aren’t able to find the words they want to say, miss out words in their
sentences, or even come up with incorrect words, resulting in them speaking gibberish. 
 Since the filing system is muddled, they also have difficulty in understanding others.
 Language and speech are two different things. If the damage is to the language centre of
the brain, it presents difficulty in understanding or being able to find the right words. 
 But aphasia can also be caused by loss of motor function, where one knows exactly what
they want to say but the brain does not give command to, so they are unable to say it.
 The condition is brought on by damage to the brain from a stroke, head injury, or a
tumor, with stroke being the most common cause in India.
 Depending on how the brain is affected and the pattern of impediments, aphasia can be
categorised into groups: Expressive aphasia, Comprehensive aphasia and Global aphasia.

Rehabilitation: After a stroke, you may need rehabilitation (rehab) to help you recover.
Rehab may include working with speech, physical, and occupational therapists.
Language, Speech, and Memory: You may have trouble communicating after a stroke. You
may not be able to find the right words, put complete sentences together, or put words
together in a way that makes sense. You also may have problems with your memory and
thinking clearly. Speech and language therapists can help you learn ways to communicate
again and improve your memory.
Muscle and Nerve Problems: A stroke may affect only one side of the body or part of one
side. It can cause paralysis (an inability to move) or muscle weakness, which can put you at
risk for falling. Physical and occupational therapists can help you strengthen and stretch your
muscles. They also can help you relearn how to do daily activities, such as dressing, eating,
and bathing.
Bladder and Bowel Problems: A stroke can affect the muscles and nerves that control the
bladder and bowels. You may feel like you have to urinate often, even if your bladder isn’t
full. Medicines and a bladder or bowel specialist can help with these problems.
Swallowing and Eating Problems: You may have trouble swallowing after a stroke. Signs of
this problem are coughing or choking during eating or coughing up food after eating. A
speech therapist can help you with these issues.
 Emotional Issues and Support: After a stroke, you may have changes in your behaviour or
judgment. For example, your mood may change quickly. Because of these and other changes,
you may feel scared, anxious, and depressed. Recovering from a stroke can be slow and
frustrating. Talking to a professional counsellor also can help. Joining a patient support group
may help you adjust to life after a stroke. You can see how other people have coped with
having strokes. Talk with your doctor about local support groups or check with an area
medical centre.

International
Classification Impact of Aphasia
of Functioning
Impairment Auditory/ Reading disturbances, Verbal or Written expression,
Dyspraxia
Activity Ability to communicate thoughts and ideas, Ability to learn, Ability to
express wants and needs
Participation Everyday activities, Work/ Education, Social life, Reduced anatomy,
Diffident in control over life, Limitations in decision making,
Limitations in Social Integration, Independence, Risk of abuse due to
reduced autonomy
Wellbeing Emotional distress, Relationships, Depression, Anxiety

Motivation: India is socio-demographically diverse, with a significant portion of the

population that is illiterate. Development of a culturally, linguistically, and educationally
appropriate tool to evaluate stroke aphasia, where item selection fits the population, is
therefore, a challenge. To plan a mode of therapeutic /rehabilitative work it is important to
assess problems from a neuropsychological perspective focused on remediation of
impairment processes or compensation. Stroke is becoming an important cause of premature
death and disability in low-income and middle-income countries like India, largely driven by
demographic changes and enhanced by the increasing prevalence of the key modifiable risk
factors. At present in India, the only treatment available is speech and language
therapy, where therapists retrain the brain to sort out the filing system. After exploring the
available solutions, this is found that these apps are not very suitable to Indian environment
in terms of item familiarity, as well as some are with very basic functionalities just to add
cognitive load in the understanding of the patients.
This motivated us to propose a feasible solution which would be designed in the
directions of medical practitioners and domain experts.

Literature Survey
The current available solutions for Aphasia patients involve assistance from speech and
comprehension therapists. Speech pathologists can assess a stroke patient's ability to put words
together and communicate. They may assess how effectively a person answers questions,
identifies things, converses, and follows orders. Therapists also work with individuals to help
them re-create fundamental sounds. Patients are frequently shown photos and asked to describe
what they perceive. But the expense of therapy may prevent some people from receiving the care
they require. With mobile devices growing more widespread in daily life, they have a significant
impact on how people access information, teach, and learn. Given the speed and value of
information gained through these technologies, using it for study or as instructional material has
become very common. Knowing that the younger generation lives in a completely different
environment that is almost totally centered on technology [2], it is extremely useful to employ
cutting-edge technologies such as Augmented Reality (AR) to assist an individual focus and
advance their knowledge. With the support of AR-enabled healthcare software solutions,
Aphasia patients may take an active role in their own care. Two self-care practices where AR
may be quite helpful are physical therapy and rehabilitation. Patients can enhance their
movements and restore physical fitness by watching Augmented Reality-based digital
demonstrations of their body motions and behaviors. Playing mini-math comprehension games
can also help them enhance their reasoning ability. Patients may also use their smartphone
camera to point at a cityscape to acquire information about nearby hospitals and driving
directions.
A study of Augmented Reality technologies has led to the development of various applications in
the field of neuro cognitive sciences and medical healthcare. The literature review shows how
AR is applied in various ways and approaches to come up with solutions on helping Aphasia
patients post stroke.

Literature Reviews were done to proceed with the proposed project and mentioned below are
the briefs derived from the same.
Ting-Chia Hsu proposed a method of ‘Learning English with Augmented Reality: ‘Do learning
styles matter’ [4]. Students were free to choose which situational stage or challenge to begin
with. No matter what sequence they learned in, they had to compete to learn all the situated
English distributed in the surroundings with the support of the augmented application on the
Tablet PC. The other system was designed based on the task-based learning approach. In other
words, the students followed the arrangement of the task sequence. They had to solve the
situated tasks according to the target guided by the augmented application on the Tablet PC. If
they did not solve a task, they could not go on to the next one. They had to learn the AR
material and be sufficiently proficient to pass the test at the end of each learning target. There
was high positive correlation for the students using the self-directed game but a low positive
correlation for the task-based game.
Pei-Hsun Emma Liu and Ming-Kuan Tsai in their ‘Using augmented-reality-based mobile
learning material in EFL English composition’ [6], intended to develop AR-based mobile
learning material. Specifically, this learning material installed in mobile phones enables users to
access information about scenic spots nearby so that they can learn about buildings/places/views
of interest in English. If this learner uses the mobile phone to point in a specific direction, the
learner’s location is rapidly identified, and the embedded camera automatically captures the
peripheral images. In the meantime, the AR-based mobile learning material generates related
information (e.g., names and descriptions of the buildings).

METHODOLOGY

1. Design Paradigm

An app that uses AR technology to display visual aids or prompts to help people communicate
more effectivelyto develop a social platform specifically for people with communication
deficits due to stroke, where they can connect with others who are experiencing similar
challenges and share tips and strategies for overcoming them. This platform could also include
features such as video chat or messaging to facilitate communication between users.

It will be important to work closely with speech therapists, rehabilitation specialists, and other
experts in the field to ensure that the platform is effective and useful for people with
communication deficits due to stroke. It may also be helpful to gather feedback from users of
the platform to fine-tune it and make it as helpful as possible.

A view of the physical real-world environment with superimposed computer-generated images,

thus changing the perception of reality, is the AR [2]. The introduction of Augmented Reality
applications is altering the way learning is offered which enhances learning through interaction.
Patients often overlook or neglect to communicate mild symptoms to doctors. As a result,
healthcare professionals are not able to diagnose and give adequate treatments to patients.
However, with the use of AR technology, the two can more effectively assess the symptoms and
diagnose, which leads to better results.

However, for Aphasia patients who want to learn various subjects, there are not many
Augmented Reality applications available for download on smartphones.

From the literature survey it was found that game-based learning environments give an
individual engaging, immersive social, emotional, and cognitive experiences that are helpful to
their learning. Games can reinforce the player's confidence that they can improve and try again,
turning failure into incentive for learning.

Using the concepts mentioned thus far as a foundation, this study of Augmented Reality
Solution for Aphasia Patients has been created to aid in their cognitive development post stroke.
The approach involves building an application which aids people with Aphasia in improving
their communication and comprehension abilities. The goal aims to create an interface to
provide a practical and natural approach to enhance their speech and comprehension skills
through AR-based learning exercises. Augmented Reality creates unique digital experiences
that blend the best of digital and physical worlds and facilitate understanding. Additionally, no
specialized hardware or software is required for the experience and can be used simply via
mobile devices.

Figure 1 depicts the flow of how an AR system works thereby displaying the virtual
components in a real-world environment.

Figure 1: Working principle flow diagram of an AR Application

The application attempts to concentrate on two key elements, namely giving patients access to:

- Voice Therapy, and

- Improving their Comprehensive power.

Three main steps make up the entire process of providing with the aforementioned therapy
elements:

 Assessment Stage: The patients will be required to go through an assessment. This

entails a series of questions that individuals must respond to in order to determine the
degree and nature of their Aphasia and receive the proper assistance to treat it.
 Figure 2: Identification of Appropriate Module

As depicted in Figure 2, the results of the assessment will help determine whether the patient requires
speech therapy, comprehension treatment, or both, and at what level. They will then need to go
through the necessary modules and exercises.

2. Rehabilitation Stage: Based on the Aphasia discovered at the assessment level, the patients
get provided appropriate rehabilitation. The stage contains different types of modules based on
the degree and type of Aphasia. The approach entails a variety of exercises, many of which are
based on Augmented Reality (AR) technology and showcase real life objects with the primary
goals of enhancing the participants' speaking and comprehension skills. Game-based tasks are
given to them as they are typically more participatory and facilitate learning through visual
representations. Exercises involve exhibiting and saying the artifacts to the patient, then asking
them to repeat them, or asking them to match up the same items on a matrix grid.

The modules consist primarily of:

A) Speech Based Therapy Modules: It entails exercises that will aid in the patients' improved
pronunciation and vocabulary. Learning by Speech: The exercises showcase a series of
objects/artifacts accompanying their name and pronunciation. The patient must speak it
aloud and the system catches their response and then checks for the correctness of the
answer, i.e. the spoken response matches the object displayed. The system thus helps with
the speaking and oration skills of the patient, pronouncing the objects correctly to move on
to the next object.

B) Comprehension Based Therapy Modules: Patients with Aphasia can struggle to

understand even straightforward instructions. Therefore, the exercises that follow aid in
improving their fundamental comprehension skills.
Spell Game: This involved spelling of the word from the given set of jumbled letters.

Matrix Game: The goal of this game exercise involves matching the objects in the matrix.
This will be accomplished by having the player drag one object over another that appears to
be comparable in the matrix.

Hero Game: The game's objective is stated at the outset and is written at the top of the
screen. All the prefabs will be visible after the game has begun. The object is intended to be
destroyed by the hero, present in the middle of everything else that is spoken or written
above.

Bananas in the Basket: An individual goes through similar levels in this module which
showcases daily life objects, serving as a teaching tool and attempting to increase their all-
around ability by having them carry out the instructions in the task statement. All these
modules also assess the patient's retention (memory) capacity and determine whether they
can follow up with the previous steps or not.

 Evaluation Stage:

The step includes evaluation. Patients will once more be asked several questions to
determine whether they have improved following the rehabilitation procedure and to
determine whether the product has produced positive or negative results, which will
determine its practical applicability.

The framework of the above discussed therapy game modules is depicted in Figure 2 below.
It mainly consists of three systems namely Interactive, UI and Audio, dealing with user
touch interaction, user display, and sound system respectively.
 Figure 3: Game Modules Framework

RESULT
The modules in the proposed application have been designed and developed using a variety of
tools. The game scenes have been created and animated in Unity software along with being
coded for their respective logics in C# using the Visual Studio Community. Unity asset store has
been utilized to obtain the objects being placed in the scene. Additionally, Vuforia Engine (an
AR platform integrated with Unity) has been used to detect spaces and surfaces to configure the
application to interact and place objects in the real world. Any Android smartphone which
supports the AR technology can be used to see objects in the physical world.
As seen in the accompanying graphics, each scene in this program features unique objects that
have been exported to Unity and have been attached with Lean Touch scripts to make the scenes
interactable:

Speech Based Therapy Modules:

(a) (b) (c)

Figure 4: (a) Display of an Elephant in the Real World, (b) Counting Exercise, (c) Arrangement of Objects
in the Unity Editor

Comprehension Based Therapy Modules:

(a) (b)
 (c) (d)

Figure 5: (a) Dragging specified number of objects in the basket module, (b) Spelling Game,
(c) Matrix game display of 3D Animals, (d) Instruction based Hero game

2. Implementation
 Technical
Some of the major C# functions used in the application have been listed below:
1) SetActive():
Helps to activate/ deactivate the object/prefab from the script. This function enables us to
disable every component, turning off any attached renderers, colliders, rigid bodies,
scripts, etc.
2) Destroy():
The function enables us to destroy the prefab on clicking or on collision of another
prefab.
3) SpeechRecognizer():
The function enables the user to start and stop recording on a button press and return the
permission and availability information. It then sends callback methods when speech has
been detected.
4) OnTriggerEnter():
Function checks for collision detection between two objects.

Currently, there are not many applications available or accessible that use Augmented Reality as
a technology to deliver therapy for patients with Aphasia.
Table 1 highlights the key characteristics of several of the current applications that offer therapy
alternatives (none of which utilize AR technology within them).
 Table 1: Features of Currently Available Applications (Not using AR technology)

Application Owned By OS Support Price Available in

Name India

Language Tactus iOS and Android Rs. 5,500 Yes

Therapy: Therapy
Aphasia Solutions Ltd.
(USA)

Constant Constant iOS, Android, Free for 14 days Yes

Therapy Therapy and Amazon only (then Rs.
Health, Inc. Kindle Fire 2400
(USA) devices subscription).

Voice4u Spectrum iOS and Android Rs. 200 Yes

Visions
Global, Inc.
(USA)

Speech A-Soft iOS and Android Free No

Assistant
AAC

It must be noted that almost all these applications have been developed outside of India and those
that are currently in use have a fee associated with using their full range of features.

CONCLUSION AND FUTURE SCOPE

Every day, more people are using mobile devices as hardware development and affordability
have made it possible to run such technology on devices that are already at our disposal [2, 3].
Additionally, it is highlighted how technologies like virtual reality and Augmented Reality
provide a wide range of advantages and aid in the creation of applications in the field of
healthcare and education as it makes the ability to learn and teach intriguing due to its qualities
of usability, dynamism, and ability to foster interaction between the real and virtual worlds.
With the use of mobile Augmented Reality technology, the therapy solution for Aphasia patients
proposes to support the post stroke individuals to improve and boost their comprehension and
speech abilities. The application gives adults with neurological impairment, like stroke or
Aphasia, the practice they need in between reading words and comprehending given tasks in
order to address the problems brought on by the stroke, and assess the degree of their Aphasia
and subsequently receive treatment as necessary. In addition, it saves time and effort as it can be
accessed anywhere and anytime and provides a more comfortable experience for a user.
Future enhancements are suggested to include additional features like several different game
strategies focusing on the major aspects of the issues caused post stroke, namely, reading,
writing, speaking, and listening and providing a bigger experience to the user.
Additionally, this AR Solution for Aphasia Patients will be further divided into modules
depending upon the degree of severity of Aphasia of the post stroke individual at the same time
keeping in consideration the age group that the patient falls under. Besides all this, the
application will have the feature to keep the track and progress of a patient for a better evaluation
procedure.
REFERENCES

[1] Paplikar, A., Iyer, G. K., Varghese, F., Alladi, S., Pauranik, A., Mekala, S., Kaul, S., Sharma,
M., Dhaliwal, R. S., Saroja, A. O., Dharamkar, S., Dutt, A., Divyaraj, G., Ghosh, A., Kandukuri,
R., Mathew, R., Menon, R., Narayanan, J., Nehra, A., Padma, M. V., … ICMR Neuro-Cognitive
Tool Box Consortium (2020). A Screening Tool to Detect Stroke Aphasia: Adaptation of
Frenchay Aphasia Screening Test (FAST) to the Indian Context. Annals of Indian Academy of
Neurology, 23(Suppl 2), S143–S148. https://doi.org/10.4103/aian.AIAN_499_20

[2] Asaad, R. R., Sulaiman, Z. A., & Abdulmajeed, S. S. (2019). Proposed System for Education
Augmented Reality Self English Learning. Academic Journal of Nawroz University, 8(3), 27–32.
https://doi.org/10.25007/ajnu.v8n3a366

[3] Morais, Ceres & Silva, Carlos & Mendonça, Antonio & Silva, José. (2020). Cubo Kids: A
Proposal for an Educational Application with Augmented Reality. IEEE Transactions on
Education. https://doi.org/10.1109/TALE48869.2020.9368462

[4] Hsu, T.-C. (2017) Learning English with Augmented Reality: Do Learning Styles matter?
Computers & Education, 106, 137-149. https://doi.org/10.1016/j.compedu.2016.12.007

[5] Antkowiak, D., Kohlschein, C., Krooß, R., Speicher, M., Meisen, T., Jeschke, S., & Werner,
C.J. (2016). Language therapy of aphasia supported by augmented reality applications. 2016
IEEE 18th International Conference on e-Health Networking, Applications and Services
(Healthcom), 1-6, https://doi.org/10.1109/HealthCom.2016.7749511

[6] Liu, P.E., & Tsai, M. (2013). Using augmented-reality-based mobile learning material in EFL
English composition: An exploratory case study. Br. J. Educ. Technol., 44, 1-.
https://doi.org/10.1111/j.1467-8535.2012.01302.x
[7] Kylie Janine Wall, Toby Borland Cumming, Sebastian Thomas Koenig, Anita Maria
Pelecanos & David Andrew Copland (2018). Using technology to overcome the language
barrier: the Cognitive Assessment for Aphasia App, Disability and Rehabilitation, 40:11, 1333-
1344. https://doi.org/10.1080/09638288.2017.1294210
[8] Burke, J., McNeill, M., Charles, DK., Morrow, PJ. Crosbie, J., & McDonough,
S. (2010). Augmented Reality Game Design for Upper-Limb Stroke Rehabilitation. In Unknown
Host Publication (pp. 75-78). IEEE. https://doi.org/10.1109/VS-GAMES.2010.21

[9] Marrahí-Gómez, V., & Belda-Medina, J. (2022). The Application of Augmented Reality
(AR) to Language Learning and its Impact on Student Motivation. International Journal of
Linguistics Studies, 2(2), 07–14. https://doi.org/10.32996/ijls.2022.2.2.2