MULTISENSORY SIMULATOR FOR STIMULATING ATTENTION

IN AUTISTIC CHILDREN

FARAH FADHILAH HERMAWATI

UNIVERSITI TEKNOLOGI MALAYSIA

 PSZ 19:16 (Pind. 1/13)
UNIVERSITI TEKNOLOGI MALAYSIA

DECLARATION OF THESIS / UNDERGRADUATE PROJECT REPORT AND

COPYRIGHT
Author’s full name : Farah Fadhilah Hermawati

Date of Birth : 11 June 2000

Title : Multisensory Simulator for Stimulating

Attention In Autistic Children

Academic Session : 2021/2022-1

I declare that this thesis is classified as:

CONFIDENTIAL (Contains confidential information under the

Official Secret Act 1972) *

RESTRICTED (Contains restricted information as specified by

the organization where research was done) *

✓ OPEN ACCESS I agree that my thesis to be published as online

open access (full text)

1. I acknowledged that Universiti Teknologi Malaysia reserves the right as

follows:
2. The thesis is the property of Universiti Teknologi Malaysia
3. The Library of Universiti Teknologi Malaysia has the right to make copies for
the purpose of research only.
4. The library has the right to make copies of the thesis for academic
exchange.
Certified by:

SIGNATURE OF STUDENT SIGNATURE OF SUPERVISOR

PROF. MADYA DR. MAHEZA

A18MB0215
IRNA BINTI MOHAMAD SALIM
MATRIC NUMBER NAME OF SUPERVISOR

Date: 04 JANUARY 2022 Date: 04 JANUARY 2022

NOTES : If the thesis is CONFIDENTIAL or RESTRICTED, please attach with the letter from
the organization with period and reasons for confidentiality or restriction
 “I hereby declare that I have read this proposal and in my
opinion this proposal is sufficient in term of scope and quality for the
award of the degree of Bachelor of Biomedical Engineering”

Signature : ________________________________
Name of Supervisor I : PROF. MADYA DR. MAHEZA IRNA BINTI
MOHAMAD SALIM
Date : 4 JANUARY 2022
BAHAGIAN A - Pengesahan Kerjasama*

Adalah disahkan bahawa projek penyelidikan tesis ini telah dilaksanakan melalui
kerjasama antara ________________________dengan ________________________
Disahkan oleh:
Tandatangan : Tarikh :
Nama:
Jawatan:
(Cop Rasmi)
* Jika penyediaan tesis atau projek melibatkan kerjasama.

BAHAGIAN B - Untuk Kegunaan Pejabat Sekolah Pengajian Siswazah

Tesis ini telah diperiksa dan diakui oleh:

Nama dan Alamat Pemeriksa Luar :

Nama dan Alamat Pemeriksa Dalam :

Nama Penyelia Lain (jika ada) :

Disahkan oleh Timbalan Pendaftar di SPS:

Tandatangan : Tarikh : 04JAN2022
Nama :
MULTISENSORY SIMULATOR FOR STIMULATING ATTENTION
IN AUTISTIC CHILDREN

FARAH FADHILAH HERMAWATI

A proposal submitted in partial fulfilment of the

requirements for the award of the degree of
Bachelor of Biomedical Engineering

School of Biomedical Engineering and Health Sciences

Faculty of Engineering
Universiti Teknologi Malaysia

JANUARY 2022
 DECLARATION

I declare that this proposal entitled “Multisensory Simulator for Stimulating Attention
In Autistic Children” is the result of my own research except as cited in the references.
The proposal has not been accepted for any degree and is not concurrently submitted
in candidature of any other degree.

Signature : ....................................................
Name : FARAH FADHILAH HERMAWATI
Date : 4 JANUARY 2022

iii
 DEDICATION

I dedicate this thesis to the beloved people who have meant and continue to
mean so much to me. First and foremost, a special feeling of gratitude to my loving
parents, Deni Hermansah and Kurniawati whose words of encouragement and push
for tenacity ring in my ears, and also to my younger brother who has never left my
side and is very special. I also dedicate this dissertation to my best friends and life
partner who have supported me throughout the entire bachelor program. Thank you.
My love for you all can never be quantified.

iv
 ACKNOWLEDGEMENT

I would like to thank the following people, without whom I would not have
been able to complete this final year project (FYP 1), and without whom I would not
have made it through the fast-track degree program.

The lecturer’s team at Universiti Teknologi Malaysia, especially my supervisor

Dr. Maheza Irna Binti Mohamad Salim, whose insight and knowledge into the subject
matter steered me through this research. Then other thanks to Dr. Siti Aisyah Mualif
as my academic advisor who has helped me navigate academic and personal concerns
that may influence my academic performance. And special thanks to all lecturers who
have taught me since year one till at the moment, without whom I would have zero
knowledge in working on this research.

Afterward most importantly, my biggest thanks to my family for all the support
you have shown me through this research, the culmination of three and a half years of
distance learning. And last but not least, thanks to my distanced life partner who
pushed me to be brave entering UTM and stands behind my back every time I may
feel like giving up during this study.

v
 ABSTRACT

Sensory integration problems are common in autistic children, and they may
lead to developmental, information processing, and behavioral disorders. When they
are given sensory input, their brains react in unexpected ways, either failing to
integrate or organize incoming information properly, causing some difficulties in
making connections between their tactile, vestibular, and proprioceptive sensory
systems (muscle and joint movement), all of which can be overactive or underactive
as they interact with their environment. Multisensory simulators may help autistic
children who are easily distracted improve their concentration, attention, and alertness
in this way. This method may succeed because children with autism spectrum disorder
(ASD) like and are interested in using computers or other electronic devices when they
are in a safe and secure environment. As a result, the purpose of this work is to give a
grounded theory on the use of technology to promote concentration in children with
ASD, where the creation, assessment, and deployment of systems and apps are all
extremely promising. The utilization of technological breakthroughs such as
Augmented Reality (AR) and the Leap Motion Controller (LCM) to create an
atmosphere of social interaction and learning amongst individuals in the form of social
storytelling may surely offer a relaxing setting. The game item with a three-
dimensional model on the social story may also be manipulated with motions of both
hands using the Leap Motion Controller. After a number of steps, a hybrid of
Augmented Reality and Leap Motion Controller systems was created, allowing users
to interact with virtual items. As a result, combining the Leap Motion Controller with
Augmented Reality may encourage continuous learning and be useful as a therapeutic
tool for children with autism.

vi
 ABSTRAK

Penyepaduan deria sering menjadi masalah bagi kanak-kanak autistik, yang

boleh menjadi punca masalah perkembangan, pemprosesan maklumat dan tingkah
laku. Apabila input deria diberikan, otak mereka bertindak balas dengan cara yang
tidak dijangka, sama ada gagal untuk mengintegrasikan atau menyusun maklumat
yang masuk dengan betul, menyebabkan beberapa kesukaran dalam membuat
perkaitan antara sistem deria sentuhan, vestibular dan proprioseptif mereka
(pergerakan otot dan sendi), yang kesemuanya boleh menjadi terlalu aktif atau kurang
aktif semasa mereka berinteraksi dengan persekitaran mereka. Dalam hal ini, simulator
multisensori boleh membantu dalam peningkatan fokus, perhatian dan kewaspadaan
pada kanak-kanak autisme yang sering terganggu. Strategi ini mungkin berkesan
kerana kanak-kanak yang mengalami gangguan spektrum autisme (ASD) suka dan
berminat semasa melibatkan diri dengan komputer atau peranti elektronik lain, selagi
interaksi berlaku dalam persekitaran yang selamat dan terjamin. Hasilnya, tujuan kerja
ini adalah untuk memberikan teori asas tentang penggunaan teknologi untuk
menggalakkan penumpuan pada kanak-kanak dengan ASD, di mana penciptaan,
penilaian dan penggunaan sistem dan apl semuanya amat menjanjikan. Penggunaan
penemuan teknologi seperti Augmented Reality (AR) dan Leap Motion Controller
(LCM) untuk mewujudkan suasana interaksi sosial dan pembelajaran di kalangan
individu dalam bentuk penceritaan sosial pastinya menawarkan suasana yang santai.
Item permainan dengan model tiga dimensi pada kisah sosial juga boleh dimanipulasi
dengan gerakan kedua-dua tangan menggunakan Leap Motion Controller. Selepas
beberapa langkah, gabungan sistem Augmented Reality dan Leap Motion Controller
telah dicipta, membolehkan pengguna berinteraksi dengan item maya. Akibatnya,
menggabungkan Leap Motion Controller dengan Augmented Reality mungkin
menggalakkan pembelajaran berterusan dan berguna sebagai alat terapeutik untuk
kanak-kanak autisme.

vii
 TABLE OF CONTENTS

TITLE PAGE

DECLARATION iii
DEDICATION iv
ACKNOWLEDGEMENT v
ABSTRACT vi
ABSTRAK vii
TABLE OF CONTENTS viii
LIST OF TABLES xi
LIST OF FIGURES xii
LIST OF ABBREVIATIONS xiii

CHAPTER 1 INTRODUCTION 1

1.1 Problem Background 1

1.2 Problem Statement 3

1.3 Research Objectives 3

1.4 Scope of Study 4

1.5 Significance of Study 4

CHAPTER 2 LITERATURE REVIEW 6

2.1 Autism Syndrome Disorder 6

2.1.1 Attention Behavior in Autism 6
2.1.2 Social Interaction in Autism 8
2.1.3 Motor Development in Autism 8

viii
 2.2 Augmented Reality 9
2.2.1 Augmented Reality in Stimulating Autistic
Children 10

2.3 Interactive Display 11

2.3.1 Unity Game Engine 11
2.3.2 Vuvoria SDK 11

2.4 Social Story 12

2.4.1 Social Story in Stimulating Autistic Children 12

2.5 Hand Gesture Recognition Technology 13

2.5.1 Hand Gesture in Stimulating Autistic Children 13
2.5.2 Leap Motion Controller 14
2.5.3 Leap Motion Development Kit In Unity 15

2.6 Previous Research and Proposed Method 16

CHAPTER 3 RESEARCH METHODOLOGY 20

3.1 Introduction 20
3.1.1 Operation Framework 21

3.2 Development of Multisensory Simulator 23

3.2.1 Story Ideation 25
3.2.2 User Flow 25
3.2.2.1 Flow of Help Menu 26
3.2.2.2 Flow of Setting Menu 27
3.2.2.3 Flow of Exit Menu 28

3.2.3 Three-Dimensional Object Design 28

3.2.3.1 Characters Design 28
3.2.3.2 Game Design 29

3.2.4 Movement and Hand Gestures 30

ix
 3.3 Usability Testing 31

3.4 Gantt Chart 31

3.4.1 Gantt Chart FYP 1 33
3.4.2 Gantt Chart FYP 2 34

CHAPTER 4 EXPECTED RESULT 35

CHAPTER 5 CONCLUSION 36

REFERENCES 37

x
 LIST OF TABLES

TABLE NO. TITLE PAGE

Table 2.6 Previous Research and Proposed Method 16-19

Table 3.4.1 Gantt Chart FYP 1 33
Table 3.4.2 Gantt Chart FYP 2 34

xi
 LIST OF FIGURES

FIGURE NO. TITLE PAGE

Figure 2.5.2a Leap Motion Controller Hardware 14

Figure 2.5.2b Internal Schematic of Leap Motion Controller 14
Figure 2.5.2c Interaction Area of Leap Motion 14
Figure 2.5.2d Hands Gesture 15
Figure 2.5.3 Unity Coordinate System 16
Figure 2.6a AR textbook of New Ukrainian school 17
Figure 2.6b Marker used in the MARVoc 17
Figure 2.6c Construct a word 18
Figure 2.6d Character Model at Run Time 19
Figure 3.1.1 Framework of The Entire Study 21
Figure 3.2 General Architecture of The Whole System 23
Figure 3.2.1 User Position 24
Figure 3.2.2 User Flow of the Multisensory Simulator 26
Figure 3.2.2.1 Flow of the Help Menu 27
Figure 3.2.2.2 Flow of Setting Menu 27
Figure 3.2.2.3 Flow of Exit Simulator 28
Figure 3.2.4a Hands Position While Using LMC 30
Figure 3.2.4b Hands Movement While Using LMC 31

xii
 LIST OF ABBREVIATIONS

ASD - Autism Spectrum Disorder

GA - Three Dimensional
PSO - Augmented Reality
MTS - Leap Motion Controller
MD - Virtual Reality
WHO - World Health Organization
UTM - Universiti Teknologi Malaysia
API - Application Programming Interface
SDK - Software Development Kit
JPG - Joint Photographic Recognition
PNG - Portable Network Graphics
PA - Physical Activity
HGR - Hand Gesture Recognition
TUI - Touchless User Interface
UI - User Interface
3D - Three-Dimensional
2D - Two-Dimensional
OS - Operating System

xiii
 CHAPTER 1

INTRODUCTION

1.1 Problem Background

Background noise, bright lights, hunger, or the highly intriguing thoughts

rushing in their minds sometimes may make it difficult for children to filter out
distractions, or they may get bored and lose interest [2]. So, they become unable to
focus. Meanwhile, the problem might get worse in autistic children. ASD (autism
spectrum disorder) is a lifelong, non-progressive neurodevelopmental disorder that
typically begins in childhood and is diagnosed before the age of six [4]. According to
the World Health Organization (WHO), one in every 270 people are on the autism
spectrum disorder by 2021, whereas in Malaysia, the prevalence of autism spectrum
has been observed to have grown over time, which is likely due to increased awareness
of the condition among parents nowadays [35].

Aside from the fact that ASD is highly heritable and is considered a brain-
based condition, the biological underpinnings of the disorder are mostly unclear, which
typically leads to lifelong impairment in functioning and social adaptation issues [7].
It seems that focusing on tasks that needs to share attention [22], such as reading a
book, playing puzzles, or simply crossing the street safely, is more difficult for children
with ASD and even is worsened by their inability to make eye contact while having a
conversation.

In addition, the variety of sensory input in a normal environment may be

difficult for children with ASD, and past research shows that early therapies and
approaches may be introduced to help children with ASD navigate a world that relies
on multisensory processing [1]. During each episode, children should be exposed to a
variety of modalities, including auditory, visual, tactile, and kinesthetic, as well as

1
numerous brain functions including face and emotion processing, language processing,
and emotional responses [1].

Unfortunately, the latest COVID-19 has worsened ASD-related symptoms.

Quarantine has been imposed on people all over the globe, preventing them from
engaging in physical activity (PA) for many weeks to months, causing total disruption
of daily routines for most school-aged children [36, 37]. Children with ASD, who tend
to routines and schedules, may experience significant stress as a result of such
disruptions in schooling, treatment, and everyday activities [37, 38].

Providentially, a viable treatment option for improving the quality of life and
for minimizing the issues mentioned above towards the children with ASD, caregivers,
and parents have become accessible throughout the interventions on the use of
technology that has been skyrocketing during the past few decades [8]. One of the
interventions offered is Augmented Reality (AR) and Leap Motion Controller (LMC)
that have been used to treat a variety of neurodevelopmental problems, including ASD.
Specifically, AR has proven some benefits over more conventional therapies in this
area, as it offers children with ASD to be treated in more realistic surroundings and
customized to the distinct and varied traits that children with ASD have [12].

That’s why in controlled environments, this kind of activity may possibly allow
more possible ecological validity and the skills learned would be generalizable to other
areas as well as everyday life contexts [13,14], which is depicted in the form of social
stories where things are taught how to behave properly in social situations. According
to the findings of the research, social story approaches may aid to increase social
interactions and their focus, so that such social isolation and academic difficulties can
be avoided [23].

Additionally, research on applications in the form of a Leap Motion Controller

has also been proven in supporting motoric development when direct hand contact or
touch is no more required, only hand and finger movements in an open-air as the input
[24]. In comparison with (VR) Virtual reality, the contact often requires the use of
specialized tools such as headsets and goggles, which many children with ASD may

2
find challenging to utilize. On the other hand, AR technologies are simpler and more
versatile since they can be utilized without additional tools and are believed to have
better adaptation for interaction to the real world [11].

1.2 Problem Statement

Because ASD children have a lower disease burden from COVID-19, it is clear
that they are suffering from the pandemic's adverse effects. The suspension of in-
person education, social activities, and basic healthcare puts children's physical and
mental wellbeing at risk [38]. It's because the very nature of social isolation during the
COVID-19 pandemic makes this very hard to engage with anyone other than their
family members, whereas children with ASD benefit from regular daily routines,
which COVID-19 has completely prohibited [9]. Furthermore, social activities at
school or in healthcare facilities may have a more soothing and regulating effect, since
they may not have access to indoor or outdoor settings for such activities while isolated
at home [39].

Furthermore, mandated mask restrictions are difficult for any child who seems
to have sensory sensitivity and may not understand why they are necessary [9]. As a
result, they communicate their discomfort with the pandemic's uncertainty via
aggression, tantrums, or refusals to engage in normal activities. Thus, through this
paper, Augmented reality to bring up social stories is proposed to be used as a
therapeutic medium for the autism children in primary school age. In addition,
practicing hand movements using Leap Motion Controller can help the autism children
in receiving instructions and train motor nerves.

1.3 Research Objectives

This study has two main objectives:

3
(a) To develop home-based activities for autistic children as a part of therapeutic
necessities of communication and functional skills using Augmented Reality
and Leap Motion Controller

(b) To improve core deficits in social communication and interactions through

social stories driven by Augmented Reality and stimulate psychometric and
behavioral domains in autistic children through Leap Motion Controller

1.4 Scope of Study

The scope of this study covers a smart real-time application that is designed
with today's technological sophistication to make therapy easier for children with ASD
at home. So then expectedly, the social communication of ASD children and
interactions can be improved through a method where a social story is driven by
augmented reality that offers a realistic and lifelike playing experience. Besides that,
because the social stories being played can also be controlled by using Leap Motion
Controller, the psychometric and behavioral domains in autistic children may be
stimulated as well.

The system that is used to train their optical, acoustic, and tactile must be in a
fixed position, where the leap motion controller is below the hands of the user to detect
hand and finger movements. Then, the social stories to be followed are simulated
through 3-D contextual backgrounds on a screen by using Unity 3D. Overall, this
system is addressed for the ASD children in the age of 5-10 years old since expectedly
they can receive instructions with accompanying parents or any people at home.

1.5 Significance of Study

The significance of the study lies in helping children with autism to receive
instructions, communicating in simple terms, and having their motor nerves trained

4
during the limited condition and limited space, where they only can do such activities
at home due to Covid 19 pandemic surges.

Besides that, when this social story platform that is viewed through Augmented
Reality and controlled by Leap Motion Controller is ready and the test is precise, the
method can be believed to be very efficient and productive for autistic therapy in the
future.

5
 CHAPTER 2

LITERATURE REVIEW

2.1 Autism Syndrome Disorder

Autism spectrum disorder (ASD) is an umbrella term for a range of

neurodevelopmental diseases characterized by impaired speech, a lack of social
interaction, and limited, repetitive, and stereotyped behaviors [3]. Although autism has
been recognized as a distinct disorder that has been identified early in life since the
1960s, its exact cause is still unclear. However, what is certain today is that it is caused
by the combination of nature and nurture. Numerous biological derangements (nature)
are favored by the unfavorable neuro-pathological influence of many environmental
risks (nurture) due to genetic sensitivity in the form of polygenes [17].

ASD has a substantial influence on children's social life [5,6], and it may have
long-term negative consequences in a variety of contexts such as peer social
interaction, cognitive ability, daily living skills, school performance, and mental health
[6,7]. As a result of this diagnosis, children with ASD need effective therapies that
address both the core symptoms of the illness and the disorder's related disease
characteristics.

2.1.1 Attention Behavior in Autism

When we pay attention, we focus on one thing while ignoring other things or
background noises, and then shift to something else when desired. It means that paying
attention involves being aware of what we need to pay attention to in order to sort out
the relevant information from our surroundings and put it together [2]. Due to that,
paying attention becomes a key skill for learning. For example, children need to pay

6
attention to a teacher’s instructions to be sure that they can learn and do things the right
way.

After all, people with autism spectrum disorder (ASD) most often fail to attend
to salient behaviorally-relevant information in their environment, such as when their
name is called, but seem to be distracted by subtle behaviorally-irrelevant details in
their surroundings, such as light shining through blinds or the sound of air flowing
through a duct [31]. Several empirical accounts of attention in ASD have portrayed
people as being both overly focused and easily distracted [51]. Aside from that, paying
attention is more difficult when they are tired, uncomfortable, or just uninterested in
the ongoing task they are doing [22]. It can even get worsened when they are unable
to do the task or have been sitting still for too long, and if there are any distractions are
going on around them [7].

Moreover, in the case of giving them more than one type of stimulus, they may
get more distracted as had been proven by one research that tested ASD children to
have a vibrated hand-held device while ignoring a task-irrelevant, a beam of flashlight
flashes. As a result, when they were required to lift their toes for a single pulse and
their heel for a double pulse, they made more mistakes; the visual stimulus was
incongruent with the tactile vibration, e.g., single vibration, double flash [42]. While
some interference of concentration and impulsivity that happened in daily life may
lead them to behave like having difficulty in learning to tie shoelaces, brushing teeth,
cutting food, and buttoning up their clothes [46].

However, autistic children are still able to pay attention if only they get some
practice and do interesting activities with clear endpoints and effective instructions,
for instance, engaging them to do some model tasks like making a bracelet from a
piece of string and beads [40]. Nevertheless, these can make it difficult for them if they
don’t know what they need to do and when to finish [9].

7
2.1.2 Social Interaction in Autism

Social skills teach children how to act in a variety of social contexts, from
talking to their parents to playing with their peers at school. Social skills may assist
children in making friends, learning from others, developing hobbies and interests, and
feeling a sense of belonging [43]. Thus, good social skills are important to the
children’s overall quality of life.

Alas, children with ASD often struggle with back-and-forth communication in

a conversation, where they may not wait their turns and think that the conversation is
done [16]. Thus instead, they will interrupt the person who is still talking to them and
walk away. Many of them also have a very limited understanding or no understanding
at all of someone else's feelings and ideas [2]. They have little understanding of how
their words and actions may influence others; for example, they might not understand
the purpose of waving hello or goodbye to someone, have trouble maintaining eye
contact with others during conversation, or may not make any facial emotions [16].
Typical deficiencies include a lack of proper greeting behaviors and an inability to
recognize the presence of familiar people [44].

Consequently, children with ASD who lack social skill pose significant
problems for schools, teachers, parents, and peers across disciplinary, instructional,
and interpersonal domains [43]. They tend to become indifferent to their surroundings
and often seem to prefer being alone. Due to that, some strategies and practices are
suggested for building up the children’s social skills, for instance, social skills training,
social stories, video modeling, and role-play [44].

2.1.3 Motor Development in Autism

Motor skills are the ability of a person to perform easy and difficult movements
that involve various muscle actions [46]. And the motor skills of autistic children are
unlike common children. Children with ASD face significant stress when it comes to
imitating motor standards; they learn to imitate gestures and the use of objects slowly

8
[44]. There is also a deficit in enthusiasm and overall ability to learn new abilities,
which contributes to uneven motor skill development. This matter is due to system
interference of the central nervous system [3]. Therefore, children with autism
generally have motor skills that are lower than their fellows in common, and the more
serious their disorder, the slower their development in being able to grasp things and
move around [45]. All in all, ASD exhibits persistent difficulties in a wide range of
motor behaviors including fine and gross motor abilities as well as postural control
[46].

Fine motor skills relate to one’s capability with less exertion or minimal
muscles involvement [46] such as to make hand movements, but it also typically
involves complex object interaction and looks more vulnerable to delay in ASD
relative to general gross motor behaviors, such as walking [47]. Typically, the
deficiencies in gross motor development are indicated by the inability to do motor
activity for the tasks that require fast movement or complex motor coordination [49].
In addition to clumsiness in overcoming obstacles, poor control balance, lack of
coordination, energy, and muscle strength [50]. On top of that, most children with ASD
also show such stereotypes, for example, clapping and shaking the body, or
hyperactivity, nonetheless, hypoactivity may also occur [46].

2.2 Augmented Reality

Augmented reality technology is a recent VR (virtual reality) modality that

enables interaction with the actual environment and is not as artificial as Virtual
Reality [13]. Augmented Reality augments reality visuals and sounds by merging
virtual and real elements to create a mixed and interactive 3D world, adding virtual
computer-generated perceptual information across various sensory modalities such as
visual, aural, haptic, somatosensory, and olfactory.

The primary benefit of augmented reality is how digital world components

blend into a person's perception of the actual world, not as a modest display of data,
but through the integration of immersive sensations, which are perceived as natural

9
parts of an environment. So then, the entertainment technology offered by augmented
reality can create environments that manage and minimize the anxiety caused by real
social situations [9].

2.2.1 Augmented Reality in Stimulating Autistic Children

Considering certain characteristics that have been demonstrated by several

research, the method of implementing augmented reality may be effective for kids with
ASD. They discovered that augmented reality apps not only enhance sense of social
skills and new methods of learning among people with ASD, but also provide them
with an interesting and cognitively demanding experience [19].

AR is also able to assist the way of children with ASD learn daily life skills in
a form of the visually displayed information, as well as to facilitate the understanding
of social-communicative behavior, improve attentional abilities, and aid in the
perception of facial expressions, among other benefits [20]. The higher degree of
realism provided by AR aids in the promotion of a broad variety of abilities that aid
the quality of autistic children’s life, enabling further approximation to the interactions
with the real world [14].

Furthermore, certain people with ASD may have a natural tendency to utilize
digital tools, a fondness of electronic media, and a preference for game-like features.
This form of technology may help people with autism cope with the stress that real-
world social interactions may bring up [20]. As a result, AR enables the creation of
exciting learning aids which help children with ASD in remaining focused,
encouraging activity engagement, on-task behavior, motivation, and positive emotions
that enhance the learning process [29].

10
2.3 Interactive Display

2.3.1 Unity Game Engine

Unity 3D is a multiplatform game engine for making games easy, even for
beginners because it provides the core functionality needed to create three-dimensional
(3D) and two-dimensional (2D) games, including approximately 90 percent on
emerging augmented reality platforms and the engine provides a basic scripting API
in C# for both the Unity editor in the form of plugins and the games itself, as well as
drag and drop features [52]. Unity 3D is free with high optimization and can be used
on PC, Nintendo Wii, PS3, Xbox 360, iPad, iPhone, and Android. Unity 3D uses a free
navigation system in-game creation so that each side of 3D during the object creation
process can easily be viewed [25].

2.3.2 Vuforia SDK

Vuforia is a development kit developed that supports the use of augmented

reality and is applicable for Android, IOS, and Windows-based computerized systems.
Vuforia plugin of Unity 3D provides the necessary API for detecting planes and object
placement in AR environments, scanning text, recognizing marker frames, virtual
buttons, identifying object surfaces intelligently, scanning cloud-based, recognizing
image targets, and recognizing predefined target objects [34]. It supports target
tracking, with the concept of the target divided into two main classes: Traceable and
Trackable. Traceable has a reference to matching a status, targets, and pose matrix,
whereas Trackable is defined and currently has four subtypes: Image-Target, Object-
Target, Multi-Target, and Cylinder-Target [25].

The focus of the image target doesn't require a specific code or black and white
form to be recognized since it will render the 3D object over the detected target, as the
tracking of the image is targeted from the device’s camera, calculate the target's
orientation and position, and render the 3D object over the detected image in real time

11
using JPG or PNG input images [34]. Then after being extracted, these images will be
stored in a database as a unity3D package.

2.4 Social Story

A social story is a short real-life story devised by Carol Gray in 1991 that can
support the safe and meaningful exchange of information between parents,
professionals, and people with autism spectrum disorder [15]. Social stories can be
written or visually displayed to help them learn socially appropriate behavior and
responses in different situations which for each give very specific step-by-step
information for that situation by considering the “who,” “what,” “where,” “when” and
“how” components [33]. Thus, a social story is considered accurate, convincing, and
easy to understand. All in all, the purpose of a social story is to minimize potentially
perplexing instructional interactions and provide direct access to social information,
so then it allows the children with ASD to improve their understanding of an event or
expectation [33].

2.4.1 Social Story in Stimulating Autistic Children

Since children with autism often fail to benefit from traditional social-skill
instruction due to deficits in communication skills [15], to stimulate their social
responses is by having a social story that is written based on a particular situation of
concern and using language that matches the age or skill of the child [30].

The rationale behind Social Stories is that they can provide autistic children
with various social interactions, situations, skills, concepts, and common social
behavior they are lacking, moreover when the social story is drawn visually [31],
where the visual learning method itself is one of the strengths for many autistic
children [8]. Therefore, they will get eased in interpreting and understanding the social
situations and in determining a response appropriately in specially designated styles
and formats only after they have been thoroughly observed [31]. Such as, for ASD

12
children who often face frustration and anger, the social story shall provide
information about how they ought to respond to their behavior [25].

2.5 Hand Gesture Recognition Technology

In head-mounted Augmented Reality (AR) systems, hand gesture recognition

(HGR) is a primary input method. Gesture recognition is classified as a type of
touchless user interface (TUI), which, unlike touchscreen devices, is controlled
without the use of touch. The use of gesture recognition technologies allows detecting
specific physical gestures in order to operate an interface, including hands and fingers
where this part of the body is also considered an important part of the fields of human-
computer interaction [21]. With the goal of Human-Computer Interaction being to
improve communication between users, dynamic hand gestures are important for
recognizing key hand movements such as waving the hand, for users to communicate
effectively [21].

2.5.1 Hand Gesture in Stimulating Autistic Children

Hand gesture is one of the physical activities (PA) engagement used to

communicate about non-present objects in mind among children with autism spectrum
disorder, where PA itself has been identified as an important goal for kids with autism
[32]. Studies have suggested that intense exercise for more than 20 minutes may
effectively reduce stereotyped behaviors, hyperactivity, aggression, and tension, as
well as enhance cognitive performance. [5]. Thus, it is encouraging that many
educational programs provide brief periods of physical exercise for their students
where a few aspects also need to be considered: the condition or mood of the child,
tools and equipment, atmosphere, and playtime.

13
2.5.2 Leap Motion Controller

The Leap Motion Controller is an infrared light-based stereoscopic camera

designed to track hand and finger movements in the open air and can be used for
human-computer interaction in the form of a small USB [18]. The majority of research
on the Leap Motion Controller falls heavily on the robustness of gesture capabilities
and the fundamental problems associated with body movements or gestures. This Leap
Motion device can be used with Microsoft Windows, Mac OS, and Linux.

Figure.2.5.2a Leap Motion Controller Hardware [23]

Figure.2.5.2b Internal Schematic of Leap Motion Controller [23]

It can capture the user's hands and fingers by illuminating the close area near
the cameras with infrared light and identifying the location and orientation of the hand
and its fingers. Tracking quality might be affected by strong sunlight or low lighting
conditions. The controller's action area is about 150° by 120° wide and extends around
80 cm from the device [18].

14
 Figure.2.5.2c Interaction Area of Leap Motion [18]

Tracking precision has been shown to be remarkably excellent when the

standard deviation is less than a millimeter, though a precision drop has been seen with
increase in distance [23, 24]. Although the camera can’t perceive occluded parts of the
hands, the algorithm can estimate normal hand motions to some degree, and hence the
device operates best when the whole hands are exposed.

Figure.2.5.2d Hands Gesture [23]

2.5.3 Leap Motion Development Kit in Unity

The Leap Motion controller system consists of hardware and software that runs
on the host computer. The software component analyzes images collected by the
hardware and transmits tracking data to apps. To get data, the Leap Motion Unity
plugin that’s connected to this service includes scripts to convert Leap Motion
coordination to the Unity coordination system. These scripts and additional graphic
assets make it easy to add the motion-controlled hands to a Unity scene [21].

15
 Figure.2.5.3 Unity Coordinate System [18]

The Unity left-handed coordinate system is placed on the Leap Motion device
in its desktop orientation in Figure.2.5.3 above. Unity has a left-handed coordinate
system, but the Leap Motion API has a right-handed coordinate system (the z-axis
points in the other way) [21]. Unity also has a default unit of measurement of meters,
but the Leap Motion API has a unit of measurement of millimeters. Internally, the
plugin scripts translate the tracking data to utilize a left-handed coordinate system and
scale distance values to meters.

2.6 Previous Research and Proposed Method

Research Topic Researcher Year Description

Augmented Liubov F. 2020 When the pages of the storybook are

Panchenko, seen through a handheld Augmented
reality books:
Tetiana A. Reality display, the characters told in
concepts, Vakaliuk, will appear out of the pages in a form
Kateryna V. of 3D virtual to make the story an
typology, tools
Vlasenko immersive virtual environment. In
addition, the users can interact with
the characters in the story [26].

16
 Figure.2.6a AR textbook of
New Ukrainian school

Mobile Kamran 2020 Mobile augmented reality program

that teaches children with ASD 209
Augmented Khowaja, Dena
vocabulary from 11 categories (fruits,
Reality App for Al-Thani, animals, birds, etc.) through an app
for tablet PCs or smartphones with
Children with Asma Osman
cameras and AR-API compatibility.
ASD to Learn Hassan, The MARVoc can be utilized in a
variety of situations, including the
Vocabulary Asadullah Shah,
classroom, at home, or outside, where
(MARVoc) Siti Salwah the children's performance can be
monitored online. [27].
Salim

Figure.2.6b Marker used in the

MARVoc

17
 Figure.2.6c Construct a word

An Augmented Hawar Bahzad 2019 To use Augmented Reality systems in

Reality system Ahmad assisting children with ASD to
to enhance facial understand facial expressions better
expressions with six models of basic human
recognition in emotions or facial expressions by
autistic children overlaying suggested 3D objects over
the physical environment.
Expectedly, this way can be more
interesting and more attractive
compared to emotion cards in
improving their social skills [48].

18
 Figure.2.6d Character Model at Run
Time

A social story Farah Fadhilah 2021 A controllable simulation of the real-

that is played Hermawati life game that is based on a social
through story and displayed in reality mode
Augmented through Augmented Reality, where
Reality and Leap the purpose is to stimulate the
Motion attention of children with ASD by
Controller approaching their multisensory
system, either towards the
communication and sensorimotor,
since it is also provided with some
motoric games through the function
of Leap Motion Controller.

Table.2.6 Previous Research and Proposed Method

19
 CHAPTER 3

RESEARCH METHODOLOGY

3.1 Introduction

This chapter describes and illustrates the approach and flow of the work that
has been executed and applied to this project, from the starting point through a series
of steps to a successful outcome and final action, so that the proposed simulator for
autistic children can finally be achieved. Firstly, it started with the operation
framework, which outlines and explains the overall procedures has been done
throughout this project. After that, the next chapter is followed by the development
stage which includes a list of features with the idea for the social story to be played in
the game, prototyping, and implementation where the application is about to be ready
to be continued on the stage of testing. Last but not least, the last subchapter is
reviewing and focuses on the Gantt chart planning for both the first and second
semesters.

20
3.1.1 Operation Framework

Figure.3.1.1 Framework of The Entire Study

A flow chart is used to show concisely the set of procedures demonstrating the
system workflow used in this project’s system. Steps that have been taken for the
operation framework in developing a home-based application for autistic children

21
multisensory stimulation shown in the flowchart below. The steps that are highlighted
with three different colors indicated each of the objectives of this study. The red color
is done to achieve the first point, while the blue one is to obtain the second, and the
yellow one is for the third point.

The operation framework begins with problem identification based on the

approved title for this final year project, as shown in Figure 3.1.1 above. Furthermore,
it is followed by the writing of the FYP proposal, which consists of five chapters:
introduction, literature review, research methods, anticipated findings or actual result,
and comes to the end, conclusion. Aside from that, the operation framework flowchart
follows with the literature review on related scientific research articles or journals, and
also some books including the benchmarking to measure products, services, and
processes against the existing method in the society. Yet in fact, the literature review
is actually done throughout the overall project since every single thing written
throughout this study needs evidence or facts to support it.

On top of that, the flowchart is followed by the development process that is

very crucial for the implementation of the proposed ASD simulator home-based
application where its step will involve the ideation of the social story, prototyping, and
implementation. After that, prior to the designated simulator can be used and to be
tested towards the participants, it is required to undergo results extraction and analysis.
Therefore, any bugs or errors that may detain the running process can be scanned and
if that’s so, the simulator then can be analyzed back by the software used to develop
them.

Subsequently, testing needs to be conducted towards the addressed participants

to identify any usability problems and exactly determine the participant’s satisfaction,
like in which parts of the design frustrate people, where they get confused, and what
keeps them from converting. Besides that, it is also to collect qualitative and
quantitative data in order to improve an evaluation by ensuring that the limitations of
one type of data are balanced by the strengths of another. To run an effective usability
test, a solid test plan, invite participants, then analyze and report the findings are the
most important things to be aware of.

22
 Following that, when the data is successfully obtained, the result and
discussion will be conducted to discuss and address the overall result reached in this
project. Last but not least, the conclusion is aimed to wind up the project.

3.2 Development of Multisensory Simulator

Figure.3.2 General Architecture of The Whole System

In the proposed system, its general architecture is divided into three parts which
are input, process, and output:

1. Input

While capturing the object in real-time, several things need to be considered

which are the quality of the web camera, lighting, and distance to capture the object.
Then, markers are placed on each sheet of the storybook that is used as a base to display
the 3D animation. Under those circumstances, the Leap Motion Controller is then
placed under the hand to control the 3D animated objects.

23
 Figure 3.2.1 User Position

2. Process
a) Identify Markers
The markers on the book sheets will be detected and recognized by the camera. The
hand movement above the leap motion will be detected by the leap motion tool
controllers. Then both will be reviewed and positioned in 3D orientation as virtual
objects on top of the marker image.
b) Database Checking
After that, the bookmarkers will be adjusted by the markers data that has been stored
in the database, as well as the hand gesture adjustments. If the information matches
then it will continue to the estimation of the placement of the virtual object on top of
the markers in the storybook.
c) Positioning Object
At this stage, the virtual coordinates X (right to left position or the opposite), Y (front
to back position or the opposite), and Z (up to down position or the opposite) on the
markers is to define the laying position of the virtual object that corresponds to the
location of the marker on the book’s sheet.
d) Rendering
After that, the process of the graphical appearance of the entire modelling process is
carried out by computer animation. In rendering, all the data that has been input in the
process of modelling, animation, texturing, and lighting with certain parameters will
be translated as the output (the final look of the animation model).
e) Interaction between 3D objects
At this stage, the 3D objects in the social story animation will move according to the
instructions given at the time the input was performed.

3. Output
The output to be generated is in the form of an updated movable 3D object and
displayed in the Augmented Reality which is placed above the bookmarker.

24
3.2.1 Story Ideation

Depiction of story ideas that will be displayed in the form of animation in this
study appointed based on the difficulties of children with autism in social interaction.
And in order to be enjoyed by children, the use of simple stories, language, and light
delivery methods is adopted. The story that will be told consists of four main topics
which are:

1. How to Get Acquainted

The story begins with a child who is meeting with another child in public. One child
invites another child to get acquainted by greeting, handshake, and introducing
themselves.

2. How to Invite New Friends to Play

After getting to know each other, one of the children invites another child to play
together.

3. Winning and Losing

By the end of playing the game, since there will be two possibilities of passing or
repeating the game, the users are directed on how it feels to win or lose. It feels bad to
lose but it also offers suggestions for what to do, for example, the story ends by telling
the users to say, "That was fun!" and ask to play the game again.

4. Saying Thank You and Good-Bye

After all, he or she will say thank you for having played together to show off as having
manners, and being polite when they have to go back or leave.

3.2.2 User Flow

User flow represents the workflow or process from the user's perspective,
where figure 3.2.2 below shows the flowchart of the user when using the application
of the autistic multisensory simulator. User flow is required to be done during the

25
development to layout the user's movement through the product, mapping out every
step the user takes. This written outline can also do wonders for helping to catch errors.

Figure.3.2.2 User Flow of the Multisensory Simulator

3.2.2.1 Help Menu

The flow of the help menu on this application describes the process of system
activities in displaying information on the help menu. When the user runs the system,
the system will perform the process of starting the application and display the main
menu display. If the user selects one of the available menus which is the help menu,
the system will display a help panel display with information on how to use the
application. On this panel, there is a home icon button where if the user selects the
button will make the system return to the main menu.

26
 Figure.3.2.2.1 Flow of the Help Menu.

3.2.2.2 Setting Menu

The flow of the setting menu on this application shows the supporting game
system which is the sound system and music system. Both the sound system and music
can be turned on and off. On this panel, there is a home icon button where if the user
selects, the system will return to the main menu

Figure.3.2.2.1 Flow of Setting Menu

27
3.2.2.3 Application Exit

When the user starts the application, the system will carry out the process of
running the application and displaying the main menu display, and if the user selects
one of the available menus which is the exit, the application will close down from the
application’s main menu display.

Figure.3.2.2.1 Flow of Exit Menu

3.2.3 Three-Dimensional Object Design

The animation design process is carried out using the unity software and its
additional components or packages installed. In addition to the Vuforia SDK that will
make the design process easier and faster to result in applications that involve
Augmented Reality technology where it helps to scan objects surfaces, have virtual
buttons, recognize marker frames and the image targeting.

3.2.3.1 Character Design

The three-dimensional characters used in this study are the girls and boys in
primary school age, around 10 years old. Along with the simulation of the social story,
these characters will be interacting with each other to show the users how to
communicate in a way that these characters show. Three-dimensional objects will be
designed using Unity 3D application before being displayed in Augmented Reality. In

28
planning the three-dimensional object characters, there are several steps of work to
provide maximum results.

Designing the body of a three-dimensional character starts by observing the

initial view of the model body. In this study, the model used is a model that has the
posture of a 10 years old kid. Thus, adjustments are made to height, weight, and other
looks on the early design. Following this, adjusting the body shape and facial
expressions, the characters need clothing that can support their looks as a primary
school kid.

3.2.3.2 Game Design

The types of games carried out in this study were decided based on children's
difficulties in receiving instructions and training motor nerves. Games that involve
simple and less complicated movements are adapted to the ability level of children
with autism in the primary school range of age. The game types that will be designed
are:

1. Moving and parking cars to the correct lining, that is to strengthen the coordination
ability between hands and eyes. As well as improving hand dexterity when moving
their cars.

2. Find the differences between two objects that will teach children to categorize and
organize different objects since they have to encourage children to pay close
attention to detail.

3. Sorting and matching games, by putting various-shaped things into the box based
on their color and shape (classify objects that are grouped by similar traits). It is to
improve cognitive skills and problem-solving as they have to recognize and
memorize various patterns and colors.

29
3.2.4 Movement and Hand Gestures

As with a mouse and keyboard, the Leap Motion Controller can replace the
function of those tools, by placing both hands in the air right above the Leap Motion
Controller that has already been connected via USB to the computer. On the computer
screen, the user will see the three-dimensional hand model.

Figure.3.2.4a Hands Position While Using LMC [35]

Unity 3D as a system builder is used as the game engine, it acts as a merger of

three-dimensional models and media for running Leap Motion that uses the C#
programming language. The game engine will be processed after the presence of hand
and finger movement data captured by Leap Motion. The script that has already been
stored in Unity will be run after the data is collected in advance. The list of hand
movement data used as system control consists of:

1. Swipe gestures
By doing right or left-hand movements to shift in the direction of the x-axis and y-
axis. Having the palm of the hands opened to move the objects according to the
direction of the game.
2. Hand gripping movements of the right hand or left hand
To lift the object in the direction of the x, y, and z axes with the palm clenched as
if picking up objects.

30
 Figure.3.2.4b Hands Movement While Using LMC [35]

3.3 Usability Testing

The purpose of usability testing is to examine the level of system use approval.
The designated system will be tested towards several children with autism who are in
therapy at the age of 5-10 years old. They will be accompanied and directed while
using the system by the therapist or their parents. Test results that need to be obtained
mainly are:

1. Implementation of AR Camera on marker detection done with a distance and

certain conditions.
2. Augmented Reality and hand motion reading with leap motion work well
together.

Along with that, the therapist will be requested to complete a questionnaire

with questions and statements on the degree of conformity treatment to the system. All
questions and statements are paired. After testing the application, the results are stored
as a proof or verification that the application has successfully passed the test phase.

3.4 Gantt Chart

A Gantt chart is a common and effective graphical tool that depicts the
activities or tasks performed against time. It is known as the visual presentation of a
project where it trains and drills down activities that need to be done within a fixed

31
period. In short, a Gantt chart depicts or portrays the overall schedules for the
completion of this project.

By referring to the Gantt chart, the writer of this paper has managed to plan
and schedule well the overgoing of the project. In addition, this Gantt chart also helps
to manage dependencies between tasks and keep updated on when the project will get
completed.

Gantt chart of the first and second semesters planning of the project is shown
in subchapter 3.4.1 and 3.4.2 respectively. Note that both of the Gantt charts illustrated
the activities against time which have been specified in the week.

32
3.4.1 Gantt Chart FYP 1

Table 3.4.1 below shows the first-semester planning that has been conducted
in this project.

Week
Activity/ Week
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Problem
Identification
Introduction of FYP
(Chapter 1)
Literature Review
(Chapter 2)

Research Methodology
(Chapter 3)
Draft project proposal
submission
Expected Results
(Chapter 4)
Conclusion of FYP
(Chapter 5)
Progress
Evaluation 1
Proposal Correction

Proposal Défense

Progress
Evaluation 2
Progress evaluation 3

FYP 1 Project Report

Submission
Logbook and Turnitin

FYP 1 Revision

Table.3.4.1 Gantt Chart FYP 1 Planning

33
3.4.2 Gantt Chart FYP 2

Table 3.4.2 below shows the second-semester planning that will be conducted
in this project.

Week
Activity/ Week
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Story Ideation and
Realization
Wireframing

UI Design

Unity 3D Design

Progress
Evaluation 1
AR Design Testing
with LCM
Progress
Evaluation 2
AR and LCM
Validation
Usability Testing

Progress
Evaluation 3
FYP Conference and
Demonstration
FYP logbook and
Turnitin
Final Thesis

Table.3.4.2 Gantt Chart FYP 2 Planning

34
 CHAPTER 4

EXPECTED RESULT

The expected result for the outcome of this study is the development of a
method for the multisensory simulator to stimulate attention in autistic children by
making it an interactive social storybook that can be viewed in three-dimensional
shape through Augmented Reality (AR) and can be played by hands gestures through
the hardware of Leap Motion Controller (LCM). This interactive real-life 3D
application is obtained from a software of Unity 3D that uses C# programming
language. Under those circumstances, several children with autism who are in therapy
at the age of 5-10 years old and their therapist will be asked to test the effectiveness of
the designated simulator.

At the end of this study, 4 types of social stories are told to teach the autistic
children about the basic behavior that's expected in specific settings like how to get
acquaintance when they just meet new people, how to invite their friends to play
together, and how to accept the feeling if they lose, withal saying thank you and
politely leaving their friends. Along with this, 3 games are also provided to stimulate
their vision, touch, hearing, and proprioception through moving and parking cars
games, finding the differences between two objects, correspondingly to sorting and
matching games.

35
 CHAPTER 5

CONCLUSION

When all is said and done, briefly this thesis is a study towards a series of
papers about Augmented Reality, Leap Motion Controller, and a Social Story that are
addressed for stimulating attention of autistic children expectedly can become one of
the therapy methods, especially during the current Covid 19 pandemic that limits them
to have conductive and sufficient space to play, including due to lack of chances to
have social interaction other than with those living with them.

Moreover, most importantly due to the number of ASD cases in most countries
globally including Malaysia which is critical throughout the year and needs to be
followed up. By having the alternative to develop a multisensory stimulator for
stimulating attention in autistic children as a substitute or supporting media of the
therapy center will be very beneficial and applicable in providing a comfortable
atmosphere and minimizing any circumstances that may distract their attention.

36
 REFERENCES

1. T. Tavassoli et al., “Measuring Sensory Reactivity in Autism Spectrum Disorder:

Application and Simplification of a Clinician-Administered Sensory Observation
Scale,” Journal of Autism and Developmental Disorders, vol. 46, no. 1, pp. 287–
293, Sep. 2015, DOI: 10.1007/s10803-015-2578-3.
2. N. Turoman, “How Do Kids and Grown-Ups Get Distracted in Everyday
Situations?” Frontiers for Young Minds, vol. 5, 2017, DOI:
10.3389/frym.2017.00008.
3. R. Simeoli, N. Milano, A. Rega, and D. Marocco, “Using Technology to Identify
Children with Autism Through Motor Abnormalities,” Frontiers in Psychology,
vol. 12, May 2021, DOI: 10.3389/fpsyg.2021.635696.
4. M. Petrou, A. Soul, B. Koshy, H. McConachie, and J. R. Parr, “The impact on the
family of the coexisting conditions of children with autism spectrum disorder,”
Autism Research, vol. 11, no. 5, pp. 776–787, Feb. 2018, DOI: 10.1002/aur.1932.
5. K. Hatta, M. Hosozawa, K. Tanaka, and T. Shimizu, “Exploring Traits of Autism
and Their Impact on Functional Disability in Children with Somatic Symptom
Disorder,” Journal of Autism and Developmental Disorders, vol. 49, no. 2, pp.
729–737, Sep. 2018, DOI: 10.1007/s10803-018-3751-2.
6. L. W. ten Hoopen et al., “Children with an Autism Spectrum Disorder and Their
Caregivers: Capturing Health-Related and Care-Related Quality of Life,” Journal
of Autism and Developmental Disorders, vol. 50, no. 1, pp. 263–277, Oct. 2019,
DOI: 10.1007/s10803-019-04249-w.
7. D. van Rooij et al., “Cortical and Subcortical Brain Morphometry Differences
Between Patients with Autism Spectrum Disorder and Healthy Individuals Across
the Lifespan: Results from the ENIGMA ASD Working Group,” American
Journal of Psychiatry, vol. 175, no. 4, pp. 359–369, Apr. 2018, DOI:
10.1176/appi.ajp.2017.17010100.
8. M. H. L. Abdullah, “Technology-Based Intervention for Supporting the
Development of Children with Autism Spectrum Disorder,” Queensland
University of Technology. Accessed: Dec. 22, 2021. [Online]. Available:
http://dx.doi.org/10.5204/thesis.eprints.122561

37
9. M. M. Spaniol et al., “Association between attention, nonverbal intelligence and
school performance of school-age children with autism spectrum disorder from
a public health context in Brazil,” Research in Developmental Disabilities, vol.
116, p. 104041, Sep. 2021, DOI: 10.1016/j.ridd.2021.104041.
10. P. Wozniak, O. Vauderwange, A. Mandal, N. Javahiraly, and D. Curticapean,
“Possible applications of the LEAP motion controller for more interactive
simulated experiments in augmented or virtual reality,” Sep. 2016. Accessed:
Dec. 22, 2021. [Online]. Available: http://dx.doi.org/10.1117/12.2237673
11. C. Berenguer, I. Baixauli, S. Gómez, M. de E. P. Andrés, and S. De Stasio,
“Exploring the Impact of Augmented Reality in Children and Adolescents with
Autism Spectrum Disorder: A Systematic Review,” International Journal of
Environmental Research and Public Health, vol. 17, no. 17, p. 6143, Aug. 2020,
DOI: 10.3390/ijerph17176143.
12. S. Abou El-Seoud, O. Halabi, and V. Geroimenko, “Assisting Individuals with
Autism and Cognitive Disorders: An Augmented Reality-Based Framework,”
International Journal of Online and Biomedical Engineering (iJOE), vol. 15, no.
04, p. 28, Feb. 2019, DOI: 10.3991/ijoe.v15i04.9835.
13. J. Quintero, S. Baldiris, R. Rubira, J. Cerón, and G. Velez, “Augmented Reality
in Educational Inclusion. A Systematic Review on the Last Decade,” Frontiers in
Psychology, vol. 10, Aug. 2019, DOI: 10.3389/fpsyg.2019.01835.
14. R. S. Baragash, H. Al-Samarraie, A. I. Alzahrani, and O. Alfarraj, “Augmented
reality in special education: a meta-analysis of single-subject design studies,”
European Journal of Special Needs Education, vol. 35, no. 3, pp. 382–397, Dec.
2019, DOI: 10.1080/08856257.2019.1703548.
15. M. F. Syahputra, D. Arisandi, A. F. Lumbanbatu, L. F. Kemit, E. B. Nababan, and
O. Sheta, “Augmented reality social story for autism spectrum disorder,” Journal
of Physics: Conference Series, vol. 978, p. 012040, Mar. 2018, DOI:
10.1088/1742-6596/978/1/012040.
16. E. Chong et al., “Detecting Gaze Towards Eyes in Natural Social Interactions and
Its Use in Child Assessment,” Proceedings of the ACM on Interactive, Mobile,
Wearable and Ubiquitous Technologies, vol. 1, no. 3, pp. 1–20, Sep. 2017, DOI:
10.1145/3131902.

38
17. E. A. Zaky, “Autism Spectrum Disorder (ASD); The Past, The Present, and The
Future,” Journal of Child and Adolescent Behavior, vol. 05, no. 03, 2017, DOI:
10.4172/2375-4494.1000e116.
18. R. A. Pambudi, N. Ramadijanti, and A. Basuki, “Psychomotor game learning
using skeletal tracking method with leap motion technology,” Sep. 2016.
Accessed: Dec. 22, 2021. [Online]. Available:
http://dx.doi.org/10.1109/elecsym.2016.7860991
19. D. F. Cihak, E. J. Moore, R. E. Wright, D. D. McMahon, M. M. Gibbons, and C.
Smith, “Evaluating Augmented Reality to Complete a Chain Task for Elementary
Students With Autism,” Journal of Special Education Technology, vol. 31, no. 2,
pp. 99–108, May 2016, DOI: 10.1177/0162643416651724.
20. N. Sahin, N. Keshav, J. Salisbury, and A. Vahabzadeh, “Safety and Lack of
Negative Effects of Wearable Augmented-Reality Social Communication Aid for
Children and Adults with Autism,” Journal of Clinical Medicine, vol. 7, no. 8, p.
188, Jul. 2018, DOI: 10.3390/jcm7080188.
21. A. Haria, A. Subramanian, N. Asokkumar, S. Poddar, and J. S. Nayak, “Hand
Gesture Recognition for Human-Computer Interaction,” Procedia Computer
Science, vol. 115, pp. 367–374, 2017, DOI: 10.1016/j.procs.2017.09.092.
22. J. A. Crowell, J. Keluskar, and A. Gorecki, “Parenting behavior and the
development of children with autism spectrum disorder,” Comprehensive
Psychiatry, vol. 90, pp. 21–29, Apr. 2019, DOI:
10.1016/j.comppsych.2018.11.007.
23. A. C. Daub and T. Huber, “Effectiveness of Social Stories on Social Skills for
Elementary-Aged Students with Autism: A Literature Review,” International
Journal of Education, vol. 12, no. 4, p. 12, Nov. 2020, doi:
10.5296/ije.v12i4.17759.
24. A.-P. Huang, F. Huang, and J.-S. Jhu, “Unreal Interactive Puppet Game
Development Using Leap Motion,” Journal of Physics: Conference Series, vol.
1004, p. 012025, Apr. 2018, doi: 10.1088/1742-6596/1004/1/012025.
25. H. Bahzad Ahmad, “An augmented reality system to enhance facial expressions
recognition in autistic children,” Journal of Advanced Computer Science &
Technology, vol. 8, no. 2, p. 46, Nov. 2019, DOI: 10.14419/jacst.v8i2.29873.
26. L. F. Panchenko, T. A. Vakaliuk, and K. V. Vlasenko, “Augmented reality books:
concepts, typology, tools,” Proceedings of the 3rd International Workshop on

39
 Augmented Reality in Education, Kryvyi Rih, Ukraine, May 13, 2020, vol. 2731,
no. 0100, pp. 283–296, 2020.
27. K. Khowaja, D. Al-Thani, A. O. Hassan, A. Shah, and S. S. Salim, “Mobile
Augmented Reality App for Children with Autism Spectrum Disorder (ASD) to
Learn Vocabulary (MARVoc): From the Requirement Gathering to Its Initial
Evaluation,” in Lecture Notes in Computer Science, Cham: Springer International
Publishing, 2020, pp. 424–437.
28. I.-J. Lee, L.-Y. Lin, C.-H. Chen, and C.-H. Chung, “How to Create Suitable
Augmented Reality Application to Teach Social Skills for Children with ASD,”
in State of the Art Virtual Reality and Augmented Reality Knowhow, InTech,
2018. Accessed: Dec. 22, 2021. [Online]. Available:
http://dx.doi.org/10.5772/intechopen.76476
29. Ms. P. Karamanoli, “The Application of Augmented Reality for Intervention to
People with Autism Spectrum Disorders,” IOSR Journal of Mobile Computing &
Application, vol. 04, no. 02, pp. 42–51, Jul. 2017, doi: 10.9790/0050-04024251.
30. S. Balakrishnan and A. Alias, “Usage of Social Stories in Encouraging Social
Interaction of Children with Autism Spectrum Disorder,” Journal of ICSAR, vol.
1, no. 2, pp. 91–97, Jul. 2017, doi: 10.17977/um005v1i22017p091.
31. W. Lui, “Effectiveness of a relationship development intervention program for
aloof-autistic children,” The University of Hong Kong Libraries.
http://dx.doi.org/10.5353/th_b3709134
32. C. L. Valle, K. Chenausky, and H. Tager-Flusberg, “How do minimally verbal
children and adolescents with autism spectrum disorder use communicative
gestures to complement their spoken language abilities?” Autism &
Developmental Language Impairments, vol. 6, p. 239694152110350, Jan. 2021,
DOI: 10.1177/23969415211035065.
33. S. A. Mason, “Social Stories,” in Encyclopaedia of Autism Spectrum Disorders,
Cham: Springer International Publishing, 2021, pp. 4476–4479.
http://dx.doi.org/10.1007/978-3-319-91280-6_170
34. M. Sarosa, A. Chalim, S. Suhari, Z. Sari, and H. B. Hakim, “Developing
augmented reality-based application for character education using unity with
Vuforia SDK,” Journal of Physics: Conference Series, vol. 1375, no. 1, p. 012035,
Nov. 2019, DOI: 10.1088/1742-6596/1375/1/012035.

40
35. https:/\/autismmalaysia.com\/2021\/04\/2021-world-autism-awareness-month-
highlights\/#author, “2021 World Autism Awareness Month Highlights - Early
Autism Project Malaysia,” Early Autism Project Malaysia -, Apr. 22, 2021.
https://autismmalaysia.com/2021/04/2021-world-autism-awareness-month-
highlights/
36. P. Chen, L. Mao, G. P. Nassis, P. Harmer, B. E. Ainsworth, and F. Li,
“Coronavirus disease (COVID-19): The need to maintain regular physical activity
while taking precautions,” Journal of Sport and Health Science, vol. 9, no. 2, pp.
103–104, Mar. 2020, doi: 10.1016/j.jshs.2020.02.001.
37. A. Narzisi, “Handle the Autism Spectrum Condition During Coronavirus
(COVID-19) Stay at Home period: Ten Tips for Helping Parents and Caregivers
of Young Children,” Brain Sciences, vol. 10, no. 4, p. 207, Apr. 2020, doi:
10.3390/brainsci10040207.
38. P. Hiremath, C. S. Suhas Kowshik, M. Manjunath, and M. Shettar, “COVID 19:
Impact of lock-down on mental health and tips to overcome,” Asian Journal of
Psychiatry, vol. 51, p. 102088, Jun. 2020, doi: 10.1016/j.ajp.2020.102088.
39. J. M. Garcia et al., “Brief Report: Preliminary Efficacy of a Judo Program to
Promote Participation in Physical Activity in Youth with Autism Spectrum
Disorder,” Journal of Autism and Developmental Disorders, vol. 50, no. 4, pp.
1418–1424, Dec. 2019, doi: 10.1007/s10803-019-04338-w.
40. S. Bölte, “Autism-Europe International Congress 2016—Autism research and
practice in Europe 2016,” Autism, vol. 21, no. 1, pp. 3–4, Nov. 2016, doi:
10.1177/1362361316674570.
41. B. Keehn, A. Nair, A. J. Lincoln, J. Townsend, and R.-A. Müller, “Under-reactive
but easily distracted: An fMRI investigation of attentional capture in autism
spectrum disorder,” Developmental Cognitive Neuroscience, vol. 17, pp. 46–56,
Feb. 2016, DOI: 10.1016/j.dcn.2015.12.002.
42. D. Poole, E. Gowen, P. A. Warren, and E. Poliakoff, “Visual-tactile selective
attention in autism spectrum condition: An increased influence of visual
distractors.,” Journal of Experimental Psychology: General, vol. 147, no. 9, pp.
1309–1324, Sep. 2018, doi: 10.1037/xge0000425.
43. F. M. Gresham, “Social skills assessment and intervention for children and
youth,” Cambridge Journal of Education, vol. 46, no. 3, pp. 319–332, Jul. 2016,
doi: 10.1080/0305764x.2016.1195788.

41
44. P. Zikl, “Motor skills of children with autistic spectrum disorder,” SHS Web of
Conferences, vol. 26, no. 01076, 2016, DOI: 10.1051/shsconf/20162601076.
45. B. Choi, K. A. Leech, H. Tager-Flusberg, and C. A. Nelson, “Development of
fine motor skills is associated with expressive language outcomes in infants at
high and low risk for autism spectrum disorder,” Journal of Neurodevelopmental
Disorders, vol. 10, no. 1, Apr. 2018, doi: 10.1186/s11689-018-9231-3.
46. D. T. P. Phytanza, E. Burhaein, and R. Pavlovic, “Gross Motor Skills Levels in
Children with Autism Spectrum Disorder during the COVID-19 Pandemic,”
International Journal of Human Movement and Sports Sciences, vol. 9, no. 4, pp.
738–745, Jul. 2021, doi: 10.13189/saj.2021.090418.
47. L. Ketcheson, J. Hauck, and D. Ulrich, “The effects of an early motor skill
intervention on motor skills, levels of physical activity, and socialization in young
children with autism spectrum disorder: A pilot study,” Autism, vol. 21, no. 4, pp.
481–492, Jun. 2016, doi: 10.1177/1362361316650611.
48. H. Bahzad Ahmad, “An augmented reality system to enhance facial expressions
recognision in autistic children,” Journal of Advanced Computer Science &
Technology, vol. 8, no. 2, p. 46, Nov. 2019, doi: 10.14419/jacst.v8i2.29873.
49. M.-È. Marchand-Krynski, O. Morin-Moncet, A.-M. Bélanger, M. H. Beauchamp,
and G. Leonard, “Shared and differentiated motor skill impairments in children
with dyslexia and/or attention deficit disorder: From simple to complex sequential
coordination,” PLOS ONE, vol. 12, no. 5, p. e0177490, May 2017, doi:
10.1371/journal.pone.0177490.
50. P. Zikl, D. Petrů, A. Daňková, H. Doležalová, and K. Šafaříková, “Motor skills
of children with autistic spectrum disorder,” SHS Web of Conferences, vol. 26,
p. 01076, 2016, doi: 10.1051/shsconf/20162601076.
51. B. Yılmaz, M. Azak, and N. Şahin, “Mental health of parents of children with
autism spectrum disorder during COVID-19 pandemic: A systematic review,”
World Journal of Psychiatry, vol. 11, no. 7, pp. 271–276, Jul. 2021, doi:
10.5498/wjp.v11.i7.271.
52. J. Miles, Unity 3D and PlayMaker Essentials. A K Peters/CRC Press, 2016.
Accessed: Dec. 24, 2021. [Online]. Available:
http://dx.doi.org/10.4324/9781315686165.

42