Entertainment Computing 42 (2022) 100495

Contents lists available at ScienceDirect

Entertainment Computing
journal homepage: www.elsevier.com/locate/entcom

Dynamic difficulty adjustment technique-based mobile vocabulary learning

game for children with autism spectrum disorder
Samaa M. Shohieb a, *, Ceymi Doenyas b, Abdelghafar M. Elhady c
a
Faculty of Computers and Information, Mansoura University, Egypt
b
Department of Psychological Counseling and Guidance, Yıldız Technical University, Istanbul, Turkey
c
Deanship of Scientific Research, Umm Al-Qura University, Saudi Arabia

A R T I C L E I N F O A B S T R A C T

Keywords: This article describes the design, implementation, and evaluation of an educational mobile game. Pickstar aims
Autism spectrum disorder to make vocabulary learning training more accessible to families with ASD, in a way that increases engagement
DDA games by offering a multisensory experience and by adapting to the level of each player. The game includes two
Human-computer interface
learning approaches: the classical levels advancement approach where the levels gradually increase in difficulty
Mobile learning
Pedagogical issues
and Dynamic Difficulty Adjustment (DDA) approach that adapts to the skill level of the player as determined via
Special needs education feature extraction. Through the AdaptiveMiniMax algorithm, Pickstar’s DDA feature ensures that everyone has
the same winning rate of approximately 70%, encouraging players of all levels to play again. The engagement
rate of children with ASD in the learning process was 88.4% higher for DDA compared with classical levels.
Usability evaluation of Pickstar, based on ISO 25062:2006, on caregiver interviews concluded that Pickstar not
only decreases caregivers’ teaching and assessment loads, but also increases the time children spend on learning.
Caregiver interviews, usability testing, and engagement assessments show that Pickstar can be used with minimal
guidance to improve vocabulary skills of individuals with ASD and can mitigate caregiver problems in accessing
good-quality, engaging, and affordable education for their children.

1. Introduction
Autism spectrum disorder (ASD) is a neurodevelopmental condition
characterized by social communication and interaction difficulties and
restricted and repetitive behaviors and interests [3]). Individuals with
ASD usually display repetitive or rigid language, poor nonverbal con­
versation abilities, and their speech and language skills do not develop
to the typical level and show uneven progress [4]. It has been reported
that 25–30% of individuals with ASD do not talk at all or do not develop
functional language throughout their lives [23]. Such challenges in
communication, in conjunction with difficulties in social interactions,
tend to isolate children with ASD from others. The child with ASD lives
closed to oneself (also reflected in the etymology coming from ‘autos’
[self] + ism) and hardly responds to what is around and the surrounding
individuals, events or phenomena [30,17]. ASD may affect anyone and
is not limited to a specific sociocultural, educational, ethnicity or eco­
nomic level. Autism cannot be cured; however, individuals may reach
optimum outcomes, their difficulties may be decreased, and their life
quality may be increased with early diagnosis and intervention. Speech
therapy is one of several therapies that are often used with individuals
with autism, which also includes vocabulary teaching [24]. According to
DSM-5 Diagnostic and Statistical Manual of Mental Disorders (2013) and
ICD-10 International Classification of Diseases (1993) that formulate
criteria for the diagnosis of infantile autism, most children with autism
who are on the moderate and severe parts of the spectrum experience
deficits in linguistic development and inability or absence of verbal and
nonverbal communication to express themselves and understand others.
Thus, vocabulary learning is very important for individuals with ASD.
In Egypt, more than 840,000 children with autism were recorded in
2015 [34]. In the Middle East, the numbers range from 15.6 to 18 per
1000 [2]. Caregivers in the Arab countries express concern that the
education centers for individuals with developmental disabilities are
solely established in big cities, yet not all children with ASD live in cities
[38,39]. This leads to children with ASD residing in rural areas to not
receive proper education. Remote areas, even if they do not have edu­
cation centers available, can use mobile applications. Mobile phones are
no longer a luxury item. Every person is likely to have one or more
mobiles, and even children use mobile phones. Based on these

* Corresponding author.
E-mail address: sm_shohieb@mans.edu.eg (S.M. Shohieb).

https://doi.org/10.1016/j.entcom.2022.100495
Received 12 April 2021; Received in revised form 11 December 2021; Accepted 15 April 2022
Available online 23 April 2022
1875-9521/© 2022 Elsevier B.V. All rights reserved.
S.M. Shohieb et al.
developments, mobile technology is predicted to be one of the tools that
can be used to facilitate the access to education for children with ASD.
Many researchers acknowledge the importance of educational games for
autism [9]. Educational games can help children learn about the world
in which they live, consequently improving their social abilities
[7,25,41]. Also, games can help develop different cognitive aspects of
children. Moreover, educational games can stimulate participation in
the learning process [47]. Therefore, mobile educational games
designed for children with ASD not only hold the potential to make
education more accessible to those not living in big cities, but also to
make educational materials more engaging for those living in big cities.
What is more, during global emergency periods like the current Covid-
19 pandemic, mainstream education for typically developing children
continues online, yet special education for children with ASD does not.
Considering that special education is the only validated intervention for
them, creating mobile education applications designed for their specific
needs and characteristics carry importance not just for regular times but
also global emergency periods where face-to-face special education may
halt for individuals with ASD.
Dynamic difficulty adjustment (DDA) is a method of automatically
modifying a game’s behaviors, features, and scenarios depending on the
player’s skill level, in real-time [54]. Malone [33] recommended that if
the challenge, fantasy, control, and curiosity in games could be balanced
and linked to the gradual enhancement of difficulty level, the resulting
game could keep the user fascinated. Consequently, applying DDA in
learning games results in keeping the player absorbed and engaged in
the learning process targeted by the game. Yet, DDA carries a compli­
cated task as it needs to successfully perform a real-time estimation of
the next entry to the game at the right instance [26].
To address the aforementioned issues, this paper presents the Pick­
star game. It aims to mitigate the problems encountered by the care­
givers of children with ASD in terms of access to education centers.
Pickstar is an Arabic language vocabulary learning game based on the
DDA approach and on gaming techniques and strategies to increase user
motivation. These include utilizing fun, cartoon-like pictures of animals,
engaging sounds such as animal sounds, and rewards such as clapping.
The aim is to increase the vocabulary and pronunciation knowledge of
players with word matching games that also voice each word. By
appealing to more than just the sense of vision as done in traditional
paper-based methods, and creating a multisensory experience with
engaging elements, this application is designed to increase the time
children with ASD spend with educational content and especially that
related to vocabulary and communication. This work is grounded in
Arab ASD social communities, DDA game approach, and Human-
Computer Interaction (HCI) research. HCI includes various aspects
such as user-centered design (UCD), user interface (UI) design, and user
experience (UX) design. HCI can be said to be, in many ways, the fore­
runner of UX design.
This paper presents the design of Pickstar and the results of the pilot
study conducted to assess its learning outcomes and usability evalua­
tion. Usability according to ISO 25062:2006 refers to the effectiveness
(completeness and accuracy), efficiency, and satisfaction with which
specified users achieve specified goals in particular environments. The
design and testing of the proposed game followed the recently developed
framework to design vocabulary-based serious games for children with
ASD. Khowaja and Salim [31] developed a new Serious Game Design
Framework (SGDF), where they show that before and after utilizing
their prototype, children with ASD improved in their vocabulary
acquisition. The authors recommend giving an overview of the game to
be developed, which includes its title, description, target audience, and
platform, as well as specifying the game components used structured
according to IPO (input, process, and output). Additionally, they suggest
using applicability surveys based on the quality model of ISO [31],
which are all included in our presentation of Pickstar. First, to under­
stand how children with ASD acquire and use vocabulary, the authors
carried out a corresponding user research as is done in HCI research,
2
Entertainment Computing 42 (2022) 100495
where they talked with teachers and parents of children with ASD and
observed the children to determine the main problems that will be
solved by the design and how the user will use the system. Using these
observations and the guidance of an ASD specialist, the initial prototype
game was created. Then, primary utilization and feedback from in­
dividuals with ASD and their families were collected. User engagement
and satisfaction were evaluated via participant scores in the game and
parent reports in the interviews. The details of the game and applying
the DDA in the game are presented after reviewing the relevant litera­
ture, which is followed by the reported user feedback about the game
from parents of children with ASD.
The following parts introduce the proposed game, Pickstar, by
explaining why such a game is needed, its design, testing, and usability
results. Specifically, first, a literature review is presented of the tech­
nological rehabilitation systems, applications, and digital games for in­
dividuals with ASD. It is followed by the full design and methodology of
Pickstar. Next, the results obtained by having children with ASD use this
game and usability evaluations of Pickstar are reported and discussed.
Finally, conclusions about what these results mean and recommenda­
tions for future work are presented.
2. Literature review
2.1. Systems, applications, and serious games for interventions and
education for ASD
Studies dating back to two decades introduced and showed the
effectiveness of computer-based assistive technologies for children with
ASD. Some even show that completing educational tasks on computers
were more engaging for children with ASD than paper-based methods,
such as children with ASD spending more time on the reading material
accessed through a computer compared to book-based learning [48].
Computer-based techniques for ASD are categorized into two types,
rehabilitation systems and educational gaming systems, and are
explained in detail below.
2.1.1. Technological systems for intervention and training
There have been many attempts to facilitate the delivery of in­
terventions for ASD using assisted technologies. Galitsky [22] presented
a reasoning rehabilitation strategy for individuals with ASD to help them
overcome their difficulties in learning social rules from examples. They
describe the user interface (UI) of the model they built of the mental
world comprising 12 steps. Another group [1] used a design method
based completely on a dynamic manner that merges technology
benchmarking, concept generation, and activity design into constantly
enriching actions through using tangible audio-augmented artefacts for
therapy and academic intervention for children with ASD. Though their
prototype is a tool to help the therapist engage the child with ASD in a
storytelling activity, they highlight the broader benefits of using this
audio-augmented method in therapy and educational interventions for
individuals with ASD. Other researchers experimented with developing
artificial intelligence-based systems and machine learning to detect
learning patterns and develop training systems for individuals with ASD.
One machine-learning technique makes use of the Indian Buffet tech­
nique (IBP) that is based on a binary matrix of infinite columns to assign
groups of intervention skills to children. Using this method on data
collected by an intervention application for ASD, the best number of
learning patterns was developed, and subgroup assignments have been
automatically inferred from statistics. The researchers classified the
required intervention for the child with ASD based on 32 skills in
imitation, language, and sensory domains. They came to the conclusion
that Bayesian nonparametric models are more suitable to infer the
optimal number of patterns of learning as well as subgroup assignments
than parametric rivals [46], showing the effectiveness of machine-
learning approaches to classify possible interventions utilizing the
data of individuals with ASD. Similarly, a Rehabilitation Training
S.M. Shohieb et al.
System (RTS) was developed to train children with ASD to learn basic
plants, shapes, animals, colors, and find the odd one out from a
mentioned set of objects, and for other skills [53]. Their applied pro­
totype motivated the children to learn the patterns by providing in­
centives via audio signals in addition to easing teachers’ efforts.
Technology has also been used to transform the traditional methods
utilized to increase the communicative skills of individuals with ASD.
Chien et al. [11] introduced a tablet-based system that adopted aspects
of the traditional Picture Exchange Communication System (PECS)
approach. PECS refers to a visual communication method where in­
dividuals with delayed or no speech use pictures to express their needs
and desires [8], which yields significant improvements in social-
communicative skills of children with ASD including joint attention,
request, initiation, and cooperative play [32]. Chien et al. [11] incor­
porated advantageous features such as support for digital, voice capa­
bilities, and visualization. Their system improved the willingness of
children with ASD to learn and engage with others, and it also decreased
the time it took caregivers to prepare content by over 70%. All this
previous work shows how technology can contribute to digitalizing
intervention systems for children with special needs and especially those
with ASD, which can help make these sources more accessible to and
affordable by a wider population.
2.1.2. Applications and mobile games as a teaching method for ASD
Previous work has demonstrated the feasibility of creating and using
educational mobile games for individuals with ASD, specifically in
relation to communication, including apps to complement language
learning, language production, and social communication. A vocabulary
learning application has been designed on Android-based mobile de­
vices and implemented for the use of children with ASD, which has not
yet been tested. It is designed to have one interface where caregivers can
edit or download the teaching materials, and another where children
with ASD can select the learning category, learning item, and play audio
or music [27]. In a similar vein, a tablet-based software was designed in
collaboration with individuals with ASD and communication therapists
that aimed to assist the learning of manual motor and oral motor skills
that are essential for communication [6]. The present authors created
and tested a tablet-based application to teach sequencing skills to chil­
dren with ASD using reinforcement and cueing principles used in ASD
education [20]. Another application was developed that helps children
with ASD enhance their social life by communicating and interacting
with others, while supporting their caregivers and parents to recognize
what the children desire [5]. Moreover, in another mobile platform,
children with ASD can develop generalization skills and language [35]
and caregivers can construct custom-designed exercises with multiple
options that take into account the unique characteristics of every child.
Furthermore, a game-based prototype was created to teach the Qur’an to
children with low-functioning ASD, though it has not yet been tested
with users yet. These authors digitalized a teaching strategy by devel­
oping an interactive game that integrated listening, arranging, con­
structing, imitation, and pronunciation skills (Kamaruzaman et al.,
2015). Present authors have designed a mobile application that uses a
simulation game to help children with ASD understand and follow
Covid-19-related safety measures by gaining points and reinforcers in
this game [19]. All these gaming examples share the aim of making the
learning materials more portable and more easily accessed by children
with ASD by digitizing and placing the educational materials into a
single device.
In terms of technology designs for vocabulary acquisition in ASD,
Winoto and Tang [50] present two mobile picture-based text-learning
applications technologies (i.e. Swipe-N-Tag, and Snap-N-Recognize)
3
Entertainment Computing 42 (2022) 100495
that are flexible and individually adaptable for individuals with ASD.
As they enable caregivers and teachers to use readily available items at
home or outside to build databases according to the child’s learning
process, these applications can help the words acquired in the classroom
to be transferred to actual situations. A small number of teachers and
parents in a private autism center in the city tested the IOS version of
Swipe-N-Tag (for both the iPhone and the iPad) with satisfactory results.
There are also early virtual reality [49,51,52] and augmented reality
[44,45] applications designed for children with ASD. These applications
aim to help children with ASD with word recognition [49,51] and vo­
cabulary learning [44,52], with promising and satisfying results. Winoto
et al. [52] designed a VR room in which players learn new words on one
of the walls of the room through the look of their visual shape, while the
integration of audio and visual modalities of the VR environment further
assists word recognition. Their study provided early insights into the
acceptability of such intervention techniques for Chinese parents and
their children with ASD. Tang et al. [45] present a mobile vocabulary
learning app especially for outdoor and home use for Chinese children
with ASD. In the deep educational platform, TensorFlow, the core object
recognition module is deployed, but the algorithm must be run offline
unlike other advanced systems. Two pilot studies were conducted to
assess the system feasibility and usability. Their studies indicated that
the performance of the object-recognition module is the negative aspect
of the application. Since the latest AR-technology literature on Chinese
word-learning is still in its infancy, particularly with regards to design
and assessment, it offers limited insights into its therapeutic effective­
ness, feasibility and applicability to individuals, and especially children,
with ASD. Moreover, these studies do not reveal whether the children
with ASD improved their vocabulary with these methods or not.
It is seen that none of the available applications and games for
children with ASD that target language and communication skills are in
the Arabic language; they are designed in different languages such as
English, German, Indian, and Indonesian. Moreover, the only existing
Arabic application is designed for teaching the Holy Qur’an to low
functioning children with ASD [29], which does not teach language or
vocabulary, but verses of the Qur’an. To address these shortcomings, the
authors developed an Arabic vocabulary learning game for Arab chil­
dren with ASD to enhance their learning motivation and help them learn
more vocabulary or improve their existing verbal skills. One study took
the feedback of children with ASD on rewards in educational computer-
based games. From the choices of still image, animation, sound, fire­
work, and certificate, most children selected all the features, and as the
rewards progressed, they preferred adding additional animations,
additional sounds or certificates, sounds, and fireworks or certificates in
the following levels [14]. Therefore, a preference for both visual ani­
mations and audio stimuli of children with ASD emerges with the reports
provided by individuals with ASD. These types of reinforcements have
been previously implemented in educational tablet games for ASD [20]
and follows the principles of Applied Behavior Analysis (ABA), which is
the most commonly used educational principle for teaching individuals
with ASD in special education centers and schools. ABA refers to using
data about behaviors to develop hypotheses about why a certain
behavior happens in a particular context and creating interventions to
change the identified behaviors. It is the science of applying information
obtained from the analysis of behavior to systematically and purpose­
fully modify behavior. It uses prompting, shaping, and reinforcement to
increase desired behaviors. ABA interventions consistently yield signif­
icant increases in functional skills and decreases in ASD symptoms, with
these positive outcomes maintained across time [28]. Therefore, such
rewards have been implemented into the current application.
S.M. Shohieb et al.
2.2. The Pickstar game
The Pickstar game has been designed as an assistive teaching
application that runs on Android-based devices (mobiles or tablets).
Pickstar is a DDA-based game designed for children with ASD, with the
aim to maximize their engagement, independence as well as enjoyment.
Moreover, it intends to reduce the complexity and effort of teaching and
assessment for caregivers of children with ASD, especially in the
developing countries. The Materials section describes the development
of the game, its components, and design methodology; the Methods
section explains its testing on individuals with ASD; and the Results
section delineates the user experience and satisfaction outcomes for the
Pickstar game.
3. Materials
The main material is the educational application, which was
designed according to the design process for HCI science (Fig. 1) and
developed based on the guidelines for developing learning games for
children with autism [15].
The explanation of the design methodology is as follows:
a. Aims and problem Identification:
The problem was identified via the following steps:
(1) Performing field observation by going to a school for children
with ASD and observing the daily and educational activities of
children with ASD.
(2) Analyzing relevant findings from existing research studies.
(3) Conducting personal interviews with the caregivers of children
with ASD.
b. Analysis Phase: The following steps delineated above revealed
that children with ASD have little interest in conventional learning
methods and paper-based learning methods. Relatedly, after performing
field observation and reviewing existing research, the traditional (PECS)
technique that uses paper-based cards was seen not to be as attractive as
technological game-based techniques for children with ASD.
After conducting personal interviews with the caregivers of children
with ASD in the developing country of Egypt, hereafter termed pre-study
interviews, we concluded that it is not financially feasible for parents to
hire special education instructors for children with ASD. Also, special­
ized centers and schools for autism are very rare, so the parents are not
able to send their children every day, if at all. To address these critical
issues, the authors aimed to create an educational game-based appli­
cation for children with ASD that also decreases the efforts of caregivers
Fig. 1. HCI-Based Design Process [14].
4
Entertainment Computing 42 (2022) 100495
while teaching and assessing the development of their children.
c. Design Phase: This phase pertains to the software development
and prototyping for the application. The design was created via con­
struction models and methods, where these suggested models were built
and evaluated for their ability to solve the problem. Authors began by
sketching the game as a paper-based prototype, which is a throwaway
prototyping of user interfaces created as hand drawings to enable rapid
designing, in the design phase before creating the technological appli­
cation for mobiles.
d. Implementation and Deployment Phase: Then, based on the
paper-based design, the game’s prototype interface was created to be
displayed on a full screen of a mobile or a tablet device. The game was
developed as an Android mobile application. Unity engine was the used
platform, as unity is mainly a cross platform engine. This will ease the
conversion process in the future to other platforms if needed (like web,
desktop or IOS). A ‘how to play’ tutorial was included that simulates the
function of every feature and demonstrates the main components to
children and their caregivers.
e. Applying Guidelines for Developing Learning Computer Games
for Children with Autism: The design of Pickstar was based on guide­
lines for developing computer games for children with autism that are
stated by Davis et al. [15] and used in many other systems for children
with ASD later in more than 55 developed systems. Table 1 presents the
guidelines and their description. We used their guidelines for the
“Interface & software guidelines” as they relate to the design phase that
we have in this stage. The other guidelines are related to Trial guidelines
and Session guidelines and therefore were not included as they are not
pertinent to the present research. Table 2 shows how we applied each
guideline in the Pickstar game to address and accommodate various
characteristics of individuals of ASD.
The entire game interface is in the Arabic language. The game
Table 1
Guidelines for Developing Learning Games for Children with Autism [15].
Guideline Guideline Description
Number
Guideline 9 Considering the number of people in the room during using the
software and if they would overwhelm or affect the child’s
performance.
Guideline 13 All parts of the SW interface, for example feedback, should
reinforce the intended task.
Guideline 14 Keep the software interface, as simple as possible with low
differentiation between global and local cohesion.
Guideline 15 Be aware that autism may well extend far from what may be
expected in non-autistic cases. Avoid unnecessary sounds and use
a simple palette in the animations or SW interface. Similarly, be
aware of the physical environment.
Guideline 16 Recognize that children with autism vary highly in their interests
and abilities.
Guideline 17 Consider whether you will use fears and special interests in the
functionality and SW interface.
Guideline 18 Be aware of repetitive behavior patterns, which are likely to be a
feature of autistic behaviors. Consider the balance of control
between the system and participant.
Guideline 19 In the matter of record keeping: build in data logging software
that records user data even if there is no plan for the software to be
adaptive. Then, at the design stage, consider whether and how
this data is to be integrated. If there is an intention to use third
party software for integration or evaluation, then consider
compatibility issues between the third party software and the logs
at the design stage.
Guideline 20 Remember that children might have no recourse to strategies for
dealing with failure, which may be particularly debilitating.
S.M. Shohieb et al.
Table 2
Aspects of Autism and the Corresponding Applied Guideline in Pickstar.
Aspects of autism
Social challenges of autism
Children with autism find it
difficult to be flexible and
accommodating. They
prefer having routines,
certainty, and reliable and
predictable environments
Children with autism may
have powerful special fears
or interests, and display
repetitive behaviors
Table 2 (continued )
Guideline How the guideline applied
applied in the in Pickstar
Pickstar
Interface
Guideline 9 Gl 9: We advised the
caregivers to not have other
individuals in the room while
using Pickstar to not
overwhelm or affect the
performance of the child with
ASD.
It is worth noting that
Pickstar may alleviate the
social and communication
challenges of autism by
increasing their vocabulary,
and also creating instances of
social interaction with the
caregiver who is supervising
their playing of Pickstar, if
not more people are in the
room to distract or distress
the child with ASD.
Guideline 14 Gl 14: Pickstar followed this
Guideline 16 guideline closely. By keeping
the UI as simple and
predictable as possible,
Pickstar aimed to
accommodate these qualities
of children with ASD.
Gl 16: The DDA feature in
Pickstar enabled this game to
adapt to the children’s
learning needs, as children
with ASD differ in their
abilities.
Guideline 13 Gl 13 & 14: the design of
Guideline 14 Pickstar
Guideline 15 followed these guidelines
Guideline 17 very carefully, with the
Guideline 18 addition of some
Guideline 20 motivational and reward
sounds after success as
recommended in (Aziz, 2014)
and was shown not to affect
the learning process of the
behaviors of the child
negatively.
Gl 15: Pickstar does not have
any bothersome noisy sounds
or background music in order
not to distract the players or
bother children who have
sensory sensitivities to loud
noises.
Gl 17: Pickstar made use of
concepts that generally
interest and engage children
with autism, for instance
icons of objects such as
pencils and books, animals
with their relevant sounds.
Gl 18: To discourage
repetitive behaviors while
using Pickstar, the buttons
and the game were presented
in a logical order, and if the
child pressed a button
multiple times, it responded
only once.
Gl 20: In Pickstar, we tried to
match this guideline about
possible challenges children
with ASD may face dealing
with failure. For this purpose,
instead of allowing only a
single trial, we let the child
5
Aspects of autism
Children with autism may be
visual thinkers and learners,
and may have particular
sensory sensitivities.
Autistic children head for
trying to find meaning in
cases of detail (local
cohesion)
Children with autism may find
failure highly debilitating
Children with autism may
have their own routines and
daily activities
Autism-independent
guidelines
Entertainment Computing 42 (2022) 100495
Guideline How the guideline applied
applied in the in Pickstar
Pickstar
Interface
try many times to reach the
goal of the game (while
logging all the tries)
Guideline 14 The design of Pickstar
Guideline 15 followed these guidelines
very carefully by making the
visuals and icons as appealing
as possible, and selecting the
color scheme, brightness, and
sound levels to be at levels
that will not trigger their
possible sensory sensitivities.
Guideline 13 Gl 13, Gl 14 & Gl 15: the
Guideline 14 design of Pickstar followed
Guideline 15 these guidelines very
carefully
Gl 13: All parts of the Pickstar
interface, for example
feedback, reinforce the
intended task.
Gl 14: The Pickstar interface
was kept as simple as possible
with low differentiation
between global and local
cohesion.
Gl 15: Pickstar avoids
unnecessary sounds and uses
simple animations or SW
interface. This way, by
simplifying the environment
and removing potential
distractions, Pickstar aims to
allow the child with ASD to
focus on the learning task and
its detail without being
distracted by details
irrelevant to the learning
process
Guideline 20 Gl 20: In Pickstar, we tried to
match this guideline as we let
the child try many times to
reach the goal (while logging
all the tries)
Guideline 13 Gl 13 & 15: the design of
Guideline 15 Pickstar followed these
guidelines very carefully by
using simple animations and
following the same palette of
design throughout the game
in order not to distract the
child with ASD.
Moreover, motivational and
reward sounds were added
after success to reinforce
correct responses, such as
clapping or saying “Heeey”.
The reinforcements given
with sounds help them learn
how to play the game and
learn the routine of the
actions they need to perform
within the game. As children
with ASD like having stable
routines, using the same type
of design and instructions
throughout the game and at
different levels in Pickstar
conserves their routine.
Guideline 19 In the case of Pickstar, each
time a participant does
something, it is logged in the
log items mentioned in the
caregiver button subsection
S.M. Shohieb et al.
includes different word categories such as vegetables (e.g., cucumbers,
tomatoes, lettuce), fruits (e.g., banana, apple, peach), animals (e.g., dog,
fish, horse), room contents (e.g., table, chair, mat), school tools (e.g.,
pen, pencil, ruler), and personal tools (e.g., hairbrush, toothbrush,
towel).
The purpose of the game is to select a written word and its corre­
sponding picture (i.e. select a picture of the fish with the word fish,
where all words are in Arabic) to make a star fall to a designated target
place (Fig. 4). The player wins the game when all the stars reach their
target place. Each picture of animals has the corresponding animal real
sound. So, when the animal picture is selected, the player can hear the
noise made by the animal. Also, each word is pronounced by the game in
Arabic language loudly and correctly. The first level is always a training
level to teach the player how to play the game. In this level, the child is
given a hint to solve the given problem. The application waits for the
child to solve it. If the player does not make the correct response, then
another hint is given, and so on [20] for a similar hinting system in an
autism education application). After completing each level, there is a
soft sound of clapping to motivate the player and inform them that they
have completed the level successfully, following the aforementioned
reward types preferred by children with ASD [14] and previously used in
another autism education tablet application [20].
Each time a new icon appears on the game board, the child has to
make a decision. The first one, which we call match, is to decide the
word that goes with the given. If the child is able to match the word with
the correct image, they disappear, and new ones appear, which corre­
spond to “current” and “next” images, or “icons”. For each match deci­
sion (i.e. for each entry in the game) we extract the following features
[15,36]:
• The current and next icon that may enter the board.
• The time taken to match the current icon.
• The maximal points the child might have obtained if he/she could
have performed the best possible action (according to a greedy AI
player and heuristic that reduces the number of mismatches and the
number of moves to solve the game).
• Number of icons that the child recognized.
• Best score in the time-based level.
• Start time and end time of the last 10 played games. Total time for
the last level.
• The percentage of winning (successfully responding).
• Longest and shortest time spent in a level.
To apply DDA in a game, first we need to classify the skill level of the
child from the features extracted through at least ten levels of the game
6
Entertainment Computing 42 (2022) 100495
in order to make a prediction with some level of confidence. The child’s
skill level is decided by a Majority Vote (MV) function, a function in the
algorithm for finding the majority of a sequence of elements using
constant space and linear time, which computes the majority element if
one exists [10]. If there are several sequences of icons available, the MV
function finds the icons that have already been recognized by the user
based on the previous responses that are logged. If for more than half of
the cases, the user has recognized an icon, then the MV detects it as a
previously recognized icon. This computation is then used by the DDA-
mode to formulate the new level that adapts to the skill level of the child
and provides the personalized practice the user needs. In the DDA-mode,
the Adaptive-Ranker algorithm [36] (Algorithm 1) is utilized as the
adaptive agent of Pickstar. The adaptive agent should be able to adapt
even to children with very low skill levels. This way, the skill level can be
classified based on the similarity between the beginnings of the games
stored in the training set and the current partial game. In the imple­
mentation of the Adaptive-Ranker algorithm (Algorithm 1) for Pickstar,
we use the MiniMax [40] algorithm (Algorithm 2) to establish ranking.
This is called AdaptiveMiniMax and was used before in the connect four
game where four items with the same color are connected [36]. The
basic concept behind the AdaptiveMiniMax algorithm is that the
implemented artificial intelligence algorithm, the agent, chooses actions
that have the same rank as the ones chosen by the child in the previous
turns. This way, throughout the game, it enables a way to rank the
child’s and the agent’s actions. To sum, the AdaptiveMiniMax algorithm
is based on the minmax and ranker algorithms, which make estimations
based on users’ logged data to predict the level that should be presented
to the user.
When calculating the skill level of a player, we weigh the values ch1,
ch2, ch3, .. chn with weights that reflect the skill level of the algorithm.
We estimate child’s (ch) skill level as follows:
∑n
i=1 w chi
s(ch) = ∑ n
i=1 chi
where w is the weight assigned to AdaptiveMiniMax to represent the
relative skill of the algorithm, the w is set in an ad hoc manner.
After we know the player’s skill level, which can be approximated to
one of these categories (beginner (0 to 0.35), average (0.35 < s(ch) to
0.70) or advanced (0.7 < s(ch)), we have to decide how and when to
help them. The difficulty in Pickstar depends mainly on two parameters:
the number of icons and the solving speed. This work focused only on
these parameters because the new images/icons that appear in the game
are intended to be carefully selected, so that there is at least one match
(an image with a corresponding word). The flowchart of Pickstar that
depicts the game’s step-by-step logic is shown in Fig. 2.
S.M. Shohieb et al. Entertainment Computing 42 (2022) 100495

d. “Exit” Button: It enables the player to exit the game. This option
The main screen of the game has four buttons (Fig. 3a), as follows: exists to increase the interchangeability criteria of the software
(SW) design rules [12], which require multiple options that do the
a. “Play” Button: The “play” button enables the children to start same action to be implemented. This option is for flexibility of the
playing and initiate the learning process. At the beginning, only the design, and if needed, the participants can quit the app without
classical levels are active, which include 30 default levels that using this button as well.
gradually increase in difficulty. After the child plays some levels, the
DDA-mode is activated (Fig. 3b). There are three sub-buttons that
represent the following ways for teaching.
3.1. Game elements
• “All levels” sub-button: This is the classical level-mode. The player
can select any level they want (from 30 different levels ranging in
This game includes the nine formal game elements that every game is
difficulty) to achieve mastering of the vocabularies (Fig. 3c &
recommended to include [37], which are Players, Game Procedures,
Fig. 4).
Objectives, Probability, Rules, Resources, Boundary, Conflict, Game
• “Time-based level” sub-button: This selects a random grid con­
Outcome, as explained in detail below. They are elements used by game
taining different pictures and words to train the player to collect
designers, such as one of the present authors previously [21], because it
more points in less time (Fig. 5). This option trains the child in time
is suggested that including these elements will make the game design
management.
satisfy multiple aspects of human thinking, enjoyment, and motivation.
• “level selection” sub-button: This button selects a level for the
player to begin from. This option uses the DDA approach as
a. Players: The game is designed for a single player. The player can
described in subsection III-C This button isn’t activated until after
choose any one of the eleven games included and play. The game is
the child plays some regular levels in order to extract the features
not competitive.
of his/her behavior.
b. Game Procedures: The outset action of the game requires every
b. “Caregiver” Button: This allows therapists and caregivers to follow
player to gain the necessary knowledge to win. The progressive ac­
children’s improvements and performance across time. The overall
tion is to handle the increasing requirements and complexity.
information provided to the caregivers to follow their children’s
c. Objectives: The game has the main objective of enhancing the vo­
performance includes:
cabulary acquisition of children with ASD.
• Number of icons the child recognized.
d. Probability: Probability refers to the probability of acquiring new
• Number of icons in each level.
vocabulary.
• Number of solved levels.
e. Rules: Some levels are timed to control the game flow. If a player’s
• Time taken to solve the levels.
time runs out, he/she will be prohibited from continuing playing the
• Best score in the time-based level.
game. There are virtual commands during the level to define what
• Time of starting the most recent play and time of ending it.
must be done by the player.
• Total time for the most recently played level.
f. Resources: The resources to be used in order to complete the task are
• Total time in playing the game.
the available time and environmental aids, such as the hints.
• The percentage of winning.
g. Boundary: The game area is restricted to the board/screen of the
• Longest time spent in a level.
game.
• Shortest time spent in a level.
h. Conflict: Players or caregivers need to identify the relevant infor­
mation and specify the settings (objects, levels etc.) to determine the
Also, after verification, the caregiver can do the following from their
task objectives.
control panel:
i. Game outcome: Pickstar aims to enable children with ASD to collect
the highest number of vocabulary they can and to enhance their
• Change the child’s name.
ability to identify the meaning of each word in their daily
• Choose the categories of words that they wish the child to learn
environment.
(vegetables and fruits, animals, or non-living items).
c. “About Us” Button: It contains general information and infor­
mation about the designers of the game. 4. Method

Before designing this user study methodology, the following research

7
S.M. Shohieb et al. Entertainment Computing 42 (2022) 100495

Fig. 2. Flowchart of Pickstar.

questions were determined: the game, and then feedback was collected from their caregivers, as
Research Question#1 explained in detail in the subsequent sections.
In what ways does the use of Pickstar impact the learning outcomes
of Arab children with ASD?
4.1. Sample selection
Research Question#2
To what degree can Pickstar reduce the burden of assessing Arab
Convenience sampling technique was used for participant recruit­
children with ASD on parents and caregivers?
ment. The participants were seven children aged between 5 and 10
To answer these questions, participants were recruited, asked to play
years. All participants were diagnosed with ASD by autism therapists or

8
S.M. Shohieb et al. Entertainment Computing 42 (2022) 100495

Table 3
Participant information.
Caregiver Child

ID Sex Age Device Type Sex Age Diagnosis Level

P1 F 21–25 Dr.eye N101 M 6 Moderate

P2 F 31–35 Sonny EXZ 6 F 5 Severe
P3 M 36–40 Dr.eye N101 F 8 Moderate
P4 F 26–30 Dr.eye N101 F 7 Moderate
P5 F 41–45 Sonny EXZ 6 M 10 Moderate
T1 M 46–50 Sonny EXZ 6 M 8 Severe
T2 F 31–35 Samsung Note 3 Neo M 5 Severe

Table 4
Questions Asked in the Interview before the User Study.
Fig. 3. User interface screens of the game. Q# Interview Questions

1. If your child has used manually created image cards, for how many years have
they used them?
2. What kind of education materials have you utilized for instructing your child?
What was the technique for using them?
3. Regarding the teaching materials, what troubles and recommendations for
improvement did you have?
4. When using the teaching materials for the first time, what specific challenges
did you experience?
5. Do you recall any specific feedback from your child when using the teaching
materials for instruction?
6. Did you buy or borrow these teaching materials for your child? Why?
7. After using these teaching materials, did your child show any progress? Please
elaborate.
8. Did you experience any problems getting to special education centers to obtain
teaching sessions or materials for your child?

specialists according to DSM-5 criteria [3]. ASD severity ranged from

moderate to severe, and they evinced verbal skills from none to low
(Table 3). The caregivers of these children consisted of two special in­
structors (T#) and five parents (P#) aged between 21 and 50 years, who
were all volunteers from a local autism community in Cairo, Egypt.

4.2. Ethical considerations

Fig. 4. The 5th Level selected in “All levels” sub-button.

Ethical approval was obtained from the ethics committee of [BLIN­
DED] University. Informed consent has been obtained from caregivers
who accepted to participate in the study, after providing a detailed
description of the aim, procedure, risks, and benefits of the study. All
participants have been assured that participation in the study is volun­
tary and that refusing to take part in the study does not have negative

Table 5
Questions Asked In the Interview after the User Study.
Q# Interview Questions

1. Describe how frequently and where you used Pickstar to teach your child. Did
your frequency of administering teaching sessions to your child increase after
using Pickstar? Also, describe the time and place of using it most frequently.
2. Describe any specific issues experienced while using Pickstar, and any notes
you may have in that regard.
3. Describe any noteworthy responses shown by your child while using Pickstar,
and upon which features/instances that response was given.
4. Do you feel that Pickstar presented instructions and verbal communication
clearly, i.e. was Pickstar clear enough to teach verbal communication through
its instructions?
5. Which one do you consider to be the most suitable device size to utilize Pickstar
on: tablet or smartphone?
6. Was your child able to use Pickstar alone (without assistance)?
7. What would you recommend adding to Pickstar, and why?
8. What do you consider the positive and negative aspects of Pickstar to be, and
why?
Fig. 5. Playing in the “Time-based” Level.

9
S.M. Shohieb et al. Entertainment Computing 42 (2022) 100495

Table 6
Usability Objective and Subjective Measures.
Usability The Question Number and the Number of positive
Attribute corresponding Metric in Pickstar responses

Effectiveness Q5: Device performance with the game 7/7

Q7: Increase child’s concentration 4/7
Q8: Lessons go smoothly 7/7
Efficiency Q4: Grading the action of the child 7/7
Q6: Needs a little help from the caregiver 6/7
Q7: Willing to express the names of 6/7
things after Pickstar 6/7
Q7: Willing to make oral exercise after 5/7
Pickstar
Q7: Willing to speak after Pickstar
User Q1: Ease of teaching 6/7
Satisfaction Q2: Ease of calibrating difficulty levels of 7/7
vocabulary 5/7
Q3: Fits the caregivers’ needs 4/7
Q8: Willing to continue learning

Fig. 7. Children’s skill levels (X) against their winning rate (Y) in classical
levels. The correlation coefficient = 0.32.
consequences. Participant confidentiality has been maintained through
and after the completion of the study.
game independently.
Authors conducted quantitative analyses on the collected post-study
4.3. Data collection interview content.

There were two types of data collected in Pickstar. Data from care­
givers of children with ASD and data logged while children with ASD
were playing the game.
Data collection from caregivers was performed in two stages. In the
first stage, the caregivers were asked certain questions before the user
study (Table 4), which concerned their perceptions of what methods
they used with their children, the problems they observed during
learning processes, and the progress they noticed with their children.
Then, another set of questions was asked to the caregivers after the user
study (Table 5) that in turn concerned the perceptions of the caregivers
about utilizing Pickstar. All presented questions were created based on
usability measurements for game development standards [18] and ISO
25062:2006 (last reviewed and confirmed in 2019) (International Or­
ganization for Standardization (ISO), 2019). Questions were in Arabic
language and translated into English after the study was conducted for
readability reasons.
Children were asked to play the game about 10–15 min every day for
the duration of four weeks. This time is recommended by autism spe­
cialists who helped us. As explained above, Pickstar is developed to keep
the log complete record for data obtained from each child playing the
4.4. Statement of community involvement
This mobile application was developed according to the feedback
and via the involvement of the autism community in Cairo, Egypt.
Parents of children with ASD provided valuable information that guided
the creation of the application via the pre- and post-study interviews
conducted with them by the researchers. In the pre-study interviews,
parents shared the challenges they experienced especially in terms of
education, and in the post-study interviews, they reported their feed­
back on the designed application after they had their children use it and
their observations on how much their child interacted with, learned
from, and was able to independently use the application.
5. Data analysis and results
5.1. Analysis of data collected from the pre-study interviews
The initial analysis was performed on questions that the caregivers
have been asked before the study to guide the design of Pickstar
(Table 4), about the caregiver perceptions on learning processes of their
children, relevant problems they experience, and improvements they
observe during those processes in their children. Their opinions were
gathered on using traditional methods such as the traditional PECS
technique [13].
The answers to the initial questionnaire revealed that parents of
children with ASD usually used pictures, videos with audio and educa­
tional sheets given by the therapists. When asked about the problems
they experienced with the traditional teaching procedures and mate­
rials, parents reported that reaching the location of the special education
session was a problem and the children sometimes got bored of the
materials. They also indicated that special education materials were
expensive to purchase.
Therefore, Pickstar was designed to address these issues of accessi­
bility, affordability, and creating engaging educational materials for
individuals with ASD.

5.2. Post-Study User Interviews and Usability Evaluation of Pickstar

Fig. 6. Children’s skill levels (X) against their winning rate (Y) in Adaptive­
MiniMax or DDA. The correlation coefficient = 0.04. (Based on ISO 25062)

The second analysis was performed on questions that were asked to

10
S.M. Shohieb et al. Entertainment Computing 42 (2022) 100495

Fig. 8. Results of the post questionnaire with the criteria measured for each question.

they may prefer tablets. Parents appreciated the accessibility of the

Table 7
application as it could be used anywhere, how it decreased the care­
Objective and Subjective Measures of Usability.
givers’ burden, how it could be used semi-independently or with simple
Child# Time spent in Classical levels in Time spent in DDA levels in guidance, and how smoothly the lessons went. They recommended the
minutes/week minutes/week
inclusion of more vocabulary and especially verbs and facial
1 29 42 expressions.
2 32 47 Comparing Performance of Children When Playing the DDA-Mode
3 26 62
4 23 51
vs. Playing the Classical Levels-Mode
5 19 63 The DDA-mode that is activated after enough data accumulates with
6 31 49 the classical levels-mode is compared with classical levels. The perfor­
7 39 61 mance of children when playing the DDA-mode vs. playing in the clas­
Total 199 375
sical levels-mode is presented in Figs. 6 and 7 respectively.
A comparison of two figures shows that in Figs. 6 and 7 depicting
the caregivers after the user study, which are formulated as seen in DDA, all the plotted points lay in a region of [50% to 80%], independent
Table 5. This analysis was based on ISO 25062:2006 [42], which is a from the players’ skill levels, whereas in Fig. 8 depicting classical levels,
standard method for reporting the usability evaluation of an application. the range is between 20% and 100%. Therefore, in DDA, the strongest
These questions investigated caregivers’ experience with and feed­ and the weakest players have the same winning rate, approximately
back about using Pickstar. The usability measurements with each metric 70%, which is important for the motivation of players with all skill levels
response are shown in Table 6. to continue playing the educational game. The correlation coefficient of
Interpreting logged data in combination with parent reports revealed DDA is smaller than that of the classical levels-mode (see Figs. 6 and 7).
that the Pickstar application was attractive to children, as seen from their This smaller correlation indicates better adaptation of the model to the
interest in clicking on the images and words on the Pickstar UI. Also, the child’s levels, therefore, the correlation coefficient of 0.04 indicates that
children were engaged by both the voiced pronunciation of the word and AdaptiveMiniMax shows good adaptive qualities.
the sound effect of the noise the animal makes. After the application voiced Analyzing the logged data for the children participants yielded the
the words, the children would react posteriorly by repeating the words after data in Table 7, which shows the time spent in classical levels and DDA
listening to them pronounced through Pickstar’s UI. Caregivers, addition­ levels for each participant. From this data, we calculated the engage­
ally, mentioned that the application sounds provided uniformity in com­ ment rate of the children in the learning process. Engagement rate for
parison with enunciating pronunciations to the children, which better the DDA feature was 88.4% higher than for the classical level. We can
accommodated the children’s learning. conclude that, for Pickstar, using AdaptiveMiniMax to implement the
Most of the participants gave positive responses to the questions, DDA feature resulted in the agent successfully adapting to the various
which revealed satisfactory user experience with the game. Fig. 8 reveals skill levels of human players. The DDA feature in Pickstar game also
that parents believed the application to fit caregiver needs, they used the seems to have increased the independence and engagement in the
game almost every day at home, and found it easy to use the application, learning process of children with ASD, consequently leading them to
to calibrate the difficulty levels of teaching materials, and to teach acquire much more vocabulary, which is what the game set out to
language via Pickstar. accomplish.
The parents reported that before Pickstar, they could do teaching
sessions with their children 3–4 times per week but with Pickstar, they 6. Discussion
were able to hold teaching sessions almost every day. Parents indicated
that the application ran well on mobile phones, though they added that Carrying paper-based education for ASD into technological platforms

11
S.M. Shohieb et al.
is an important mission that can make education more accessible and
affordable for families with ASD, for whom costs are a significant
burden. One of the most important topics included in the educational
curricula for ASD is communication and language. To improve the
communication skills of children with ASD, using paper-based cards as is
done in PECS can be difficult because of many factors, including the still
nature of the cards and the learning stimulation being restricted to
visible senses. Here, a new educational mobile game for children with
ASD was developed to overcome the limitations of traditional methods
and materials, accessibility issues of special education, and other bur­
dens on caregivers. This game we called Pickstar uses two learning ap­
proaches; the classical levels advancement approach, in which the levels
increase in difficulty, and the Dynamic Difficulty Adjustment (DDA)
approach. The DDA is performed with feature extraction of the playing
process and adapting the selected level for the player based on his/her
skill classification. We compared the performance of children when
playing the DDA-mode vs. playing the classical levels-mode. Engage­
ment rate of children with ASD in the learning process for the DDA
feature was 88.4% more than in the classical level. We concluded that in
DDA only, the strongest and the weakest players have the same winning
rate, approximately 70% and that DDA showed better adaptation of the
model to the participants’ skill levels. These two characteristics likely
yield better engagement than classical levels, since children with
different skill levels win at similar rates and therefore will be similarly
motivated to continue playing the educational game, and they will not
be disappointed by losing in a level that is not according to their skill
level as no such incongruous level will be presented by the adaptable
DDA.
Pickstar was created based on design processes for HCI science and
guidelines to develop learning games for children with ASD. A user study
was carried out to test how children with ASD acquire and use vocab­
ulary. Then, these observations were incorporated into the first proto­
type game. Finally, the game was refined through testing and the
feedback received after the initial implementation. A usability evalua­
tion was performed based on ISO 25062:2006 (we used the last version
in 2019) (ISO, 2019) on the data obtained from the interviews and it
revealed that children with ASD were able to independently use the
application and match the vocabulary with the respective pictures,
which is an important consideration when designing applications for
children with ASD. Otherwise, the aim of making education more
accessible on mobile devices would not have been attained, as children
would not be able to use the application independently and still require
assistance from special education teachers. Here, it was shown that just
with the supervision of their parents, children with ASD were able to
independently play and benefit from this application.
As an answer to the first research question that explored the ways
that Pickstar impacted the learning outcomes of Arab children with ASD,
it was seen that the DDA feature of the Pickstar game increased the in­
dependence of children with ASD from caregivers and parents, and
increased the children’s engagement with the learning process. Through
these effects, Pickstar enabled children with ASD to acquire more vo­
cabulary, which was the main target of the game.
The game was able to address certain problems of the traditional
methods, as its mobile interface stimulated children’s auditory senses
through the playing back of the sounds related to the picture playing
cards; their tactile experience through the taping of virtual artefacts
inside the UI; and visual sense via the colorful graphical display and set
of pictures.
To answer the second research question, which was the degree to
which Pickstar can reduce the burden on parents and caregivers of
children with ASD, analysis of caregiver feedback was performed. These
analyses revealed the effectiveness of Pickstar as an assistive teaching
application that decreases the burden on parents and caregivers, as
evident from the numerical responses to the questionnaires presented to
caregivers (see Fig. 8). Six out of seven parents reported that their child
was able to match the picture with the corresponding word
12
Entertainment Computing 42 (2022) 100495
independently, under the supervision of their parent/caregiver. This
means that the user interface was clear enough and the task was un­
derstood by children with ASD to the extent of accomplishing the given
vocabulary tasks by them. The caregivers expressed that it became
simpler to educate children with ASD using Pickstar. Moreover, the
caregivers mentioned that Pickstar adequately satisfied a number of
their teaching needs. Consequently, Pickstar seems to emerge as a sturdy
teaching resource that can support the teaching process for the care­
givers of children with ASD.
In sum, Pickstar alleviated some of the burden of educational ma­
terial preparation on parents, and the DDA feature of Pickstar increased
independent utilization by children with ASD and their engagement
with the learning material.
Some limitations of Pickstar were noticed during the testing process.
Some touches were not recorded by the application because of a weak
touch or the slowness of the touch response. Considering that some in­
dividuals with ASD may have weak muscles and may need longer to
respond, future versions can make the application more sensitive to
weaker touches to register them as responses and wait for longer or give
cues to expedite responding. Another limitation that segues into a future
direction was that some parents reported that they hoped the app had
included complete sentences. Future versions can include full sentences
as well as words, or create full sentences with the learned words for
improving communication skills of children with ASD. Finally, the small
sample size was a limitation. In the future, the accuracy of the DDA
algorithm can be refined after increasing the number of users and
recording more results from different users with ASD from different age
groups and with different individual characteristics. Though our appli­
cation was intended to be used independently by children with ASD for
educational purposes, future studies can design mobile applications that
aim to improve joint attention, which is considered to be a key
measurable construct for defining the social communication impairment
in ASD and is suggested to be a focus of early interventions [16]. Such
applications targeting joint attention have been started to be designed
for children with ASD in China [43,52], and similar endeavors can be
done for children with ASD in different countries and by using the DDA
framework presented here to enhance such joint attention applications
for children with ASD.
This game design, user experience information, limitations, and
recommendations by parents reveal certain future directions for Pickstar
that can be grouped in four domains, as detailed below.
6.1. Improving the interaction
During the utilization of the application, some children with ASD
were observed to have weak finger muscle coordination or weak arms,
resulting in the application not recording their response and requiring
them to touch again. While they were trying to tap on the mobile screen,
their excessive taps with their index finger sometimes caused the
existing screen to crash. In such cases, the application incorrectly
identified that action as scrolling instead of tapping, which can be
improved in further updates or designs.
6.2. Enhancing the content
One further goal emerging from this research is to enhance the
Pickstar game to teach complete sentences to children with ASD in
Arabic language. Also, some caregivers (specifically three of them P2,
P4, T2) gave us feedback that we can add acting verbs and visual re­
actions (facial expressions that can teach children the different emo­
tions) to the Pickstar vocabulary collection.
6.3. Refining the user-centered design
Another future goal is to enhance the user-centered design by
recruiting more volunteers as users and implementing their feedback for
S.M. Shohieb et al.
improvements to refine the game and enable it to address the needs and
preferences of children with ASD in a better way than before.
6.4. Heightening the accuracy of the DDA
In machine learning, the more data points there are, the more ac­
curate the system becomes, because machine learning algorithms like
the DDA cannot recognize or predict user behavior that it has not
encountered or trained with before. As more children with ASD use
Pickstar in the future and the applied algorithm is tested on this larger
data, its algorithm will become more accurate, since it will collect in­
formation about the range of behaviors that can be displayed by users
with ASD.
7. Conclusion
Pickstar aimed to address the issue of the availability of educational
and therapeutic methods for ASD and to mitigate caregivers’ problems
in accessing educational materials. Pickstar seems to have attained
considerable contributions such as developing a DDA in a game dedi­
cated for children with ASD, which maximizes their engagement, in­
dependence as well as enjoyment. Pickstar helped in alleviating the
teaching and assessment encumbrances experienced by the caregivers of
Arab children with ASD via enhancing the children’s learning motiva­
tion, and stimulating the different sensations of vision, touch, and
audition, thereby creating a more engaging and accessible learning
environment for children with ASD than the solely visual, static, tradi­
tional paper-based learning methods. It is hoped that Pickstar will serve
as a model for future educational applications for ASD communities
across the world. This framework can guide the development of mobile
applications not just for language but for other problematic domains as
well, and to be extended for utilization by individuals with other dis­
abilities, such as children with Down syndrome, cerebral palsy, and
other limited speech disabilities.
Funding
The authors would like to thank the ↱Deanship of Scientific Research
at Umm Al-Qura University ↱for supporting this work by Grant:
22UQU4331164DSR01 Code: 22UQU4331164DSR01.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
References
[1] A. Alessandrini, A. Cappelletti, M. Zancanaro, Audio-augmented paper for therapy
and educational intervention for children with autistic spectrum disorder, Int. J.
Hum Comput Stud. 72 (4) (2014) 422–430, https://doi.org/10.1016/j.
ijhcs.2013.12.001.
[2] S. Al-Salehi, E. Al-Hifthy, M. Ghaziuddin, Autism in Saudi Arabia: presentation,
clinical correlates and comorbidity, Transcultural Psychiatry 46 (2) (2009)
340–347, https://doi.org/10.1177/1363461509105823.
[3] American Psychiatric Association. Desk Reference to the Diagnostic Criteria from
DSM-5. Washington, DC: American Psychiatric Publishing, 2013.
[4] NIDHD. Autism Spectrum Disorder: Communication Problems in Children.
National Institute on Deafness and Other Communication Disorders. Retrieved on
May 30, 2020 from https://www.nidcd.nih.gov/health/autism-spectrum-disorder-
communication-problems-children, 2020.
[5] M.Z.A. Aziz, S.A.C. Abdullah, S.F.S. Adnan, L. Mazalan, Educational App for
children with Autism Spectrum Disorders (ASDs), Procedia Comput. Sci. 42 (2014)
70–77.
[6] M.K. Belmonte, E.J. Weisblatt, A. Rybicki, B. Cook, C.S. Langensiepen, D.J. Brown,
P. Karanth, in: October). Can computer-assisted training of prerequisite motor
skills help enable communication in people with autism?, IEEE, 2016, pp. 13–20.
[7] J. Bisoglio, T.I. Michaels, J.E. Mervis, B.K. Ashinoff, Cognitive enhancement
through action video game training: Great expectations require greater evidence,
Front. Psychol. 5 (2014) 136, https://doi.org/10.3389/fpsyg.2014.00136.
13
Entertainment Computing 42 (2022) 100495
[8] A. Bondy, L. Frost, A picture’s worth: PECS and other visual communication
strategies in autism, Woodbine House, 2011.
[9] B. Bossavit, S. Parsons, Designing an educational game for and with teenagers with
high functioning autism. In: Proceedings of the 14th Participatory Design
Conference: Full papers-Volume 1, 2016, August), pp. 11-20.
[10] R.S. Boyer, J.S. Moore, MJRTY - A fast Majority vote algorithm in Boyer, R. S. (ed.),
Automated Reasoning: Essays in Honor of Woody Bledsoe, Automated Reasoning
Series, Dordrecht, Kluwer Academic Publishers, 1991, pp. 105–117. doi:10.1007/
978-94-011-3488-0_5. Originally published as a technical report in 1981.
[11] M.E. Chien, C.M. Jheng, N.M. Lin, H.H. Tang, P. Taele, W.S. Tseng, M.Y. Chen,
iCAN: A tablet-based pedagogical system for improving communication skills of
children with autism, Int. J. Hum Comput Stud. 73 (2015) 79–90.
[12] J.E. Cohen, M.A. Lemley, Patent scope and innovation in the software industry,
California Law Review 89 (1) (2001) 4–21.
[13] L. Collet-Klingenberg, PECS: Steps for implementation. Madison, WI: The National
Professional Development Center on Autism Spectrum Disorders, The Waisman
Center, The University of Wisconsin, 2008, Retrieved on May 30, 2020 from htt
ps://autismpdc.fpg.unc.edu/sites/autismpdc.fpg.unc.edu/files/PECS_Steps.pdf.
[14] A. Constantin, H. Johnson, E. Smith, D. Lengyel, M. Brosnan, Designing computer-
based rewards with and for children with Autism Spectrum Disorder and/or
Intellectual Disability, Comput. Hum. Behav. 75 (2017) 404–414.
[15] M. Davis, K. Dautenhahn, S. Powell, C. Nehaniv, Guidelines for researchers and
practitioners designing software and software trials for children with autism,
J. Assistive Technol. 4 (1) (2010) 38–48.
[16] G. Dawson, R. Bernier, R.H. Ring, Social attention: A possible early indicator of
efficacy in autism clinical trials, J. Neurodevelopmental Disorders 4 (1) (2012)
1–12.
[17] Z.S. de Urturi, A.M. Zorrilla, B.G. Zapirain, July). Serious Game based on first aid
education for individuals with Autism Spectrum Disorder (ASD) using android
mobile devices, in: In Proceedings of the 16th International Conference on
Computer Games (CGAMES), 2011, pp. 223–227.
[18] Digital Game Development Standards, 2013. Retrieved on May 30, 2020 from.
[19] C. Doenyas, S.M. Shohieb, Leveraging technology for the wellbeing of individuals
with autism spectrum disorder and their families during Covid-19, Front.
Psychiatry 795 (2021), https://doi.org/10.3389/fpsyt.2021.566809.
[20] C. Doenyas, E. Şimdi, E.Ç. Özcan, Z. Çataltepe, B. Birkan, Autism and tablet
computers in Turkey: Teaching picture sequencing skills via a web-based iPad
application, Int. J. Child-Comput. Interact. 2 (1) (2014) 60–71.
[21] M. El-Hady, S. Shohieb, A comprehensive design of a structured technology-based
therapy to prevent post intensive care syndrome (CogGYM platform), Int. J.
Entertainment Technol. Manage. 1 (1) (2020) 80–94.
[22] B. Galitsky, A computational simulation tool for training autistic reasoning about
mental attitudes, Knowl.-Based Syst. 50 (2013) 25–43.
[23] D. Genç-Tosun, O. Kurt, Otizmli bireylerin kullandığı yeni nesil konuşma üreten
cihazlara ilişkin araştırmaların incelenmesi [A review of the use of new generation
speech generating devices by individuals with autism], Ankara Üniversitesi Eğitim
Bilimleri Fakültesi Özel Eğitim Dergisi [Ankara University Faculty of Educational
Sciences Journal of Special Education] 18 (01) (2017) 125–147, https://doi.org/
10.21565/ozelegitimdergisi.277442.
[24] H. Goldstein, Communication intervention for children with autism: A review of
treatment efficacy, J. Autism Dev. Disord. 32 (5) (2002) 373–396.
[25] I. Granic, A. Lobel, R.C.M.E. Engels, The benefits of playing video games, Am.
Psychol. 69 (1) (2014) 66–78, https://doi.org/10.1037/a0034857.
[26] R. Hunicke, The case for dynamic difficulty adjustment in games. Proceedings of
the 2005 ACM SIGCHI International Conference on Advances in computer
entertainment technology - ACE ’05, 2005, pp. 429-433.
[27] E. Husni, Mobile applications BIUTIS: Let’s study vocabulary learning as a media
for children with autism, Procedia Technol. 11 (2013) 1147–1155.
[28] V.K. Jensen, L.V. Sinclair, Treatment of autism in young children: Behavioral
intervention and applied behavior analysis, Infants & Young Children 14 (4)
(2002) 42–52.
[29] N.N. Kamaruzaman, N. Jomhari, Z.M. Yusoff, Digitizing ABAHATA Al-Jabari
Method using LACIP Skill Theory to Support Learning Al-Quran for Children with
Low Functioning Autism: A Review Towards al-Quran Perspective, QURANICA-
International J. Quranic Res. 7 (1) (2015) 1–16.
[30] L. Kanner, Follow-up study of eleven autistic children originally reported in 1943,
J. Autism Childhood Schizophrenia 1 (2) (1971) 119–145.
[31] K. Khowaja, S.S. Salim, A framework to design vocabulary-based serious games for
children with autism spectrum disorder (ASD), Univ. Access Inf. Soc. 19 (4) (2020)
739–781.
[32] A. Lerna, D. Esposito, M. Conson, L. Russo, A. Massagli, Social–communicative
effects of the Picture Exchange Communication System (PECS) in autism spectrum
disorders, Int. J. Language Commun. Disorders 47 (5) (2012) 609–617.
[33] T. Malone, Toward a theory of intrinsically motivating instruction, Cognitive
Science 5 (4) (1981) 333–369.
[34] N.A. Meguid, Egypt and Autism. In Encyclopedia of Autism Spectrum Disorders, in: F.
R. Volkmar (Ed.), Encyclopedia of Autism Spectrum Disorders, Springer New York,
New York, NY, 2015, pp. 1–9.
[35] V. Mendonça, L. Coheur, A. Sardinha, Vithea-kids: A platform for improving
language skills of children with autism spectrum disorder, in: Proceedings of the
17th International ACM SIGACCESS Conference on Computers & Accessibility,
ACM, 2015, October, pp. 345-346, doi: 10.1145/2700648.2811371.
[36] O. Missura, Dynamic Difficulty Adjustment. Ph.D Thesis, Faculty of Mathematics
and Natural Sciences Rheinische Friedrich-Wilhelms-University, Bonn, Germany,
2015.
S.M. Shohieb et al.
[37] L. Nacke, The formal systems of games and game design atoms. The Acagamic,
2010, Retrieved on May 30, 2020 f from http://www.acagamic.com/courses/infr1
330-2014/the-formal-systems-of-games-and-game-design-atoms/.
[38] N. Namri, Where is the Arab world today from autism?, 2015, Retrieved on May
30, 2020 from https://raseef22.com/life/2015/04/02/autism-in-the-arab-world/.
[39] Nature Middle East, Treatment of autism disorder in the Arab world, 2013¸
Retrieved on May 30, 2020 from http://www.natureasia.com/ar/nmiddleeast/art
icle/10.1038/nmiddleeast.2011.154.
[40] S. Russell, P. Norvig, Artificial intelligence: A modern approach, fourth ed.,
Pearson, 2020.
[41] J.M. Smyth, Beyond self-selection in video game play: An experimental
examination of the consequences of massively multiplayer online role-playing
game play, CyberPsychology & Behavior 10 (5) (2007) 717–721, https://doi.org/
10.1089/cpb.2007.9963.
[42] International Organization for Standardization (ISO). Software engineering —
Software product Quality Requirements and Evaluation (SQuaRE) — Common
Industry Format (CIF) for usability test reports (2019) Iso.org, 2019, Retrieved on
May 30, 2020 from https://www.iso.org/obp/ui/#iso:std:iso-iec:25062:ed-1:v2:
en.
[43] T.Y. Tang, P. Winoto, A. Guan, On active sharing and responses to joint attention
bids by children with autism in a loosely coupled collaborative play environment,
in: Proceedings of the 2017 Conference on Interaction Design and Children, 2017,
June, pp. 717-720.
[44] T.Y. Tang, J. Xu, P. Winoto, Automatic object recognition in a light-weight
augmented reality-based vocabulary learning application for children with autism,
in: Proceedings of the 2019 3rd international conference on innovation in artificial
intelligence, 2019a, March, pp. 65-68.
[45] T.Y. Tang, J. Xu, P. Winoto, An augmented reality-based word-learning mobile
application for children with autism to support learning anywhere and anytime:
object recognition based on deep learning, in: International Conference on Human-
Computer Interaction, Springer, Cham, 2019b, July, pp. 182-192.
[46] P. Vellanki, T. Duong, S. Venkatesh, D. Phung, August). Nonparametric discovery
of learning patterns and autism subgroups from therapeutic data, in: In Pattern
14
Entertainment Computing 42 (2022) 100495
Recognition (ICPR), 2014 22nd International Conference on Pattern REcognition,
2014, pp. 1828–1833, https://doi.org/10.1109/ICPR.2014.320.
[47] When Something’s Wrong: Strategies for Teachers, 2007. Retrieved on May 30,
2020 from https://jack.org/jack/media/images/PDF/WSW-teacher-final-print.
pdf.
[48] C. Williams, B. Wright, G. Callaghan, B. Coughlan, Do children with autism learn to
read more readily by computer assisted instruction or traditional book methods? A
pilot study, Autism 6 (1) (2002) 71–91, https://doi.org/10.1177/
1362361302006001006.
[49] P. Winoto, June). Reflections on the adoption of virtual reality-based application
on word recognition for Chinese children with Autism, in: In Proceedings of the
The 15th International Conference on Interaction Design and Children, 2016,
pp. 589–594.
[50] P. Winoto, T.Y. Tang, October). Two lightweight and customizable picture-based
word-learning mobile applications for Chinese children with autism, in: In
Proceedings of the 2018 ACM International Joint Conference and 2018
International Symposium on Pervasive and Ubiquitous Computing and Wearable
Computers, 2018, pp. 291–294.
[51] P. Winoto, T.Y. Tang, A. Guan, I will help you pass the puzzle piece to your partner
if this is what you want me to: The design of collaborative puzzle games to train
Chinese children with autism spectrum disorder joint attention skills, in:
Proceedings of the The 15th International Conference on Interaction Design and
Children, (2016, June), pp. 601-606.
[52] P. Winoto, C.N. Xu, A.A. Zhu, in: “Look to remove”: A virtual reality application on
word learning for Chinese children with autism, Springer, Cham, 2016,
pp. 257–264.
[53] D. Xu, Y. Song, S. Ren, L. Wang, J. Zhao, December). Rehabilitation training system
for children with autism, in: In Mobile Ad-hoc and Sensor Networks (MSN), 2013
IEEE Ninth International Conference on, 2013, pp. 494–497, https://doi.org/
10.1109/MSN.2013.88.
[54] M. Zohaib, Dynamic Difficulty Adjustment (DDA) in computer games: a review,
Adv. Human-Computer Interaction 2018 (4) (2018) 1–12.