A Low-Cost Autonomous Attention Assessment

System for Robot Intervention with Autistic Children

Fady S. Alnajjar Abdulrahman Majed Renawi Massimiliano Cappuccio

DepDUWPHQW of Computer Science and 'HSDUWPHQWRI&RPSXWHU6FLHQFHDQG DepDUWPHQW of Philosophy
Software Engineering, UAE University 6RIWZDUH(QJLQHHULQJ8$(8QLYHUVLW\ UAE University
$O$LQUAE $O$LQ8$( $O$LQ, UAE
fady.alnajjar@uaeu.ac.ae abdulrahman.m@uaeu.ac.ae PORUHQ]R#XDHXDFDH

Omar Mubain
WesternSydney University
Australia
o.mubin@westernsydney.edu.au

Abstract— Attention is an essential mental process that is yet they are applied only in controlled experimental setups and
important to achieve learning progress. We cannot get better in not intensively focusing on automatic attention level
our academic learning unless we concentrate our attention on assessment [3,4].
the person giving the educational material, such as the teacher
or trainer. Children with Autism Spectrum Disorder (ASD) may
have attention difficulties that can directly influence their
academic skills. In recent years, robot intervention in autism
therapy and assessment is becoming a popular research topic
due to its role in enhancing children's attention more than a
regular human therapist, as well as, the increasing number of
autism children compared to the availability of professional
therapists. Robot intervention helps in reducing therapy time
and makes early therapeutics sessions easier and much
promising. Many researches have been conducted to develop
robot intervention techniques for ASD children, and some
methods have already been used to assess autistic individuals’
attention during the robot intervention sessions. Yet, the
existing attention assessment methods are either very complex
or simple with one measured interaction cue only. This paper
presents a practical and low-cost automatic approach to assess
autistic individuals’ attention during robot intervention;
addressing multiple interaction cues. Experimental results
show that the proposed attention assessment system could
accurately measure the child attention and enhance therapy
progress. This automatic attention system can open a new era Fig. 1. The NAO robot with our custamized chest holder to carry a
for utilizing technologies to monitor students’ attentions in the mobile to enhance the robot’s sensory & motor capabilties.
class to enhance educational systems.

Keywords—Autism Therapy, Robot Intervention,

NAO Robot, Assessment System, Autism Diagnosis
I. INTRODUCTION
Children with autism are usually experiencing
neurodevelopmental disorders characterized by attention
deficit and learning difficulties [1]. Earning the autistic
children attention (i.e., perform eye/face contact, focus on the
information they hear while filtering the noises and perform
the task assigned to them), in a complex environment is,
therefore, an extreme challenge. This challenge can affect
their learning and academic skills and their potential for long-
term success.
Autism therapy and assessments with robots intervention
is a trending research topic, since the statistics show
noticeable increasing rates in the number of autistic
individuals in UAE compared to professional therapists [2].
There are plenty of researches addressing autism therapy and
assessments techniques with the aid of robotic interventions,
Recent researches on robotics intervention present several
techniques to diagnose autistic individuals using robots and
accompanying sensors [4,5]. Some of these techniques present
a score-like assessment to represent the interaction level
between the robot and the autistic individual [6]. These scores
have several forms to represent multiple interaction and
attention cues. Previous literature informs us that attention
definition in the context of robotic autism assessment could be
modeled to include various factors such as:
(1) eye/face attention: represents how much the individual
is paying eye/face contact to the robot. This can be measured
using a variety of measurement techniques, ranging from eye-
pupil diameter measurement [7] to image processing utilizing
the robot’s onboard camera [8].
(2) Joint attention: represents the ability of the individual
to react to robot’s motoric actions, like looking at the same
object that the robot is pointing to. This can be measured using
the same hardware used to measure attention with different

978-1-5386-9506-7/19/$31.00 ©2019 IEEE 9–11 April, 2019 - American University in Dubai, Dubai, UAE
2019 IEEE Global Engineering Education Conference (EDUCON)
Page 787
 data processing techniques [3] or it can be measured
separately using a motion tracking method [9].
(3) Facial emotions: detection of the facial reaction of the
individual interacting with the robot. Detecting facial features
can be used to detect emotions and other aversion and
attention cues [10]. A camera could be enough to detect facial
features, and a depth camera [11] can be used for better
accuracy. The output of facial features detection helps in
understanding the emotional state of the individual to relate it
to other attention measures.
(4) Sound/Linguistic response: the ability to answer the
given questions during the interventions within a certain time.
(5) Imitation response: the ability to imitate the robot’s
movements.
Most of the existing assessment techniques in robot
intervention studies either use external hardware components
not mounted on the robot, which limits portability, or target a
single attention cue due to robot’s onboard hardware
limitations. This paper presents a method that uses upgraded
robot onboard/mounted hardware targeting automatic
assessment of the above mentioned attention cues.
II. METHODOLOGY
This study introduces a simple and efficient numerical
assessment system for robot intervention without the use of
external monitoring equipment. The utilized robot for this
study is the humanoid robot Nao from Softbank Robotics with
our added mobile screen mounted on the robot body, shown
in Fig. 1.
A. Technical and hardware configurations
For the robot intervention with autistic children, beside
regular Nao robot intervention [6], we have added mobile
based hardware with Facial Features Display to enhance the
intervention, as seen in Fig. 1. The presented approach does
not compromise the mobility of the robot and ensures the
applicability in human-robot interaction scenarios. The
display is assembled with the aid of a custom chest-mount
designed for the NAO robot, shown in Fig. 2, to minimize the
changes in the moment of inertia due to the added weight. The
added display is a mobile phone that shows emotions running
a custom designed mobile application named Emotions
Selector. A mobile magnetic holder was customized to fit on
the chest-mount to carry the mobile screen. The mobile
application was designed using the MIT App Inventor [12], it
receives the emotion control data from the computer via
socket communication over Wi-Fi. The application is made
easy to use without complications other than specifying the
computer’s IP address to link it with the mobile automatically.
Fig. 2 [a] The customized NAO chest holder front side with the mobile
magnetic holder. [b] is the holder back side with the added strap to
fasten it.
978-1-5386-9506-7/19/$31.00 ©2019 IEEE 9–11 April, 2019 - American University in Dubai, Dubai, UAE
2019 IEEE Global Engineering Education Conference (EDUCON)
Page 788
The used chest holder (chest-mount) is designed in our lab
using Fusion 360 CAD Software to be a multipurpose tool for
holding any object using up to four screws for rigidity. Velcro
straps are used to fasten the holder from the back. It can be
used by other researchers to carry a mobile, a camera, a laser
sensor or any other item on the chest of NAO robot. Moreover,
the USB socket on the head of NAO robot is used to charge
the mobile phone. This technique maintains the robot’s
equilibrium ability, thus robot’s built-in features can be used
extensively during the diagnosis and therapy processes.
B. Autonomous Assessment system
Since the target of this paper is to particularly present the
autonomous assessment system of the robot intervention,
rather than the intervention technique itself, the following
sections will highlight the assessments factors:
Attention Score:
The mounted mobile’s camera is used for facial
recognition to produce and accumulate the attention score
(i.e., how much the individual is paying eye/face contact to the
robot, thus a smiley face indicates maximum attention). Image
frames are retrieved online from the camera of the mobile,
acting as an IP camera, and processed using Haar classifiers
[13] to detect the participants face. The participant’s face is
detected if his/her face is directed towards the robot’s front
body. This assumes that the participant is paying attention to
the robot. Once the participant’s face is detected, the attention
score is incremented with a set increment value. As long as the
participant’s face is not detected, the attention score is
decremented by a smaller set decrement value. The increment
and decrement values are made variables that can be reset
based on future findings from experiments with autistic
participants. The attention score is designed to be a
comparison factor between several interaction sessions’
results for a single participant. Thus, its parameters should be
constant for each participant. In this study, parameters are
shown in Table I. The used increment and decrement values
affecting attention score are shown in the table
(experimentally tuned).
TABLE I. USED INREMENT AND DECREMENT VALUES AFFECTING
ATTENTION SCORE.
Increment Value Decrement Value
(points) (points)
Attention 5 1
Joint Attention 15 1
Sound Response Eq. (1) 5
Joint Attention Score:
This score part is designed to measure how the participant
reacts to robot motoric actions and requests. For instance, if
the robot talks about a box on the side and points towards it,
the participant is expected to look towards the same box. The
participant’s head motion detection is simplified to detect the
participant’s left and right ears. If the participant looks to
his/her right side, his/her left ear should appear to the robot.
And the same applies for his/her left side and right ear. This
detection is made using the same image frames retrieved from
the mobile camera. If the detected participant’s head direction
matches the current robot’s head direction, the joint attention
score is incremented by a set increment value; else, it is
decremented by a set decrement value (Table I). In addition,
 the attention score is incremented or decremented by set the participant’s current emotion among five possible
values in each joint attention instant, since attention score is emotions: happy, sad, angry, surprised and neutral. The most
designed to report on the general attention of the participant. probable emotions are logged and graphically represented as
color bands, each color represents a specific emotion

Sound Response:
The mobile microphone is used as an IP mic and defined
as a sound input in the laptop for sound response module. The
sound response module calculates the time between the time
instant when NAO robot finishes asking a question and the
time instant when the participant starts answering. This time
difference is reported as sound response time and it is used to
calculate an increment in the attention score. Lower sound
response time (faster response) generates a bigger increment.
The maximum increment due to the lowest response time
(ideally zero) can be reset, but it should be constant for the
same participant in all sessions for comparison.
This module has some parameters that can be adjusted
based on the participant’s normal speaking volume and
background noise. These parameters are:
x Threshold-sound-peak: detected peak sound level
after which the module assumes the person has started
speaking. Set to 2000Hz in the following experiments.
x Time-out: the maximum time the module waits before
killing the process and ignoring the sound response.
This value is expected to be bigger with participants
with speech difficulties. Set to 2.5 seconds in the
following experiments.
x Text-is-done-time-lapsed: the maximum time the
module waits for the robot to reply with a text-is-done,
after which the process is killed if robot didn’t finish
speaking. This value can be set to larger than the time
the robot takes to say the longest expected question.
Set to 5 seconds in the following experiments.
x Time-delay: a delay in the code to compensate for the
mic delay; the time between capturing a sound peak at
the mic and receiving the same sound peak received
at the computer. Set to 0.25 seconds in the following
experiments.
The increment in attention score due to sound response
(‫ )ݎܿ݊ܫ‬is made as a function of response time (ܴ݁‫ )݁݉݅ܶ݌ݏ‬and
maximum score for response time ( ‫ ) ݁ݎ݋ܿܵ݌ݏܴ݁ݔܽܯ‬as the Fig. 3. Assessment System and Scenario GUIs. The first page in the GUI
following Eq1: [a] is the assessment system control interface. It supports local and IP
cameras. Plot Results will plot the latest session results. The second [b]
‫ ݎܿ݊ܫ‬ൌ ‫ ݁ݎ݋ܿܵ݌ݏܴ݁ݔܽܯ‬െ
ெ௔௫ோ௘௦௣ௌ௖௢௥௘
ൈ ܴ݁‫( ݁݉݅ܶ݌ݏ‬1) and third [c] pages are the English and Arabic versions of the desinged
்௜௠௘ି௢௨௧ scenario, respectivly. They can be accessed from Open Scenario button.
Based on the above increment equation, the faster the sound
response, the more attention score increment is granted. The C. Experimental Setup:
used value of ‫ ݁ݎ݋ܿܵ݌ݏܴ݁ݔܽܯ‬in the following experiments is
25 points. Our robot intervention scenario is divided into two parts
[6], a control part, in which the therapist can control the robot
Moreover, the IP mic allows the operator to hear what the from operator’s room, Fig. 4[b] (more details are in [6]). And
mobile hears; what the participant and the robot say, without the other part is the experimental room in which the child
interfering with the sound response module. All of these interacts with the robot and our system assesses the interaction
communications were made over Wi-Fi with different ports. (Fig. 4[a]).
Emotions Detection: The control part is made as a graphical interface as shown
The proposed assessment system is also able to estimate in Fig. 3 to be easy-to-use by non-technical operators like
the participant’s emotion from facial features with the aid of therapists. The operator should specify the maximum time
NaoQi [8] emotions detection Python module utilizing NAO limit and start the interaction session, having the ability to stop
robot onboard camera. The module returns the probability of at any time. During the diagnosis process, the operator can
control the flow of the predesigned scenario, having the ability

978-1-5386-9506-7/19/$31.00 ©2019 IEEE 9–11 April, 2019 - American University in Dubai, Dubai, UAE
2019 IEEE Global Engineering Education Conference (EDUCON)
Page 789
 to type a new response if the participant deviates from the
preset scenario. The scenario includes dialogues to interact
with the participant to grab his/her attention and test his/her
interaction capabilities. Sound responses, joint attention
requests and robot movements are automatically called in the
preset scenario parts.
5 experiments were carried out with healthy participants to
test the robustness of the designed control and assessment
systems. As stated before, the control station was placed in a
room different than the room where participant and the robot
are. Participant and robot were seated close to each other as
seen in Fig. 4[a]. A side camera was used to record and
monitor the experiments remotely.
III. RESULTS
The output assessment scores are accumulated in real-
time and shown as a separate diagnostic real-time plot as
shown in Fig. 5 to show the operator the level of interaction
between the autistic individual and the robot as attention and
joint attention scores. The real-time plot reports on the fast
responses’ values, sound response requests, attention score
and joint attention requests’ instants. The results are saved at
the end of each session as a ‘.pickle’ data file for further
processing. The operator can directly plot the recent
experiment’s results from the scoring system GUI. Fig. 6 and
Fig. 7 shows the results obtained from experiments done with
two different participants.
Fig.4. Robot Interaction Room [a] and Control Station [b]. Communication
is made over WiFi. The operator may not need any technical trainings to
use the control station. In the lower figure [b], the right screen is the monitor
showing the Robot Interaction Room to the operator, the middle screen is
the scenario control GUI and the left screen shows the real-time plot and the
robot general motions and behaviors controller.
978-1-5386-9506-7/19/$31.00 ©2019 IEEE 9–11 April, 2019 - American University in Dubai, Dubai, UAE
2019 IEEE Global Engineering Education Conference (EDUCON)
Page 790
Fig. 5. Real-Time Plot. The red line is the real-time attention score. The
blue line is the real-time joint attention score (this experiment had no joint
attention requests). Yellow ribbons mean a sound response request; a sound
repsonse was expected at this time instant. The numbers in white are the
fast sound responses; less than the set time_out value.
Final results after the end of the sessions are presented in
three subplots as:
1- Attention mixed scores plot: emotions color bands,
attention and joint attention scores
2- Sound Response Plot: shows the participants’
response time versus interaction time in seconds.
3- Emotions counter: shows how many times each
emotion has been detected.
Assessment system results show the expected interaction
level; after 100 seconds, the participant in Fig. 6 was not
looking at the robot for a duration of 95 seconds, which was
a clear drop in the attention score. Yet, the other participant,
Fig. 7, showed a better attention to the robot with smaller
drops in attention score.
Second participant, Fig. 7, didn’t have any joint attention
requests from the robot, while the first participant, Fig. 6, was
expected to show some joint attention cues, which is shown
by the increase in the joint attention score.
Sound Response results show that both participants were
partially responsive at all times of interaction, which is shown
in real-time plots of yellow bands without the response time
written on top of the yellow band. Not writing the response
time means that the response time was more than the set
timeout value. This is approved in the sound response plots,
where participants had many big values in response times,
which means that they were slow in their vocal responses at
those instants.
Emotion bands’ plots show the most probable emotion at
each instant compared to attention and joint attention scores
versus interaction time. This helps in matching participants’
interaction levels to emotional states. The emotions counter
makes it easier to know the dominant emotion compared to
other emotional states in the entire interaction session.
Moreover, some autistic traits could be tested with the
knowledge of participant’s emotional state. Both participants
show more ‘Happy’ emotion predictions at the beginning of
interactions than ‘Sad’, and more ‘Sad’ than ‘Happy’ at the
end of interaction. This shows a decay in the level of
excitement as the interaction time increases.

Fig. 6. Results of the experiment from the first participant. Upper plot are
the attention scores (red) and joint-attention scores (blue) normalized and
plot over emotions color bands. Attention and emotional states can be
mapped at each time instant. “None” emotion means that no emotion data
could be generated at that instant. Middle plot shows all sound responses
recorded. Slow sound responses are more than timeout value and they are
not plot in real-time. Fast sound responses are equal to or less than the
timeout value. Timeout value can be reset for each participant to match their
cases. Several parameters can be adjusted by the operator to customize the
scoring system based on participant’s case, this enhances attention cues
detection. Lower plot shows participants emotions counter; it is the overall
count of emotions detected during the session. These results are comparable
for the same participant in different sessions given the same parameters
values.
978-1-5386-9506-7/19/$31.00 ©2019 IEEE 9–11 April, 2019 - American University in Dubai, Dubai, UAE
2019 IEEE Global Engineering Education Conference (EDUCON)
Page 791
Fig. 7. Results of The Experiment With The Second Participant.
IV. CONCLUSION
A novel low-cost scoring system for robot intervention in
autism therapy and diagnosis was designed and tested on two
participants. Results show a match between the interaction
trend captured by the scoring system and observations. The
proposed scoring system uses a simplified hardware and
software setup that is easy to use by non-technical autism
therapists, as well as parents in domestic environments. This
scoring system facilitates using robots as daily assessment
tools and therapy assistants in autism schools, hospitals and
in domestic use. Our future direction is to apply the attention
testing in clinical setups to a cohort of autistic children with
attention deficit and compare the results with therapists’
 assessment results to test the validity of the proposed system
clinically. Having clinically validated results would help
develop a permanently available clinical system that can be
used in domestic setups as well.
REFERENCES
[1] J. L. Matson and M. Shoemaker, "Intellectual
disability and its relationship to autism spectrum
disorders," Research in developmental disabilities,
vol. 30, no. 6, pp. 1107-1114, 2009.
[2] V. Eapen, A. A. Mabrouk, T. Zoubeidi, and F.
Yunis, "Prevalence of pervasive developmental
disorders in preschool children in the UAE,"
Journal of Tropical Pediatrics, vol. 53, no. 3, pp.
202-205, 2007.
[3] S. M. Anzalone et al., "How children with autism
spectrum disorder behave and explore the 4-
dimensional (spatial 3D+ time) environment during
a joint attention induction task with a robot,"
Research in Autism Spectrum Disorders, vol. 8, no.
7, pp. 814-826, 2014.
[4] K. S. Lohan, E. Sheppard, G. Little, and G.
Rajendran, "Towards improved child robot
interaction by understanding eye movements," IEEE
Transactions on Cognitive and Developmental
Systems, 2018.
[5] J.-J. Cabibihan, H. Javed, M. Aldosari, T. W.
Frazier, and H. Elbashir, "Sensing Technologies for
978-1-5386-9506-7/19/$31.00 ©2019 IEEE 9–11 April, 2019 - American University in Dubai, Dubai, UAE
2019 IEEE Global Engineering Education Conference (EDUCON)
Page 792
Autism Spectrum Disorder Screening and
Intervention," Sensors, vol. 17, no. 1, p. 46, 2016.
[6] M. Alahbabi et al., "Avatar based interaction
therapy: A potential therapeutic approach for
children with Autism," in Mechatronics and
Automation (ICMA), 2017 IEEE International
Conference on, 2017: IEEE, pp. 480-484.
[7] Codamotion CX1-800 [Online]. Available:
https://codamotion.com/3d-measurement/.
[8] Aldebaran Nao robot [Online]. Available:
http://www.ald.softbankrobotics.com/.
[9] Kinect for Windows features [Online]. Available:
http://www.microsoft.com/en-
us/kinectforwindows/meetkinect/features.aspx.
[10] Y. Feng, Q. Jia, M. Chu, and W. Wei, "Engagement
Evaluation for Autism Intervention by Robots
Based on Dynamic Bayesian Network and Expert
Elicitation," IEEE Access, vol. 5, pp. 19494-19504,
2017.
[11] StereoLabs. zed-ros-wrapper [Online] Available:
https://github.com/stereolabs/zed-ros-wrapper
[12] S. C. Pokress and J. J. D. Veiga, "MIT App Inventor:
Enabling personal mobile computing," arXiv
preprint arXiv:1310.2830, 2013.
[13] P. I. Wilson and J. Fernandez, "Facial feature
detection using Haar classifiers," Journal of
Computing Sciences in Colleges, vol. 21, no. 4, pp.
127-133, 2006.