See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/333811621

Data Linking Testing Between Humanoid Robot and IoRT Network Server for
Autism Telerehabilitation System Development

Chapter · January 2020

DOI: 10.1007/978-981-13-8323-6_14

CITATIONS READS

2 173

5 authors, including:

Muhammad Aliff Rosly Syamimi Shamsuddin

Universiti Teknologi MARA Universiti Teknikal Malaysia Melaka, Durian Tunggal, Melaka, Malaysia
8 PUBLICATIONS   6 CITATIONS    53 PUBLICATIONS   671 CITATIONS   

SEE PROFILE SEE PROFILE

Hanafiah Yussof
Universiti Teknologi MARA
122 PUBLICATIONS   927 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Injection of methylprednisolon View project

Dept of Mechanical Eng, UKM, Malaysia View project

All content following this page was uploaded by Muhammad Aliff Rosly on 20 August 2019.

The user has requested enhancement of the downloaded file.

Data Linking Testing Between Humanoid
Robot and IoRT Network Server
for Autism Telerehabilitation System
Development

Muhammad Aliff Rosly, Mohd Azfar Miskam, Syamimi Shamsuddin,

Hanafiah Yussof and Nur Ismarrubie Zahari

Abstract The autism prevalence and rapid progress in technology lead to the
growing research of robot-mediated telerehabilitation system for autism therapy.
This solution aims to overcome the distance barrier between autism patients and
their therapist to serve as a practical telerehabilitation system. Meanwhile, the Inter-
net of Robotics Things (IoRT) has been identified as the suitable instrument for
connectivity of multiple robotic devices in development of physical telerehabilita-
tion system; capable of providing machine-to-machine (M2M) communication and
intelligent data processing in telerehabilitation environment. In related study, NAO
robot is employed as the “things” to facilitate remote communication and physical
interaction between autism patient and therapist thru IoRT network server. However,
the technical issue on two-ways communication between this humanoid robot and
IoRT network server has remains as the fundamental problem that prevents robot
actuators and sensors to be remotely control and manipulate by therapist. Hence,
viable data transfer between them cannot be tested. Therefore, a data linking testing
between NAO Robot and IoRT network servers such as Microsoft Azure and ThingS-
peak is conducted. The proposed data linking concept consists of data communication
through NAO Operating System (OS) NAOqi APIs, Python SDKs, and IoRT network
architectures. It enables two-ways communication between NAO robot and IoRT net-
works. The result shows that a set of data sent from NAO robot through Python SDK
can be visualized in graph form at ThingSpeak website and vice versa. The result
also supported by the back and forth data transfer between NAO robot dummy and

M. A. Rosly (B) · M. A. Miskam · H. Yussof

Faculty of Mechanical Engineering, Center for Humanoid Robots and Bio-sensing (HuRoBs),
Universiti Teknologi MARA, Shah Alam Campus, Selangor, Malaysia
e-mail: aliff.rosly@yahoo.com
M. A. Rosly
Faculty of Mechanical Engineering, Universiti Teknologi MARA, Bukit Besi Campus,
Terengganu, Malaysia
S. Shamsuddin
Faculty of Manufacturing Engineering, Universiti Teknikal Malaysia
Melaka, Melaka, Malaysia
N. I. Zahari
Faculty of Engineering, Universiti Putra Malaysia, Selangor, Malaysia

© Springer Nature Singapore Pte Ltd. 2020 161

A. P. P. Abdul Majeed et al. (eds.), RITA 2018, Lecture Notes
in Mechanical Engineering, https://doi.org/10.1007/978-981-13-8323-6_14
162 M. A. Rosly et al.

Microsoft Azure network. These results proved that the interchange data between
NAO OS NAOqi and IoRT network through Python SDK is feasible.

1 Introduction

Autism is a lifelong developmental disability that effects the brain’s normal develop-
ment of social and communication skills. The symptoms of autism typically appear
when the children reach 3 years of age, albeit some discover at later stage of life. Even
though the symptoms are varying widely across autistic children, its core features can
be listed into three major areas; social deficits, language impairment, and repetitive
behavior [1]. Autism, like other neurodevelopmental disabilities is generally incur-
able, but with proper treatments its deficits can be mitigated. The main objective of
autism treatments is to minimize its core features and associated deficits in autism
child while increasing their functional capacity to lead a better-quality life. Gener-
ally, treatments for autism encompasses of two main interventions; medication and
educational interventions [2]. Proven medications such as Aripiprazole and Risperi-
done are preferred by some for reducing the repetitive behaviour in children with
autism. But these medications come with sides effects such as drooling, tremors and
rapid weight gain [3, 4]. Hence it is not surprising others might opt for educational
intervention even though they consume a lot of resources. Educational intervention
for autism mostly cover a treatments combination which fostering of acquisition of
skills and knowledge to help autism child to develop their independence responsi-
bility. This includes one of the golden standard method named Applied Behaviour
Analysis (ABA) therapy, formulated by Ivar Lovaas at UCLA [5]. It is based on the
concept that when behaviour is rewarded, it is more likely to be repeated. To-date
this is the most established single ASD treatment that have recognized throughout
autism communities. This type of therapy emphasizes the importance of early and
intensive behavioural intervention to maximize its positive outcome [6, 7].
The autism prevalence and rapid progress in technology lead to the growing
research of robot-mediated telerehabilitation system for autism therapy [8–13]. This
latest solution aims to overcome the distance barrier, time and cost to serve as practi-
cal solutions in providing rehabilitation services to the autism population [14]. Most
of developed robot mediated telerehabilitation system requires patient to undergo
specific module based on robot behavior, then its generated result and analysis will
be displayed to the program facilitator (usually family member) or uploaded to the
cloud for therapist record. But, it doesn’t address the crucial two-way communica-
tion equipped with physical interaction between autism children and their respec-
tive occupational therapist. Consequently, the therapist unable to give their physical
remote feedback through robot even after the patients finished their rehabilitation
module online. To do so, the Internet of Robotics Things (IoRT) technologies has
been identified as the suitable instrument for connectivity of multiple robotic devices
in development of physical telerehabilitation system [15].
Data Linking Testing Between Humanoid Robot … 163

Our ultimate aim for this project is to develop a robot mediated telerehabilitation
system using IoRT for autism therapy. We are utilizing NAO robot as the “things”
to facilitate remote communication and physical interaction between autism patient
and therapist thru IoRT network server. However, the technical issue on two-ways
communication between this humanoid robot and IoRT network server has remains
as the fundamental problem that prevents robot actuators and sensors to be remotely
control and manipulate by therapist. Hence, viable data transfer between them cannot
be tested. Therefore, in this paper, a data linking testing between NAO Robot and
IoRT network servers such as Microsoft Azure and ThingSpeak is conducted.

2 Related Works in Autism Telerehabilitation Systems

In general, telerehabilitation simply refers to the delivery of rehabilitation services

using the means of telecommunication technologies [16]. Such services include
assessment, intervention, education and therapeutic interventions for disabled peo-
ple [17]. The first ever scientific publication about telerehabilitation that includes
patient treatment was found on 1998. The correlated research increased markedly
since then, driven by rapid development of communication technologies [18]. Ini-
tially, basic communication technologies such as textual, audio, and vision had been
used in various medical field of telerehabilitation [19, 20]. Followed by more advance
technologies such as virtual reality, web-based interface, haptic, robotics arm and
even a humanoid robot [21].
Based on several Human-Robot Interaction (HRI) studies, robot have been proved
to be a very effective therapeutic tool for autism due to several reasons [22]. Its
less complex features compare to human allow autism child to feel safe and less
intimidating. Thus, autism child will be more cooperative, and therapy can be done
more effectively [23]. Other than that, compared to visual or software therapeutic
agent, robot make tactile and physical interactions possible during therapy session.
Through appropriate interaction, it can facilitate elicitation, coaching and reinforce-
ment of social behaviour in autism therapy [24]. In fact, more recent research also
found neurobiological evidence to support robot usage in autism therapy. Chaminade
et al. highlights the facts that more brain activities were detected when autism child
interacts with humanoid robot compared to other therapeutic agents [25].
Aforementioned benefits of robot for the autism educational purpose have ignites
a great interest in increasing robot-mediated intervention coverage through telereha-
bilitation environment. The first effort was reported in 2010 by the researchers at the
Eindhoven University of Technology. Their project is aptly named ‘WikiTherapist’.
WikiTherapist is a platform where a community shares several robot-based training
scenarios for autistic children [8]. Next, another similar project is the Ask NAO ini-
tiative by Aldebaran Robotics. It is an interactive and educational implemented tool
to engage kids through customized robot applications for special education [9]. Ask
NAO claims that all robot applications are inspired by commonly practiced teach-
ing methods in autism (i.e. Applied Behavioural Analysis (ABA), Picture Exchange
164 M. A. Rosly et al.

Communication System (PECS)). In addition, the framework for categorizing 50

modules in Ask NAO into three main autism behaviours such as; Communication,
Social skills, and Repetitive behaviour, was proposed to improve the suitability of
module selection [10]. Later, Shamsuddin, et al. [11] initiated an ASD telerehabil-
itation on-line platform that can be accessed by parents and therapists at different
locations [11]. Through the website, therapists able to access and keep track on child’s
progress after undergone robotic module elsewhere. Author claim that, the modules
which were developed using NAO robot GUI: Choregraphe are based on special
curriculum provided by Malaysian Ministry of Education [12]. Furthermore, more
recent development in delivering robot-based services in telerehabilitation environ-
ment includes combination of robot element with other supportive technology such
as Microsoft Kinect Motion Sensor. Aided by this technology the imitation data
between autism child and NAO Robot able to be captured and stored in the cloud for
therapist access [13].
Most of developed robot mediated telerehabilitation system requires patient to
undergo specific module based on robot behavior, then its generated result and anal-
ysis will be displayed to the program facilitator (usually family member) or uploaded
to the cloud for therapist record But, it doesn’t address the crucial two-way com-
munication equipped with physical interaction between autism patients and their
respective therapist. Consequently, the therapist unable to give their physical remote
feedback through robot even after the patients finished their rehabilitation module
online. Moreover, it also has been proved that, in neurological rehabilitation, a type
of communication and interaction does play an important role in determining the
efficiency of the therapy [26]. Thus, the importance of two-way communication and
interaction between therapist and autism children surely cannot be neglected [27].
To do so, the Internet of Robotics Things (IoRT) technologies has been identified
as the suitable instrument for connectivity of multiple robotic devices in develop-
ment of physical telerehabilitation system; capable of providing machine-to-machine
(M2M) communication and intelligent data processing in telerehabilitation environ-
ment [15].

2.1 Robot Mediated Telerehabilitation System Using IoRT

for Autism Therapy

As a technological solution in providing physical telerehabilitation system to autism

communities, as shown in Fig. 1, a prototype of autism therapy’s telerehabilitation
system is proposed through a combination of IoRT technology and robot-mediated
intervention. The system able to potentially eliminate distance barrier and allow two-
way communication and physical interaction between patients and therapist. It con-
sists of robot-based intervention modules for autism therapy, IoRT cloud-based net-
work architectures, and therapist interface that allow humanoid robot as “things” to
communicate and interact with children with autism and transfer data over pervasive
Data Linking Testing Between Humanoid Robot … 165

Fig. 1 General architecture of robot mediated telerehabilitation system using IoRT for autism
therapy

networks using Internet protocols. The available robot-based intervention modules

for autism in telerehabilitation platform known as RoBIN (Robot-based Intervention
platform) is already planted in NAO Robot’s NAOqi framework and ready to be
used. Meanwhile, IoRT cloud based such as Microsoft Azure and ThingSpeak offers
promising solutions for connectivity of intelligent robotic devices in telerehabilita-
tion environment. Then, the therapist interface helps therapist to have comprehensive
access to therapy program conducted by parents at home.

3 Data Linking Testing Between Humanoid Robot

and IoRT Network Server

The main working principle of the proposed system is based on development of

connection between NAO Robot’s NAOqi framework and IoRT network. First of all,
to prove that viable data transfer can be done between them, a data linking testing
between NAO Robot and IoRT network servers such as ThingSpeak and Microsoft
Azure were conducted.
The data linking experimental setup includes data communication through NAO
Robot NAOqi APIs, Python SDK, and IoRT network architectures (Fig. 2). NAOqi
is the name of the main software that runs on the NAO robot and controls it. The
NAOqi can be used on Windows, Linux and MacOS. By using available API function
in NAOqi, programmer able to specify how to manipulate robot actuators and sensors.
While, Python SDKs provided by Softbank Robotic and open sources communities
allow python modules to be developed on each side and run on the robot and server
respectively.
For ThingSpeak server, we utilized HTTP (Hyper Text Transfer Protocol) as the
communication protocols with NAO robot. Guided by server Read/Write API keys,
166 M. A. Rosly et al.

Fig. 2 Data linking experimental setup between humanoid robot and IoRT network server

the communication through request-response protocols between NAO robot (client)

and server are visualized when Python module was initiated. For the first experi-
ment, a basic data transfer from NAO robot to ThingSpeak server was conducted by
requesting NAO robot to submit 5 ping data to be processed to the server. If the data
is successfully transferred, the server will response by visualize the data in graph. In
the same time, NAO robot voice will accompany each of the request and response
process.
While for the second experiment, NAO robot will be requesting textual data from
ThingSpeak server. Then the obtained data from server side will be display at client
side. Similar with the first experiment, the request-response process will be guided
by NAO robot voice. These experiments should indicate the feasibility of basic data
transfer between NAO robot and ThingSpeak when both sides Python SDKs (robot
and server) and HTTP protocol are involved.
On the other hand, MQTT (Message Queuing Telemetry Transport) is a publish-
subscribe messaging protocols that is more preferred when associating with IoT
related project due to its lightweight nature. For the sake of testing the feasibility of
Microsoft Azure server-side data transfer through MQTT protocols, a data linking
testing between Linux based controller, Raspberry Pi (as a NAO robot dummy) and
Microsoft Azure’s IoT Hub was conducted. The Python SDKs were involved at both
sides.

4 Result and Discussion

Basic data transfer from NAO robot to ThingSpeak server is being represented by
succeed request-response quantity over time. Figure 3 presents five succeed request-
responses from NAO robot to server over time. The graph interval time is currently
limited by ThinkSpeak’s free account message update interval limit. This graph
proves that, NAO robot to ThingSpeak server basic data transfer did happened.
Whereas, textual data transfer should prove the communication from server to
NAO robot had taken place. Figure 4, show the textual data like channel’s ID, Name
and from ThingSpeak server is being transfer to NAO robot (client side) through
Data Linking Testing Between Humanoid Robot … 167

Fig. 3 NAO to cloud (ThingSpeak) data transfer

Fig. 4 Cloud (ThingSpeak) to NAO data transfer

Python SDKs. These results proved that the interchange data between NAO Robot
and ThingSpeak when both sides Python SDKs (robot and server) and HTTP protocol
are involved is feasible.
Figure 5 shows the back and forth (publish and subscribe) data transfer between
NAO robot dummy (Linux based controller, Raspberry Pi) and Microsoft Azure’s
IoT hub network when MQTT protocol was utilized.

5 Conclusion

The results proved that the interchange data between NAO Robot and IoRT network
like ThingSpeak and Microsoft Azure using Python SDKs though HTTP and MQTT
protocols are feasible. Viable data transfer between them are successfully verified,
thus deeper research on finding suitable protocols and method to facilitate remote
168 M. A. Rosly et al.

Fig. 5 Cloud (Microsoft Azure) to NAO robot dummy (Linux based controller, Raspberry Pi) data
transfer

communication and physical interaction between autism patient and therapist in

robot mediated telerehabilitation system using IoRT for autism therapy can be safely
proceed. Furthermore, future work which includes study regarding larger data-size
transfer’s reliability when dealing with actual tele-rehabilitation robot-based module
will be focused.

Acknowledgements The authors gratefully acknowledge the Ministry of Education Malaysia

(MOE) for the fund received through the Niche Research Grant Scheme (NRGS), [Project file:
600-RMI/NRGS 5/3 (1/2013)], Prototype Research Grant Scheme (PRGS), [Project file: 600-
IRMI/PRGS 5/3 (4/2016)], Fundamental Research Grant Scheme (FRGS), [Project file: 600-
IRMI/FRGS 5/3 (97/2017)], COE of Humanoid Robot and Bio-Sensing (HuRoBs), Universiti
Teknologi MARA (UiTM), Universiti Teknikal Malaysia Melaka and Universiti Putra Malaysia
for their support.

References

1. (July 2) What is autism? Available: https://www.autismspeaks.org/what-autism

2. Myers SM, Johnson CP (2007) Management of children with autism spectrum disorders. Pedi-
atrics 120(5):1162–1182
3. Hirsch LE, Pringsheim T (2016) Aripiprazole for autism spectrum disorders (ASD). Cochrane
Database Syst Rev, 6
4. Sharma A, Shaw SR (2012) Efficacy of risperidone in managing maladaptive behaviors for
children with autistic spectrum disorder: a meta-analysis. J Pediatr Health Care 26(4):291–299
5. Lovaas OI (1987) Behavioral treatment and normal educational and intellectual functioning in
young autistic children. J Consult Clin Psychol 55:3–9
6. Rivard M, Terroux A, Mercier C (2014) Effectiveness of early behavioral intervention in public
and mainstream settings: the case of preschool-age children with autism spectrum disorders.
Res Autism Spectr Disord 8(9):1031–1043
7. Love JR, Carr JE, Almason SM, Petursdottir AI (2009) Early and intensive behavioral inter-
vention for autism: a survey of clinical practices. Res Autism Spectr Disord 3(2):421–428
8. Gillesen J, Boere S, Barakova E (2010) WikiTherapist. In: Presented at the proceedings of the
28th annual european conference on cognitive ergonomics, delft. Netherlands
 Data Linking Testing Between Humanoid Robot … 169

9. Yussof H, Salleh MH, Miskam MA, Shamsuddin S, Omar AR (2015) ASKNAO apps targeting
at social skills development for children with autism. In: 2015 IEEE international conference
on automation science and engineering (CASE), pp 973–978
10. Salleh MHK et al (2015) Experimental framework for the categorization of special education
programs of ASKNAO. Procedia Comput Sci, 76:480–487
11. Shamsuddin S, Yussof H, Mohamed S, Hanapiah FA, Ainudin HA (2015) Telerehabilitation
service with a robot for autism intervention. Procedia Comput Sci 76:349–354
12. Shamsuddin S, Malik NA, Yussof H, Mohamed S, Hanapiah FA, Yunus FW (2014) Telere-
habilitation in robotic assistive therapy for children with developmental disabilities. In: 2014
IEEE region 10 symposium, 2014, pp. 370–375
13. Abiddin WZBWZ, Jailani R, Hanapiah FA (2017) Real-time paediatric neurorehabilitation
system. In: TENCON 2017—2017 IEEE region 10 conference, 2017, pp 1463–1468
14. Laut J, Porfiri M Raghavan PC (2016) The present and future of robotic technology in rehabil-
itation. Phys Med Rehabil Rep 4(4):312–319
15. Simoens P, Dragone M, Saffiotti A (2018) The internet of robotic things: a review of concept,
added value and applications. Int J Adv Robot Syst 15(1):1729881418759424
16. McCue M, Fairman A, Pramuka M (2010) Enhancing quality of life through telerehabilitation
(in eng). Phys Med Rehabil Clin N Am 21(1):195–205
17. Brennan D et al (2010) A blueprint for telerehabilitation guidelines, 2010 p 4
18. Rogante M, Grigioni M, Cordella D, Giacomozzi C (2010) Ten years of telerehabilita-
tion: a literature overview of technologies and clinical applications. J NeuroRehabilitation
27(4):287–304
19. Gibbs V, Toth-Cohen S (2011) Family-centered occupational therapy and telerehabilitation for
children with autism spectrum disorders. Occup Ther Health Care 25(4):298–314
20. Boisvert M, Hall N, Andrianopoulos M, Chaclas J (2012) The multi-faceted implementation
of telepractice to service individuals with autism. Int J Telerehabilitation 4(2):11–24
21. Benham S, Gibbs V, Exploration of the effects of telerehabilitation in a school-based setting
for at-risk youth. Int J Telerehabilitation 9(1): Spring 2017
22. Shamsuddin S, Yussof H, Ismail LI, Mohamed S, Hanapiah FA, Zahari NI (2012) Initial
response in HRI- a case study on evaluation of child with autism spectrum disorders interacting
with a humanoid robot NAO. Procedia Eng 41:1448–1455
23. Cabibihan J-J, Javed H, Ang M, Aljunied SM (2013) Why robots? a survey on the roles
and benefits of social robots in the therapy of children with autism. Int J Soc Robot, J Artic
5(4):593–618
24. Scassellati B, Admoni H, Matarić M (2012) Robots for use in autism research. Annu Rev
Biomed Eng 14:275–294
25. Chaminade T, Fonseca DD, Rosset D, Lutcher E, Cheng G, Deruelle C (2012) FMRI study
of young adults with autism interacting with a humanoid robot. In 2012 IEEE RO-MAN: the
21st IEEE international symposium on robot and human interactive communication, 2012, pp
380–385
26. Shelton C, Shryock M (2007) Effectiveness of communication/interaction strategies
with patients who have neurological injuries in a rehabilitation setting. Brain Injury
21(12):1259–1266
27. Peretti A, Amenta F, Tayebati SK, Nittari G, Mahdi SS (2017) Telerehabilitation: review of the
state-of-the-art and areas of application. JMIR Rehabil Assist Technol 4(2):e7

View publication stats