2014 IEEE International Symposium on Robotics and Manufacturing Automation

Programming Platform for NAO Robot in Cognitive

Interaction Applications
l 12 l l l
Mohd Azfar Miskam, , Syamimi Shamsuddin, Hanafiah Yussof, Abdul Rahman Omar, Muhammad Zaiyad Muda
lCOE for Humanoid Robots & Bio-sensing (HuRoBs), Faculty of Mechanical Engineering,
Universiti Teknologi MARA, 40450 Shah Alam, Selangor, Malaysia
2Faculty of Manufacturing Engineering, Universiti Teknikal Malaysia Melaka,
76100 Durian Tunggal,Melaka, Malaysia
Email: mohdazfar240l@gmail.com

Ahstract- The humanoid robot NAO is a ready-made

exploration platform that comes with a Choregraphe software to
program and animate it. Choregraphe is a powerful tool that
allows programming in Python for motion control. In this paper,
we explore how Cboregraphe can aid users in building basic
cognitive interaction modules specifically for the rehabilitation of
autistic children. Difficulty to interact with others and impaired
social cognition are some of the core symptoms of autism. A robot
in human form has the ability to help encourage children to
interact and respond to dialogue cues. Comparison between
Choregraphe and another programing tool in embedding
cognitive interaction ability into the NAO robot is discussed.

Keywords-Choregraphe; humanoid rohot; cognitive

interaction; Autism Spectrum Disorder (ASD)

I. INTRODUCTION

A humanoid is simply a robot whose physical features

resemble a human. Humanoid robot NAO is a commonly used
tool in education and autism rehabilitation purposes. This biped
robot was developed by the French company Aldebaran­
Robotics with the notion to provide an open and accessible,
high-performance robot onto the market as an affordable
exploration platform for both researchers and the general
public. It is a completely custom designed robot as the whole
process of design and manufacturing is mastered in mechanics,
electronics and software.
Looking at the robot's attributes, NAO is 58 centimeters in
height with 5 kilograms of weight (Fig. 1).This robot has 25
degree of freedom (DOF) with position sensors and
temperature sensors in each motor. NAO is also embedded
with three tactile sensors on the head, three tactile sensors on
each hand and one bumper sensor in each foot that can be used
in suitable situations. The force sensor is located under its feet.
On the head of the robot, there are four microphones that
can be used for two-way communication between robot and
human. Two cameras located on the forehead and mouth helps
Nao to recognize human faces and objects. For connectivity,
NAO connects with computer using Ethernet cable or Wi-Fi
connection.
Fig. 1. The humanoid robot NAO
Humanoid robot NAO has been used in the autism therapy
research due to the benefit where there may be a greater
potential for generalization. For example, only through a
human form can children engage in imitation and emotion
recognition activities. Children with autism often become
withdrawn and avoid interactions when they see something
with a human form. Robots in the forms of animals, cars and
toys often do not trigger these same reactions, which can make
them more engaging than robots with a humanoid form. In
addition to helping the children engage in activities,
non-humanoid robots can often be much simpler and
affordable.
In this paper, we will explain about how Choregraphe can
help users to build basic cognitive modules using the robot.
This paper also covers the external software that can be used
for the same purpose. Choregraphe is graphical-based software
that comes together with NAO. It is important to develop
interaction dialogues to be realistic and as close as possible to a
typical conversation. Furthermore, the sentences need to be
kept short to minimize confusion for autistic children.

978-1-4799-5765-1/14/$31.00 ©2014 IEEE 141

 2014 IEEE International Symposium on Robotics and Manufacturing Automation

II. CHOREGRAPHE
,,4) . .,. .... ..

Choregraphe (Fig. 2) is a user friendly GUI that can control

the robot, create behaviours and access the data acquired by the
sensors. Choregraphe practically makes programming looks
simple and easy to understand for beginner users. It also offers
function that able to build cognitive in interaction using the
humanoid robot NAO. Interaction activities include two-ways ';1 • ',1"0"
I �: � .;;�;� 'n
communication, face recognition, object tracking, dancing, "

: I 1L' hd_
story-telling and many more. : : iI " ., " "'"

I r i... �i.C ...

."

Choregraphe is a powerful tool with intuitive graphical

environment that allows from simple to complicated
programming. This software is compatible with three
Fig. 2. The GUI of Choregraphe
Operating System platform which are; Windows platform, Mac
OS X and Ubuntu Linux. The application window mainly
focused on three main areas; box library that consist the pre­ D.
program behaviors of the robot such as stand-up and sit down; " 11�.a

working area that use for drop the box behavior from library
and create algorithm; graphical representation of NAO able to
execute the implemented behavior and able to connect with
real robot in real-time motion.
When several behavior boxes are chained from the input to
the output box, the sequence of the robot motion and
behaviours will be complete. The program can be run using a
real robot or can be simulated using third-party simulation
software Webot (Fig. 3). Webot allows user to create virtual
environment for NAO and execute the behaviour developed in
Choregraphe before tested in real NAO robot. Virtual NAO in
Webot can be connected with Choregraphe using given IP
address by Webot.
Fig. 3. Virtual NAO Robot in Webot Application
The new version of Choregraphe 2.0 has a function of
dialog to create the two-way communication more interaction
and help to simplify the programming. This dialog function
(Fig. 4) comes with function to create concept of several words
of same meaning into one sentence. By having concept, NAO
robot can have a smooth conversation with human especially I fIy��
1 t:0I'1C'
IqIoX "�.(II;CIX.
'1\'_:' l.a.l':"L t qH�C I

children with autism during intervention program. l�l!o: e:TIII

In the next section, the application of NAO in rehabilitation

of children with autism is discussed. The intervention program
needs a robot with simple appearance, engaging to the children,
able to speak, move and considerably intelligent to respond to
the children.

o () ,·1
""'"

III. APPLICATION OF NAO ROBOT IN THE

REHABILITATION OF AUTISTIC CHILDREN

Since the early stages of study of autism rehabilitation, our

research group at the Center for Humanoid Robots & Bio­
sensing (HuRoBs) had relied on Choregraphe as a tool to Fig. 4. Dialog Function in Choregraphe 2.0
program the robot. Currently the on-going research work at
The crucial part in rehabilitation is two-way
HuRoBs focuses on three application areas to help children
communication where the robot can interact with the autistic
with autism: a) to program the humanoid robot for two-way
children and hold the conversation compared to interaction
communication, b) to develop simple humanoid body gestures
between human-to-human interactions. By possesses of
that represents emotions and c) to improve object recognition
microphones on this robot, it helping our research to achieve
intelligence to help children recognize numbers.
this target.

142
 2014 IEEE International Symposium on Robotics and Manufacturing Automation
Autistic children have problem in communication, both
verbal and non-verbal since they do not develop enough natural
speech to meet their daily communication needs. Nearly all
children with autism exhibit some level of language
impairment, which can range from a complete lack of verbal
communication, to prolific talkers who talk incessantly about
favorite topics without allowing communication partners to add
to the conversation. They often remain aloof from others, and
fail to follow many social norms. They also might avoid eye
contact while talking with others and does not have the ability
to use simple movement to express themselves, such as
deficiency to point at things or even a simple gesture.
Children with autism also have problem in social
interaction such as sharing emotions, understanding how others
think and feel, and holding a conversation as well time spend
by a person to interact with others. They also often fail to
recognize the emotional states of others and do not understand
that their own actions affect what others do and how they feel.
To make the conversation more interactive and interesting,
the robot has been programmed with emotion gestures where it
shows to the child to express feeling using body language.
There are several types of emotions can be portrayed using
body of NAO, such as sad, happy and many more combine
with LEDs colors on the eyes.
A. Two-ways Communication
We have conducted experiment of two-way communication
with autistic children with the aim to see the respond of child if
the NAO robot makes a conversation with them. The
experiment was done in a classroom that consists of table for
robot, chair for children and chair for the guardian as the
comforting agent to the children and two surveillance cameras
for our data collection (Fig. 5).
The module of interaction start with the humanoid robot
Nao saying "Hello" while waving hand. Then, the robot will
ask permission to sit, the child need to answer YES or NO and
the robot will react according to the answer. The interactions
continued with NAO introducing itself and ask for the child's
name. The conversation between NAO robot and the autistic
children continues for several more questions; 'Are you happy
today?', 'What is your favorite color?', 'Do you like going to
school?' and 'Why do you like/dislike going to school?'.
The interaction between robot and autistic children need to
be simple but attractive. It is because we need to minimize the
initial surprise felt by the children when meeting the robot for
the first time, to promote enjoyable two-way communication
and also to help create a friendly environment during
interaction.
This module fully uses the capability of the robot on
recognized voice of the child during interaction. The
experiment of two-ways communication works as expected
where the children give their attention and answer the
questions. However, some precautions need to be taken as
there is must no echo built in the room because it will disturb
NAO to analyze the pronunciation
143
B. Emotion Gestures by NAO
NAO robot does not have facial expression as human and it
makes the face is plain and no emotions. However, the
emotions can be portrayed using combination of body
language, sound produce and colours LEDs on the eyes (Fig.
6). This function can be added into module of rehabilitation of
autistic children in learning to express the feeling in a certain
situation.
The movement of body languages is preprogramming using
Choregraphe and combines with different LEDs colours and
sound for each emotion. The patterns are mimic from human
body language in real life.
In our future work for this, we would like the robot to
autonomously react and do the emotion gesture according to
the voice emotions as example, when people shout out loud,
the robot will show the frightened emotion
C. Object or Pattern Recognition
To make the robot able to recognize patterns, first the robot
needs to be taught pattern recognition. The teaching method is
using the vision behavior in the box library inside of
Choregraphe. This method can be used for rehabilitation
module as to see if the is child able to recognize and write
numbers. There are important factors that need to be
considered in the setup such as, the lighting in the room which
will affect during the 'Draw Number Game' because it involves
the robot's vision system that are sensitive to lighting quality.
Improvement of the modules will be based on the results from
a pilot study with a typical child.
The robot will ask the children to play the game with it. In
this game, the robot will say a random integer range from 1 to
10 and the candidate will write the number on a paper and
show them to NAO. NAO will capture the number, detect and
recognize it (Fig. 7) using internal camera inside of the head.
The distance between written paper and NAO's camera must be
in range of 15cm to 30cm
Interaction Area Operator Area
�C=l
TeacherlGu.;u"Can
fable
D !,,�
!
.. J
*
ChId C:amI!f'!l1
1;
,
,
/
Fig.5. Illustration of Experimental Setup
 2014 IEEE International Symposium on Robotics and Manufacturing Automation
Fig. 6. Various postures of NAO robot when displaying
nine different emotions: angry, disgusting, happy, hungry,
loving, sad, scared, shy and tired
Fig. 7. (a) Hand written number in plain paper using marker
pen, (b) Paper shows in front of NAO's Camera, (c) The
number that captured by the camera, (d) Blue mark appear
when it detected the pattern.
144
IV. OTHER PLATFORM TO PROGRAM NAO ROBOT
Some research labs have decided not to use Choregraphe as
a programming tool as it has limitations in a certain aspects.
There are many choices of programming platform can be made
that already exist and compatible with humanoid robot Nao. It
all depends on the understanding of programing languages and
research need to be done. Table 1 shows the summary of
differentiation between each platform
A. Python
Python is a widely used general-purpose, high level
programming language. Its design philosophy emphasizes code
readability, and its syntax allows programmers to express
concepts in fewer lines of code than would be possible in
languages as C. This language is already used in Choregraphe,
however NAO also can be control using Python alone.
Python supports multiple programming paradigms,
including object-oriented, imperative and functional
programming or procedural styles. It features a dynamic type
system and automatic memory management and has a large
and comprehensive standard library.
B. Urbi
Urbi is an open-source software platform to control robots
or complex systems in general. It includes a C++ component
library called UObject that comes with a robot standard API to
describe motors, sensors and algorithms. Next to UObject, you
can use the urbiscript orchestration script language to glue the
components together and describe high level behaviors, a bit
like python or LUA, but with embedded parallel and event­
driven semantics to make the job easier.
The goal of Urbi is to help making robots compatible, and
simplify the process of writing programs and behaviors for
those robots. The range of potential applications of Urbi goes
beyond robotics, since it has been successfully used in generic
complex systems, where parallel and event-driven
orchestration on multiple agents is the rule.
Urbi comes with Gostai Studio that integrates a behavior
editor to graphically create finite state machines whose states
are simple urbiscript programs, and GUI builder to design
interfaces with drag and drop support. Gostai Studio is cross­
platform and generates freely usable urbiscript code as an
output. Choregraphe is a platform for NAO using the Python
and urbiscript languages.
C. Robot-Operating System (ROS)
ROS is an open-source, meta-operating system for your
robot. It provides the services you would expect from an
operating system, including hardware abstraction, low-level
device control, implementation of commonly-used
functionality, message-passing between processes, and package
management. It also provides tools and libraries for obtaining,
building, writing, and running code across multiple computers.
The primary goal of ROS is to support code reuse in
robotics research and development. ROS is a distributed
 2014 IEEE International Symposium on Robotics and Manufacturing Automation
framework of processes (aka Nodes) that enables executable to
be individually designed and loosely coupled at runtime. These
processes can be grouped into Packages and Stacks, which can
be easily shared and distributed. ROS also supports a federated
system of code Repositories that enable collaboration to be
distributed as well. This design, from the file system level to
the community level, enables independent decisions about
development and implementation, but all can be brought
together with ROS infrastructure tools.
Language independence in the ROS framework is easy to
implement in any modem programming language. ROS
already implemented it in Python, C++, and Lisp, and also
experimental libraries in Java and Lua.
ROS currently only runs on Unix-based platforms.
Software for ROS is primarily tested on Ubuntu and Mac OS X
systems, though the ROS community has been contributing
support for Fedora, Gentoo, Arch Linux and other Linux
platforms.
D. C++programming language
C++ is one of the most popular programming languages
and is implemented on a wide variety of hardware and
operating system platforms. As an efficient performance
driven programming language it is used in systems software,
application software, and device drivers, embedded software,
high-performance server and client applications, and
entertainment software such as video games. Various entities
provide both open source and proprietary C++ compiler
software, including the FSF, LLVM, Microsoft and Intel.
The C++ standard consists of two parts: the core language
and the C++ Standard Library; which C++ programmers
expect on every major implementation of C++, it includes
vectors, lists, maps, algorithms (find, foceach, binary_search,
random_shuffle, etc.), sets, queues, stacks, arrays, tuples,
input/output facilities (iostream; reading from the console
input, reading/writing from files), smart pointers for automatic
memory management, regular expression support, multi­
threading library, atomics support (allowing a variable to be
read or written to be at most one thread at a time without any
external synchronisation), time utilities (measurement, getting
current time, etc.), a system for converting error reporting that
doesn't use C++ exceptions into C++ exceptions, a random
number generator and a slightly modified version of the C
standard library (to make it comply with the C++ type system)
E. Java programming language
Java is a computer programming language that is
concurrent, class-based, object-oriented, and specifically
designed to have as few implementation dependencies as
possible. It is intended to let application developers "write
once, run anywhere" (WORA), meaning that code that runs on
one platform does not need to be recompiled to run on
145
another. Java applications are typically compiled to bytecode
(class file) that can run on any Java virtual machine (JVM)
regardless of computer architecture. Java is, as of 2014, one of
the most popular programming languages in use, particularly
for client-server web applications, with a reported 9 million
developers. The language derives much of its syntax from C
and C++, but it has fewer low-level facilities than either of
them.
One characteristic of Java is portability, which means that
computer programs written in the Java language must run
similarly on any hardware/operating-system platform. This is
achieved by compiling the Java language code to an
intermediate representation called Java bytecode, instead of
directly to platform-specific machine code. Java bytecode
instructions are analogous to machine code, but they are
intended to be interpreted by a virtual machine (VM) written
specifically for the host hardware. End-users commonly use a
Java Runtime Environment (JRE) installed on their own
machine for standalone Java applications, or in a Web browser
for Java applets.
A major benefit of using bytecode is porting. However, the
overhead of interpretation means that interpreted programs
almost always run more slowly than programs compiled to
native executables would. Just-in-Time (JIT) compilers were
introduced from early stages that compile bytecodes to
machine code during runtime.
V. CONCLUSION
In this paper we have already shown the possibilities of
using Choregraphe as a tool to build the cognitive interaction
for humanoid robot Nao. The team is working on the
possibilities of using various type of platform to build
rehabilitation applications for children with autism. Currently
only the two-way communication module has been used in an
actual study with autistic children.
The advantage of using Choregraphe compared to other
programming platform that can be used for NAO is the
friendly user to beginner and non-technical user. Especially
therapist with less technical skill to develop modules for
rehabilitation of children with autism.
ACKNOWLEDGMENT
Due acknowledgement is accorded to the Ministry of
Education (MOE), Malaysia and Universiti Teknologi MARA
(UiTM) for funding the research project through the Niche
Research Grant Scheme (NRGS), [Project #: 600-RMI/NRGS
5/3 (2/2013)], Fundamental Research Grant Scheme (FRGS)
[600-RMIIFRGS 5/3 (77/2013)] and Exploratory Grant
Scheme (ERGS) [600-RMIlERGS 5/3 (14/2013)].
 2014 IEEE International Symposium on Robotics and Manufacturing Automation

REFERENCES

[1] Mohd Azfar Miskam, Mohd Aminullah Che Hamid, Hanafiah Yussof,
Syamimi Shamsuddin, Siti Nora Basir, Norjasween Abdul Malik, "Study [7] D. Ricks and M. Colton, "Trends and considerations in robot-assisted
on Social Interaction between Children with Autism and Humanoid autism therapy," 2010 IEEE International Conference on Robotics and
Robot NAO'" Journal of Applied Mechanics and Materials (2013), Vol. Automatioo, Aochorage, AK, 3-8 May 2010, inpress.
393, pp 573-578. [8] Filipek, et al. (2000). "Practice Parameter: Screening and Diagoosis of
[2] Mohd Aminullah Che Hamid, Mohd Azfar Miskam, -.fish Yusso Autism", Report of the Quality Staudarda Subcommittee of the
Syamimi Shamsudctin, Hafizan Hashim, Luthffi Jsmai "Human-Robot American Academy of Neurology and the Child Neurology Society.
Interaction (HRI) for Children with Autism to Augment Communication Neurology, 55:468-479.
Skills", Journal of Applied Mechanics and Materials (2013), Vol. 393, [9] Sabine M Klaw:k (2006), "Geuetics of Autism Spectrum
598-603
Disorder", European Journal of Human Genetics.
[3] Peter Smolar, Jaroslov Tuharsky, Zlatk:o Fedor, Maria Vircikova, Peter
[10] K. Dautenhahn, "Design issues on interactive environments for children
Sincak,"Development of cognitive capabilities for robot Nao in Center
with autism." in In: Procs of ICDVRAT 2000, the 3rd Int Conf on
for Intelligent Technologies in Kosice", Center for Intelligent Disability, Virtual Reality and Associated Technologies, 2000.
Technologies, Technical University of Kosice, Slovak Republic.
[11] S. Shamsuddin, H. Yua M. A. Miskam, A. C. Hamid, N. A. Malik,
[4] David Gouailler, Vincent Hugel, Pierre Blazevic,"Mechatronic design of
and H. Hashim, "Humanoid Robot NAO as HRI Mediator to Teach
NAO humanoid", IEEE International Conference on Robotics and
Emotions using Game-centered Approach for Children with Autism,"
Automation, Kobe International Conference Center, Kobe, Japan, May
HRI 2013 Workshop 00 Applicatiooa for Emotional Robots,
2009.
Tokyo, Japao, 2013.
[5] Syamimi Shemsuddin, Luthffi Jdzhar Jsmai -.fiah yuasof, Nur Ismarubie
[12] M.A. Miskam, N. F. S. Masoin, M. H. Jarnhuri, S. Shemsuddin, A. R.
Zahari, Saiful Bahari, Hafizan Hashim, Ahmed Jaafilr,"Humanoid Robot
Omar, and H. Yussof, "Encouraging Children with Autism to Improve
NAO: Review of Cootrol and Motioo Exploratioo", 2011 IEEE
Social and Communication Skills through the Game-Based Approach,"
International Cooference 00 Cootrol Syatem, Computing and
Proceeding of International Conference on Robot PRIDE 2013-2014 -
Engineering.
Medical and Rehabilitatioo Robotics and Instrumentatioo, ConfPRIDE
[6] E. Pot, J.Monceaux, R.Gelin, B.Maisonnier, Aldebaran 2013-2014 2013.
Robotics,"Choregraphe: a Graphical Tool for Humanoid Robot
Programming", The 18th IEEE International Symposium on Robot and
Human Interactive Communication Toyama, Japan, Sept 2009.

146