Report Assignment # 5

Name: Jamal Shah

ID: 2312021020
1. Explain how does the method achieve the intrinsic
motivation. Describe them by the formula if the authors
formalize them. Use figures if necessary. Try to describe in
precise manner,
My Selected Paper

• Schembri, M., et al. (2007). Evolving

internal reinforcers for an intrinsically
motivated reinforcement-learning robot. 2007 IEEE 6th International
Conference on Development and Learning, IEEE
What is intrinsic motivation?

• Basically, Intrinsic motivation refers babies/robots inspired by a

variety of behaviors and skills that begin to develop in infancy and
early childhood, including diverse phenomena such as curiosity,
surprise, novelty seeking, and the “drive” to achieve mastery.
• In my selected paper, the robot’s life is divided into two phases:
“childhood” and “adulthood”. During childhood the robot moves
around and learns a set of basic sensory-motor skills based on its
intrinsic motivational system by either the evolved reinforcers and or
by experts’ surprise, that is, the error in the prediction of future
rewards made by experts’ critics.
During adulthood, the robot learns to combine the acquired skills in
order to accomplish different tasks. Each task consists of a series of
time steps during which the robot has to reach a given target location
starting from a particular position. During each task, when the robot
reaches the target, it receives a reward and is placed back at the
starting position (if the robot hits the wall it turns of a random angle).
The model was tested in several different environments with different
floor textures and with several different tasks.
Oudeyer and Kaplan（2007）propose a framework for organizing
research on models of intrinsic motivation, including two major
categories：
（1）knowledge-based approaches which are subdivided into
novelty-based and prediction-based approaches.
（2）competence-based approaches. Within this framework a large
number of algorithms can be defined and systematically compared
According mentioned 2nd category the method in my selected paper
achieved knowledge-based approached intrinsic motivation.
2. What did the mechanism(s) proposed in your selected paper, achieve as
intrinsic motivation, and what did not. Explain them, for example, by
comparing with the models/theories in developmental psychology.
What achieve as intrinsic motivation what
not?
1. The controller of the robot (Fig. 2) is a hierarchical modular neural
network. The system is formed by a selector and a number of experts,
with each expert including also an internal reinforcer and an actor to
compute the surprise of the expert in the standard way.
What achieve as intrinsic motivation what
not?
• During childhood, learns on the basis of its reinforcer.

• The selector’s critic, which is a 2-layer neural network like the

experts’ critic. During childhood the reinforcement signals used by
the selector are intrinsic, being formed by the surprise of the expert
which has control on action, whereas during adulthood
reinforcements are extrinsic, coming directly from the environment
which is not completely goes with the intrinsic motivation categories.
What achieve as intrinsic motivation what
not?
• The result of this developmental process is that at the end of
childhood the robot has acquired a set of basic skills (sensory-motor
mappings) which can be used for solving the particular tasks
encountered during adulthood.
This is illustrated in Fig. 3c-h, in which the behavior of the adult robot
at the end of each task’s learning phase is shown.

As clearly shown by the graphs, whenever the robot is on one color

trail the selector selects the expert which is able to follow that color
(apart from rare cases due to the stochastic nature of selection).
But, during adulthood reinforcements are extrinsic, coming directly
from the environment which I think not exactly goes with intrinsic
method.

END