SCIENCE CHINA

Technological Sciences
Progress of Projects Supported by NSFC November 2013 Vol.56 No.11: 2745–2761
doi: 10.1007/s11431-013-5369-0

The skinner automaton: A psychological model formalizing

the theory of operant conditioning
RUAN XiaoGang & WU Xuan*
Institute of Artificial Intelligence and Robots, School of Electronic Information and Control Engineering,
Beijing University of Technology, Beijing 100124, China

Received August 6, 2013; accepted September 13, 2013; published online September 30, 2013

Operant conditioning is one of the fundamental mechanisms of animal learning, which suggests that the behavior of all animals,
from protists to humans, is guided by its consequences. We present a new stochastic learning automaton called a Skinner au-
tomaton that is a psychological model for formalizing the theory of operant conditioning. We identify animal operant learning
with a thermodynamic process, and derive a so-called Skinner algorithm from Monte Carlo method as well as Metropolis algo-
rithm and simulated annealing. Under certain conditions, we prove that the Skinner automaton is expedient, -optimal, optimal,
and that the operant probabilities converge to the set of stable roots with probability of 1. The Skinner automaton enables ma-
chines to autonomously learn in an animal-like way.

Learning automata, Boltzmann distribution, operant conditioning, operant learning, simulated annealing

Citation: Ruan X G, Wu X. The skinner automaton: A psychological model formalizing the theory of operant conditioning. Sci China Tech Sci, 2013, 56:
27452761, doi: 10.1007/s11431-013-5369-0

1 Introduction and involuntary in the process of Pavlovian conditioning.

He believed that animals engaged with their environments
actively and voluntarily [1]. Skinner termed an active be-
Operant conditioning is one of the fundamental concepts in
havior as an operant [2], from which operant conditioning
behavioral psychology, and sometimes it is called the Skin-
took its name.
nerian conditioning due to Skinner’s pioneering work [1, 2].
Operant conditioning is an important form of psycholog-
Earlier work dates from American psychologist Thorndike
ical learning and sometimes called operant learning, during
who investigated trial-and-error learning in cats, dogs,
which humans and animals learn to associate their behaviors
chicks, and monkeys. He developed his famous Law of Ef-
with the consequences. Operant learning inspires the study
fect [3] that formed the conceptual starting point for Skin-
of machine learning. There are some computational models
ner’s work. Watson, the founder of behaviorism [4], argued
trying to simulate operant conditioning. The earlier work
that behavior could be researched scientifically without re-
was contributed by Grossberg [7, 8] who proposed an ide-
course to inner mental states [5]. Skinner was a radical be-
alized neural network modelling operant conditioning for
haviorist and much influenced by Watson. However, Skin-
explaining behavioral data on learning in vertebrates.
ner thought that there existed serious shortcomings in Wat-
Grossberg’s model was later used for robots to learn the
son’s theory that was totally based on Pavlovian condition-
operant behavior of obstacle avoidance by Chang and
ing [6]. In Skinner’s opinion, organisms were too passive
Gaudiano [9]. Touretzky and his team studied the operant
conditioning model for making the physical robot Skin-
*Corresponding author (email: abc0767@live.com) nerbot trainable [10–12]. Itoh et al. considered that robots

© Science China Press and Springer-Verlag Berlin Heidelberg 2013 tech.scichina.com www.springerlink.com
2746 Ruan X G, et al. Sci China Tech Sci
need to express emotions, behaviors and personality in a
human-like manner. They presented a behavior model with
operant conditioning and applied it to an emotion expres-
sion humanoid robot (WE-4RII) so that the machine could
autonomously select suitable behavior [13, 14].
Reinforcement learning is an important form of machine
learning, which is inspired by behaviorist psychology. More
exactly, operant conditioning theory underlies reinforce-
ment learning. In fact, the concept of reinforcement origi-
nates from the theory of operant conditioning in which re-
inforcement plays a key role. Sutton and Barto established
the basic principles of reinforcement learning [15]. Howev-
er, reinforcement learning can be traced back to the work on
learning automata (LA). The term “learning automata” was
first used by Narendra and Thathachar [16], who pointed
out that the first learning automata were developed in
mathematical psychology. The original notion of learning
automata corresponds to the so-called finite action-set
learning automata (FALA) [16]. An FALA has a given set
of actions and a specific reinforcement scheme, which gen-
erally operates in an unknown random environment and
updates its action probabilities in accordance with the re-
sponses from the environment. It is obvious that learning
automata are also inspired by behaviorist psychology.
Learning automata are the models of operant learning as
well as the formal frameworks for reinforcement learning.
The notion of learning automata has been extended over
the years [17]. FALA ensure convergence only to a local
maximum of the reinforcement signal [18, 19]. To get con-
vergence to global maximum, parameterized learning au-
tomata (PAL) were proposed [20]. To handle associative
reinforcement learning problems, generalized learning au-
tomata (GLA) were introduced [21, 22]. The common
theme among them is reinforcement schemes that serve as
the basis of the learning process for learning automata
[23–27]. According to their linearity, reinforcement
schemes can be categorized into linear, nonlinear, and hy-
brid schemes. A reinforcement scheme in a learning autom-
aton is the crucial factor that affects the performance of the
learning automaton. According to the properties exhibited
by learning automata using the schemes, reinforcement
schemes can be classified as expedient, -optimal, optimal,
and absolutely expedient.
As has been pointed out by Touretzky and Saksida [10],
however, animals learn much more complicated behaviors
through operant conditioning than machines acquiring
through reinforcement learning or learning automata. In this
paper, we address a so-called Skinner automaton as a new
psychological model for formalizing the theory of operant
conditioning. We identify animal operant learning with a
thermodynamic process, and derive the so-called Skinner
algorithm from Metropolis algorithm [28, 29] and simulated
annealing [30, 31]. The Skinner algorithm is the core of the
Skinner automaton and serves as its reinforcement scheme.
With simulated annealing, the Skinner automaton ensures
November (2013) Vol.56 No.11
convergence to global optimization. Our theoretical analysis
shows that the Skinner automaton governed by the Skinner
algorithm is not only expedient and -optimal, but also op-
timal and absolutely expedient. At the end of this paper,
some optimal computing and simulated experiments are
presented with the Skinner automaton to test the perfor-
mance of the Skinner automaton. Our work suggests that
operant conditioning not only is psychological and biologi-
cal, but also can be computational, psychodynamical and
thermodynamical. The Skinner automaton enables autono-
mous agents including robots to autonomously learn in an
animal- like way.
The rest of this paper is organized as follows. Section 2
unscrambles operant conditioning and extracts its funda-
mental elements. Section 3 configurates the Skinner autom-
aton for synthesizing operant conditioning and describes the
Skinner algorithm that is the core of the Skinner automaton.
Section 4 discusses the self-organizing property of the
Skinner automaton. Sections 5 and 6 reproduce famous
Skinner’s experiments of pigeons with the Skinner automa-
ton. In Section 7, an animated 3D Wienerian worm is built
and the Skinner automaton serves as its photosensory-motor
system to shape its behavior of negative phototropism by
operant learning (i.e. learning in operant conditioning way).
In Section 8, the Skinner automaton serves as the sen-
sorimotor system related posture and movement of a physi-
cal flexible two-wheeled robot, and enables the robot to
gradually develop its motor skills and to learn to balance in
autonomous operant conditioning. Finally, our conclusions
are given in Section 9.
2 Analysis of operant conditioning
A natural question about machine learning is whether it is
compatible with animal learning. We aim at formalizing the
theory of operant conditioning (OC) in a form analogous to
learning automata (LA) so that artificial systems learn in an
operant conditioning way like animals. From psychodynam-
ics [32], operant conditioning is a dynamical process. A
learning automaton is a dynamical system. In this section, we
try to unscramble the dynamical process of OC in terms of
LA so that our OC model is compatible with OC in animals.
2.1 Learning automata
Here the LA theory is briefly described to provide a back-
ground for understanding the learning process of LA and for
being compared and contrasted with OC.
Generally, a learning automaton (LA) is a stochastic dy-
namical system operating in a random environment, and can
be represented with a 7-tuple:
LA  t , S ,  ,  , F , G, A , (1)
where t{0, 1, 2, ⋯} represents the discrete time for each
 Ruan X G, et al. Sci China Tech Sci
stage in the learning process of the LA, S a set of internal
states,  a set of actions that are the outputs of the LA,  a
set of responses that are the inputs of the LA from the envi-
ronment, F: SS a function that maps the current state
and input into the next state, which can be deterministic or
stochastic, G: S (p) a function that maps the current
state into the current output (an action in ) with the occur-
rence probability p of the action , and A: pp an
algorithm (also called an updating scheme or reinforcement
scheme) that maps the current action probabilities, action
and input into the next (new) action probabilities.
The dynamical process of the LA is depicted in Figure 1,
showing that the interaction of the LA and its environment
conduce to a closed-loop feedback connection. Action (t),
the output of the LA at stage t, operates on the environment
and induces it to give the response (t) that is fed back to
the LA and serves as the input at stage t to induce the LA to
take the next action (t+1).
The LA in eq. (1) has two dynamical characteristics: One
is the state transition, and the other is the learning update.
At any stage, the next state of the LA depends not only on
the current inputs of the LA but also on the current state.
The LA in eq. (1) learns through updating the action proba-
bilities on the base of the responses from the environment.
The next (new) action probabilities depend not only on the
current action and response from the environment (i.e., the
current input of the LA), but also on the current action
probabilities.
The LA in eq. (1) has three basic elements: (i) A set of
actions (), (ii) a set of reinforcers (), and (iii) a rein-
forcement scheme (A). It appears that the LA can make re-
inforcement learning in an operant conditioning way, in
which the reinforcement scheme A always tries to reward
good actions and to punish bad ones. In terms of psychody-
namics [32], however, there is much difference between the
action probabilities updating in LA and operant conditioning
Figure 1 Learning automaton and its dynamical process, where (t), (t),
and p(t) respectively represent the action of the LA, the response of the
environment, and the action probabilities at stage t.
November (2013) Vol.56 No.11 2747
in animals.
2.2 Association between behaviors and consequences in
OC
All living organisms are dynamical systems. In terms of
system dynamics [33], at any given time an organism has an
internal state, which can be psychological and physiological,
and always evolving over time.
Operant conditioning (OC) is a form of psychological
and physiological dynamic process of organisms, which
makes an association between behaviors and consequences.
In OC, behaviors are supposed to be voluntary and active,
and called operants. Operant, termed by Skinner [2], is a
core concept in the theory of OC, which reflects the basic
characteristics of behaviorism and has twofold important
implications: (i) It is a voluntary and active behavior of or-
ganisms, and (ii) it will generate a certain consequence.
In a sense, the concept of operant in OC is identified with
the term action in the theory of LA.
Any operant of organisms must result in a certain conse-
quence that, in turn, can cause a new operant. In the process
of OC, the alternation of operant and consequence conduces
to an internal feedback loop of organisms through the envi-
ronment as depicted in Figure 2, which contains both the
feed-forward and the feedback associations between operant
and consequence. In the process of OC, consequence is
sometimes regarded as stimulus, and behavior (operant) as
response, and therefore, operant conditioning is also called
the response-stimulus (RS) conditioning. The feed-forward
in the closed loop of OC is an SR process from stimulus
(consequence) to response (operant), which refers to be-
havior selection of organisms, or operant selection in terms
of OC. Whereas the feedback is an RS process from re-
sponse (operant) to stimulus (consequence), which refers to
the state transition of organisms.
In the theory of LA, the inputs (i.e., the responses to ac-
tions) from the environment are regarded as the conse-
Figure 2 The closed loop of alternation of behavior and consequence in
OC.
2748 Ruan X G, et al.
quences induced by actions, such as reinforcers, either re-
wards or punishments for actions. However, we argue that,
in the process of OC, the responses from the environment
are not the real consequences of operants, but only the me-
diate ones. The real consequences induced by operants are
the state changes of the organism in OC. For instance, a
reward for an operant is not the final consequence of the
operant, but a medium between the operant and its conse-
quence, which tends towards bringing the organism to a
certain satisfied internal state that is the real consequence
induced by the operant. Accordingly, in the process of OC,
operant consequences can be supposed to be the state
changes of organisms. To some extent, the current state of
the organism represents the consequence of the last operant.
As a consequence, operant selection in the SR feed-
forward process of OC can be represented with the function
G(p): S in LA (1). Some learning automata formulate
behavior selection with function G: S (p) that means
behavior selection in LA depends on both the state of LA
and the input from the environment. As stated above, how-
ever, the effect of the input from the environment can be
supposed to be contain in the state of the organism in OC.
Accordingly, at each stage t, operant selection in OC can be
generalized computationally with
p  o ( t )| s ( t ) 
SR : s (t )  o(t ),
where s(t) and o(t) are the current state and operant, respec-
tively, and p(o(t)|s(t)) is the occurrence probability of the
operant o(t) at the state s(t), i.e., the probability of the oper-
ant o(t) selected at the state s(t). Eq. (2) formulates the SR
feed-forward process from consequence (state) to operant in
OC, which states that the organism in OC selects its current
operant totally based on its current state.
The state transition of LA as represented with eq. (1) fol-
lows the function F: SS, which means that the state
transition of the LA is driven by the inputs (i.e., the re-
sponses to the actions) from the environment. As we have
argued above, however, the state transition of an organism
in OC can be supposed to be consequences induced by op-
erants. Thereupon, the state transition of OC does not really
depend on the responses from the environment, but on the
operants of the organism itself. Accordingly, at each stage t,
the state transition of the organism in OC can be generalized
computationally with
RS : s (t )  o(t )  s (t  1),
where s(t) and o(t) are the current state and operant, respec-
tively, and s(t+1) is the next state. Eq. (3) formulates the RS
feedback process from operant to consequence (state) in OC,
which states that operant is the power that drives the state
transition of organisms, and the future state of the organism
in OC depends on both the current state and operant.
Actually, in OC, the state transition process of an organ-
ism is a black box. The organism can sense its own state or
Sci China Tech Sci November (2013) Vol.56 No.11
the state change, but cannot tell the process. We need to
notice some differences between eqs. (1) and (3). The state
transition in OC is pushed directly by operants, whereas that
in LA by the responses from the environment to actions.
2.3 Motivation and reinforcement in OC
Reinforcement is a key concept in operant conditioning [34],
which is defined as an event that increases the occurrence
probability of an operant by following the operant with a
certain reinforcer that is something strengthening or re-
warding the operant. The opposite is punishment, which
lowers the probability. Reinforcement plays an important
role in shaping or modifying behaviors of human beings and
animals. If reinforcement is supposed to be either positive
or negative, then both reward and punishment can be re-
garded as some reinforcement. Computationally, in that
case, positive reinforcers can stand for reward, and negative
ones for punishment.
In the process of OC, organisms seem inclined to exhibit
more frequently behaviors that lead to reward, and less fre-
quently lead to punishment. However, we argue that organ-
isms seek not so much for rewards for certain satisfied psy-
chological and physiological feelings or states. Without
doubt, the psychological and physiological feelings or states
(2) of organisms are the most radical reinforcers in the process
of OC or operant learning. Actually, in the process of OC,
the effect of any reinforcement would be finally reflected in
the change of psychological and physiological feelings or
states of organisms. Happiness, joviality, and comfortability,
etc. are positive reinforcers which inclined to increase the
occurrence probabilities of operants, whereas sadness, dis-
appointment, dejection, and melancholy, etc. are negative
ones which inclined to decrease the occurrence probabilities
of operants.
Reinforcement in OC is related to the motivation [35, 36]
that causes organisms to take action or operant. Motivation
is a kind of force, which initiates, guides and maintains
goal-oriented behaviors or operants in terms of OC. Operant
conditioning or operant learning tends towards giving rise to
positive reinforcement when the psychological and physio-
logical feeling of the organism is consistent with the moti-
vation, or when the state transition of the organism is guid-
ed towards the goal that the motivation means, otherwise to
negative reinforcement.
The mechanism of operant conditioning can be charac-
(3)
terized with psychodynamics [32] that is a blend of psy-
chology and thermodynamics. Brucke, the founder of psy-
chodynamics, supposed that all organisms were ener-
gy-systems governed by the first law of thermodynamics
[37]. It can also be characterized with biological thermody-
namics [38] that refers to bioenergetics [39] and investigates
the energy transductions in living organisms. Coincidently,
both psychodynamics and biological thermodynamics are
inspired by thermodynamics and associated with energy,
 Ruan X G, et al.
which suggests that, in the process of OC, biological or
psychological and physiological states require energy to
hold, and operants require energy to operate. Accordingly,
an organism in OC can be supposed to be an energy system
ESYS that consists of two functions: one is the internal ener-
gy function ES that assigns each biological state a nonnega-
tive energy value; the other is the operant energy function
EO that assigns each operant a nonnegative energy value,
that is,
E SYS (t )  ∆E S (t )  E O (t ),
E S : s (t )  R  , E O : o(t )  R  ,
where ES(t)=ES(t1)ES(t) is the increment of the internal
energy from t to t+1. The internal energy ES is potential en-
ergy for organisms to hold the biological states, and the
operant energy EO is kinetic energy for organisms to operate
operants. Eq. (4) implies that the state transition of an or-
ganism in OC driven by an operant is accompanied with the
change of the internal energy and the consumption of the
operant energy. At each stage t, ESYS estimates the conse-
quence (effect) of the operant o(t) at the state s(t) by a
nonnegative energy value.
It is usually supposed that low-energy states are coinci-
dental with the propensity or tropism of organisms [40]. The
motivation of the organism in OC can be supposed to be the
propensity or tropism of the organism to hold the low-
est-energy state by the lowest-energy operant. Thus, the
motivation of an organism in OC can be characterized by a
motivation system MSYS with an objective function that is
based on the energy system ESYS:
M SYS : J  o(t )    t  0 E SYS (t ),

where the objective function J is also called an operant in-
dex. Eq. (5) states that the motivation of the subject of OC
computationally implies the desired goal to minimize the
operant index and to acquire an optimal sequence of oper-
ants {o*(t)}. Therewith, reinforcement (RF) of OC can be
formulated computationally as the update of the occurrence
probabilities of operants:
 0, E SYS (t )  0,
RF : ∆p  o(t ) | s (t )  
 0, E SYS (t )  0,
where p(o(t)|s(t)) is the occurrence probability of the oper-
ant o(t) at the state s(t), p(o(t)|s(t)) is the increment of the
occurrence probabilities p(o(t)|s(t)), and ESYS(t) is the ener-
gy value from the energy system (4).
The motivation and reinforcement of OC can be depicted
with Figure 3, showing that the process of OC can be illus-
trated by a probability updating process. In the reinforce-
ment scheme (6), the operant o(t) at the state s(t) is rein-
forced by giving the increment p(o(t)|s(t)) to the occur-
rence probabilities p(o(t)|s(t)). This implies positive rein-
Sci China Tech Sci November (2013) Vol.56 No.11 2749
forcement (reward) that p(o(t)|s(t))>0, or negative rein-
forcement (punishment) that p(o(t)|s(t))<0. p(o(t)|s(t)) is
the reinforce at the stage t that follows the operant o(t). Re-
inforcement in OC depicted in Figure 3 and in eq. (6) obeys
the principle of the simultaneity of cause and effect, alt-
hough there is a lapse of time between an operant and its
reinforcer.
From the above analysis, there are three fundamental
elements in operant conditioning: (i) Operant that is active
or voluntary behavior, (ii) motivation that is embodied as an
energy system, and (iii) reinforcement that is computation-
(4)
ally be formulated as the update of the occurrence probabil-
ities of operants guided by the motivation system.
Eqs. (2)(6) generalize the behavior selection, state tran-
sition, motivation system, and reinforcement of organisms
in OC, respectively, and become the principles for us to
design the Skinner automaton.
3 Synthesis of Skinner automaton
Inspired with the ideas of psychodynamics and biological
thermodynamics [32, 37–39], a so-called Skinner automaton
(SAUTO) is built in this section as a machine learning mod-
el for artificial systems (autonomous agents) to learn in op-
erant conditioning way like animals.
3.1 Definition of the SAUTO
From Figures 13 and eqs. (2)(6), the Skinner automaton
can be defined with a 7-tuple as
(5) SAUTO  (t , S , O, M , F , G, A), (7)
where t{0, 1, 2, ⋯} represents the discrete time, S a set of
internal states, O a set of operants, M: S(t)O(t)S(t+1)
R+([0,+)) the motivated unit, F: S(t)O(t)S(t+1) the
state transition process, G: S(t)O(t) (p) the operant selec-
tion process, and A the Skinner algorithm.
In the SAUTO, the discrete time t is twofold: (i) to dis-
cretize the process of operant conditioning for digital com-
puting, and (ii) to mark the simultaneity of cause (operant)
(6)
Figure 3 Motivation and reinforcement in OC.
2750 Ruan X G, et al. Sci China Tech Sci
and effect (consequence or reinforcer). Each discrete time
can stand for a stage in the process of OC.
F is the state transition process of the SAUTO that, at
each stage t, maps the current state s(t)S and operant
o(t)O into the next state s(t+1)S. It is generally a black
box and can be deterministic or stochastic.
G is the operant selection process of the SAUTO that, at
each stage t, maps the current state s(t)S into the current
operant o(t)O with the occurrence probability p of the
operant o(t) at s(t). In other words, at each stage t, the
SAUTO selects the current operant (action) based on the
current state s(t).
M is the motivated unit of the SAUTO, which is an ener-
gy system with the internal energy function ES and the op-
erant energy function EO. Based on eq. (4), at each stage t, it
maps the internal energy increment ES(t) from s(t) to s(t+1)
and the operant energy EO(t) of the operant o(t) that con-
duces to ES(t) into R+ ([0,+)). The motivated unit M de-
sires the SAUTO to hold the lowest-energy state by the
lowest-energy operant.
3.2 Occurrence probabilities of operants
From the theory of OC, organisms or animals inclined to
exhibit more frequently behaviors that lead to reward, and
less frequently lead to punishment. In other words, operant
conditioning is inclined to reward an operant so that its oc-
currence probability increases if its consequence is coinci-
dental with the biological propensity or motivation, other-
wise, to punish it so that the probability decreases.
From psychodynamics and biological thermodynamics,
operant conditioning is a thermodynamic process, and
therefore, can be investigated in the way of thermodynamics.
This makes us associate with the Metropolis algorithm and
simulated annealing in which the Monte Carlo method is the
most fundamental.
The Monte Carlo method relies on repeated random
sampling to compute its results [41]. It can be used to inves-
tigate the relation between operants and consequences of the
SAUTO in OC. A state s(t) and an operant o(t) can be se-
lected randomly at each stage t to generate the state s(t+1).
Summing up the results of all times, one can obtain the sta-
tistical properties of F. However, there is no conditioning
between operants and consequences in the Monte Carlo
method.
In statistical thermodynamics applications, Metropolis et
al. presented a modified Monte Carlo procedure [28, 29] to
obtain a practical solution for the configurational integrals,
in which an internal energy function E is defined for ther-
modynamic systems. Their key contribution is that, instead
of unconditionally accepting the random mutations s gen-
erated from the current configuration s, they accept the s
with a probability
November (2013) Vol.56 No.11
 1, ∆E  0,
p( s  | s )    ∆E / KBT (8)
 e , ∆E  0,
where T is temperature, KB is Boltzmann’s constant, and
E=E(s)E(s) is the energy increment. Eq. (8) implies that
the lower the internal energy of the mutations s, the larger
its opportunity accepted. The mutations s will absolutely be
accepted if E0. The Metropolis algorithm is guaranteed to
produce a population of states governed by the Boltz-
mann-Gibbs distribution.
In the Metropolis algorithm, the random mutations s can
be rejected if E > 0. In OC, however, the state s(t+1) cannot
be rejected in Metropolis’ way, as it has already become the
consequence generated by the operant o(t). An organism or
an animal cannot refuse any consequence induced by its
behavior. It can only attempt to avoid the consequence dis-
satisfied by exhibiting less frequently the correlative oper-
ant. Like organisms or animals, the SAUTO regulates the
occurrence probability of the operant o(t) at the state s(t) in
operant conditioning way so that the opportunity that the
state s(t+1) occurs following the state s(t) decreases if it has
a higher internal energy value than the state s(t) and the
operant o(t) has a higher operant energy value, otherwise,
increases if it has a lower internal energy value than the
state s(t) and the operant o(t) has a lower operant energy
value. Here we define the occurrence probability p(o|s) of
an operant o(O) at a given state s(S) as
 1 ESYS ( o|s ) / KBT
 p(o | s )  Z ( s ) e ,
 (9)
 Z ( s )   e ESYS ( o |s ) / KBT ,
 o O
where ESYS(o|s)=ES(o|s)+EO(o) is the energy value of the
energy system in eq. (4), EO(o) is the operant energy of the
operant oO, ES(o|s) is the internal energy increment in-
duced by the operant oO at the state sS, and  is the
statistic of ESYS(o|s) as the o iteratively emerges at the s in
the state transition process of OC. Eq. (9) is consistent with
eq. (6) that generalizes the reinforcement mechanism of OC.
This implies that an operant that has lower operant energy
and induces the internal energy of the SAUTO to descend
tends towards a larger occurrence probability. This is also
analogous to eq. (8) employed in the Metropolis algorithm.
Generally, the state transition process F can be a proba-
bilistic model, a stochastic process, or a physical system
that is rather non-determinative and with uncertainty. We
can take the statistic of ESYS(o|s) in eq. (9) by collecting
the energy value ESYS(o|s) at each discrete time t in various
different ways. For example, ESYS(o|s) can be the arithme-
tic mean or weighted arithmetic mean of the energy values.
One of the optional statistical methods is that, at stage t,
 o  O and s  S , if o  o(t ) and s  s (t ) , then
 Ruan X G, et al. Sci China Tech Sci
E SYS (o | s )   E SYS  o(t ) | s (t )   (1   )E SYS (o | s ) , (10)
where   (0,1) . Eq. (10) is actually a low pass filter,
which is twofold: (i) | E SYS (o | s ) | gradually expands
from zero to a relative stable statistic with iterative operant
conditioning, and (ii) it is some adaptive to the changing
environments.
3.3 Skinner algorithm
The Skinner algorithm is the core of the SAUTO, which
simulates the mechanism of OC and is employed as a rein-
forcement scheme by the SAUTO. The Skinner algorithm
derives its philosophy from the Metropolis Monte Carlo
Method [28, 29] and simulated annealing [30, 31]. With the
Skinner algorithm, the SAUTO runs like a thermodynamic
system and tries to gain some optimal learning.
The Skinner algorithm employs the Monte Carlo method
to repeatedly take the sample of operant-consequence from
the state transition process of the SAUTO at each stage t and
sum up the operant effect. It defines the occurrence proba-
bility of operant with energy function, which is analogous to
eq. (8) of Metropolis algorithm. Moreover, it establishes
conditioned operant of the SAUTO in the way of the Me-
tropolis algorithm. It implies some establishment of condi-
tioned operant in OC that the SAUTO reaches thermal equi-
librium at a specified temperature T. However, the condi-
tioned operants are probably not globally optimal.
Inspired by annealing in metallurgy, simulated annealing
was introduced for globe optimization by Kirkpatrick, Ge-
latt, and Vecchi in 1983 [30], and by Černý in 1985 [31],
which generalizes the Metropolis algorithm through intro-
ducing a temperature scheme. Simulated annealing starts the
Metropolis algorithm with a high temperature, and then,
slowly cools so that the search space gradually shrinks, and
eventually to a small set of states with the global energy
minimum. As a matter of fact, both animal learning and
annealing are gradual optimization processes. Operant con-
ditioning is a process of behavior selection, or in other
words, a process of behavior optimization. At the beginning,
operant learning has a large set of optional operants for each
state, which may induce a great many states including
high-energy ones to emerge. This is analogous to the high
temperature situation in annealing. Gradually, some oper-
ants are no longer selected owing to their violating the mo-
tivation or biological propensity, and lastly, the set of op-
tional operants shrinks to a small one of optimal behaviors.
Obviously, both the simulated annealing and operant condi-
tioning has the analogous characteristics. It is most noticea-
ble that cooling in simulated annealing implies reinforcing
in operant conditioning. According to eq. (9), a ‘good’ op-
erant that induces the internal energy of F to descend will
obtain more opportunity to occur as the temperature cools.
The Skinner algorithm is with five fundamental steps.
November (2013) Vol.56 No.11 2751
1) Initializing: sS and oO, set ESYS(o|s)=0. Let the
initial time t be zero, the initial state be s(t)=s0 (S), the
artificial temperature TA (=KBT) be large enough.
2) Behavior selecting: Compute the occurrence probabil-
ities of the operants (in O) at the s(t) from eq. (9) with
ESYS(o|s(t)), and select an operant o from O using the
probabilities.
3) Operating: Put the operant o(t)=o on the state transition
process F to get the state s(t+1).
4) Summing up operant effect: Get the energy value
ESYS(o(t)|s(t)) induced by the o(t) at the s(t), take the statistic
ESYS(o(t)|s(t)) of the energy system. Replace t by t+1, and
repeat from step 2 until thermal equilibrium at the TA or
enough repetition is reached.
5) Reinforcing by cooling: Decrease the TA according to
a specified cooling schedule, and repeat from step 2 until
the TA is low enough or F reaches the frozen state.
At the start of the Skinner algorithm, all the operants in
O have the same occurrence probabilities for any given state
in S at time t=0. In the end, however, there will be rare op-
erants fired to hold the lowest-energy states.
The Skinner algorithm is able to simulate operant condi-
tioning and behavior selection, which runs in the way of the
Metropolis Monte Carlo Method and simulated annealing.
4 Thermal equilibrium and operant entropy
The Skinner automaton is a thermodynamic system as well
as a self-organizing psychodynamic system. Its self-organ-
izing property is to a certain extent reflected by its thermal
equilibrium phenomenon.
To demonstrate the thermal equilibrium of the Skinner
automaton, we assume a simpler case that F is determina-
tive, and E(S)(o|s) is the mean of the internal energy in-
crement. Eq. (5) then has the following equivalent trans-
formation:
 1  E ( S ) ( o| s )/ KBT
 p(ο | s )  Z ( s ) e ,
 (11)
 Z ( s )   e  E ( o | s )/ KBT ,
(S )
 o O
where E(S)(o|s) = E(S)(s) is the internal energy of the state s
induced by the o at the s. In eq. (11), the occurring probabil-
ity of any operant at a given state is proportional to the
Boltzmann factor [40], that is,
(S )
p (o | s )  e  E ( o| s )/ K BT
. (12)
Consider an ideal case that, s and sS, there exists an
exclusive oO to bring F from s to s. Then from eq. (11),
s and sS, there exists o and oO such that
p( sβ ) p( sα  sβ ) p(oα | sα )   E ( sβ )  E ( sα )  / K BT
  e , (13)
p( sα ) p( sβ  sα ) p(oβ | sβ )
2752 Ruan X G, et al. Sci China Tech Sci November (2013) Vol.56 No.11

where ss represents the state transition from s to s.

Eq. (13) means that, in the case of [36, 40], the states of the
Skinner automaton obey the Boltzmann-Gibbs distribution
and will reach thermal equilibrium at a specified tempera-
ture T.
Generally, the state transition process F can be a proba-
bilistic model, a stochastic process, or a physical system
that is rather non-determinative and with uncertainty. We
can take the statistic of the E(S) in eqs. (5) and (6) by
collecting the energy increment E(S) at each discrete time t
in various different ways. One of the optional statistical
methods is
∆E (S )
t 1 (ot | st )   ∆E(S )
(ot | st )  (1   )∆E (S )
(ot | st ), (14)
t t
where   (0,1) ,  ∆Et( S )  indicates  ∆E ( S )  at time t, and
∆Et( S ) (ot | st )  E ( S ) ( st 1 )  E ( S ) ( st ) is the internal energy
increment induced by the ot at the time t. Eq. (14) is actually
a low pass filter, which is twofold: (i) |  ∆Et( S )  | gradually
expands from zero to a relative stable statistic with iterative
operant conditioning, and (ii) it is some adaptive to chang-
ing environments.
The concept of entropy is originally defined by the sec-
ond law of thermodynamics and is generalized to be a
measure of the disorganization of various systems. Here we
introduce the concept of operant entropy as a measure of the
disorganization of the Skinner automaton to describe the
operant uncertainty in the Skinner automaton, which is de-
fined as
H   p (o | s ) ln p(o | s ).
sS oO
As a matter of fact, operant entropy is just behavioral en-
tropy [41] that is able to describe the behavioral uncertainty
of humans and animals and autonomous agents.
It is obvious that the operant entropy of the Skinner au-
tomaton always decreases as it cools and will lastly con-
verge to zero. This implies that operant conditioning in the
Skinner automaton is a self-organization process.
5 A hungry Skinnerian pigeon
Figure 4 A Skinner box with a hungry Skinnerian pigeon.
first, the bird pecked at the disks randomly. With operant
conditioning, however, it gradually learned to peck at the
right disk.
Like the hungry Skinnerian pigeon, a Skinner automaton
is also able to learn to peck at the right disk to get what it
wants. Here we demonstrate how a Skinner automaton re-
produces the experiment of the hungry Skinnerian pigeon.
A Skinner automaton (SAUTO) is built to simulate the hun-
gry pigeon in the Skinner box with colored disks. The sim-
ulated bird will gain a bit of grain if it pecks at the red disk,
or suffer an electric shock if at the blue. Nothing will hap-
pen if it pecks at the yellow.
The SAUTO simulating the hungry pigeon is rather small
and simple to serve as an illustration. Its state transition
process F can be illustrated with the state diagram in Figure
5 and the state table in Table 1, in which there are only three
states: (i) s(0): being satisfied, (ii) s(1): longing for food, and
(iii) s(2): being in pain. It has three optional operants for the
state s(1): (i) o(1): to peak at the red disk in the box, (ii) o(2):
(15)
to peak at the yellow, and (iii) o(3): to peak at the blue. It has
only an operant o(0) that means to do nothing for the states
s(0) and s(2) so that it spontaneously return to the state s(1).
The internal energy function is defined with E(S)(s(k))=k2, i.e.,
(i) E(S)(s(0))=0, (ii) E(S)(s(1))=1, and (iii) E(S)(s(2))=4. The state
s(0) is the lowest-energy one the simulated bird expects. No
operant energy is considered, i.e., oO E(O)(o)=0.
The simulated results are depicted in Figures 6 and 7.
The initial artificial temperature TA is 10000C, and the
cooling schedule of the SAUTO is exponential, i.e.,
TA (k  1)  σTA (k ), (16)

In a sense, operant conditioning comes from a series of pi-

geon experiments conducted by Skinner. Skinner ever con-
ducted lots of experiments with pigeons to demonstrate his
operant conditioning theory [4]. He designed a soundproof
apparatus, commonly called a Skinner box, with which he
conducted the experiments of operant conditioning. One
version of the box was equipped with a food tray and a dis-
penser, and there were three colored (red, blue and yellow)
disks for a hungry pigeon to peck at (see Figure 4). The bird
would gain a bit of grain from the food dispenser if it
pecked at the red disk, or suffered an electric shock if at the
blue. Nothing would happen if it pecked at the yellow. At Figure 5 The state diagram of the SAUTO simulating the bird.
 Ruan X G, et al. Sci China Tech Sci November (2013) Vol.56 No.11 2753

Table 1 The state transition process of the Simulated Hungry Bird

s(0): being satisfied s(1): longing for food s(2): being in pain
o(0) : to do nothing s(1) – s(1)
(1) (0)
o : to peck at the red – s –
o(2) : to peck at the yellow – s(1) –
o(3) : to peck at the blue – s(2) –

simulating the bird is self-organizing.

In each simulated experiment, the simulated bird pecks at
the colored disks about 2000 times to carry out operant
learning from the high temperature of 10000C to the low of
0.01C. During one of the simulated experiments, the simu-
lated bird pecks at the colored disks 1885 times in all: 1115
times at the red, 445 times at the yellow, and 325 times at
the blue. We assume that the discrete time interval of the
SAUTO is 15 s. It then takes the simulated bird about eight
hours to carry out operant learning for food.
Figure 6 The operant probability graph: the occurring probabilities of the
three optional operants at the state s(1) versus log10(TA(C)) where TA(=KBT)
is the artificial temperature.
6 Why Skinnerian pigeons superstitious

Skinner’s work on superstition in pigeons [42] is much fa-

mous and interesting. He found that pigeons might become
superstitious. To pray food, Skinnerian pigeons tended to
exhibit strange behaviors, such as flapping wings, spinning,
and twisting, which were so-called superstitious actions. It
seemed that Skinnerian pigeons were apt to reinforce their
behaviors, which were coincidental with the occurrence of
food pellet, whereas the food was not related with their be-
haviors at all.
Figure 7 The operant entropy and the mean of the internal energy versus Why might pigeons become superstitious? The Skinner
log10(TA(C)) where TA(=KBT) is the artificial temperature. automaton may be able to reproduce the experiment and
conduce to our understanding of it. Like Skinnerian pigeons,
the Skinner automaton may also become superstitious.
where =0.9.
To demonstrate why Skinnerian pigeons become super-
At first, as shown in Figure 6, the bird simulated with the
stitious, we programs a few simulate Skinnerian pigeons
SAUTO pecks at the colored disks at the state s(1) with the
with Skinner automata. We design an imitating Skinner box
equal probability to one-third. As cooling in the process of
depicted in Figure 8, which is equipped with a food tray, a
operant learning, the SAUTO pecks at the red disk more and
food dispenser, and a timer. The dispenser releases a food
more frequently to get food. With operant conditioning, the
pellet every 15 min on schedule. Whether a food pellet oc-
occurring probability of the operant o(1) of the SAUTO
curs is not related to the behaviors of those birds simulated
gradually rises from 1/3 to 1, and that of the operants o(2)
and o(3) gradually descend from 1/3 to zero. It is very inter-
esting that, to avoid being shocked by electricity, the proba-
bility of its pecking at the blue descends much faster than
that of its pecking at the yellow. Through operant condi-
tioning in the simulated experiment, the simulated Skinner-
ian pigeon with the SAUTO has learned to peck at the right
disk to get food from the food dispenser and not to peck at
the wrong disk.
Figure 7 shows that the mean of the internal energy of
the SAUTO becomes lower and lower as the temperature
descends. Meanwhile its operant entropy gradually de- Figure 8 A simulating Skinner box for reproducing the experiments on
creases. This implies that operant learning of the SAUTO superstition in pigeons.
2754 Ruan X G, et al. Sci China Tech Sci November (2013) Vol.56 No.11

with Skinner automata. Like Skinnerian superstitious pi-
geons, however, the Skinner automata may also reinforce
their behaviors that are accidentally coincidental with the
occurrences of food pellets. We call a superstitious action a
pseudo operant, and call it the pseudo operant conditioning
(POC) to reinforce pseudo operants.
The state transition process of the Skinner automata sim-
ulating superstitious pigeons can be illustrated with the state
diagram in Figure 9 and the state table in Table 2. In the
state diagram, there are only two states: (i) s(0): being satis-
fied, and (ii) s(1): longing for food. The simulated birds have
one operant o(0) that means to do nothing at the states s(0)
and s(1), and six pseudo operants that are likely to occur at
the s(1): (i) o(1): to flap wings, (ii) o(2): to spin, (iii) o(3): to
twist, (iv) o(4): to turn neck, (v) o(5): to sing, and (vi) o(6): to
bow.
The simulated Skinnerian birds have the same state dia-
gram but different internal energy functions, operant energy
functions, and samplings, which represents different psy-
chological and physiological states, biological propensities,
and extents of cognition and learning.
through sufficient operant learning. It has an internal energy
function of E(S)(s(i))=i (i=0,1) and an operant energy func-
tion of E(O)(o(k))=0.5k (0k6). In the process of operant
conditioning, it takes 25 sample paired data of oper-
ant-consequence (response-stimulus) at each fixed temper-
ature.
As depicted with Figure 10(a), with well-balanced oper-
ant learning, the simulated bird has learned to do nothing to
wait for food pellets occurring automatically. This means
that well-balanced pigeons are apt to learn the truth.
6.2 POC due to being hyperactive
The second simulated pigeon is hyperactive. It has an inter-
nal energy function of E(S)(s(i))=i (i=0,1) and an operant en-
ergy function of E(O)(o(k))=0.5k (0k6), which means that
the simulated bird is too energetic.
As depicted with Figure 10(b), the simulated bird being
hyperactive gets into a POC process and becomes supersti-
tious. To pray food, it bows. It seems that pigeons being too
energetic or hyperactive are apt to be superstitious.

6.1 Well-balanced operant conditioning 6.3 POC due to being hyperepithymia

The first simulated pigeon is well-balanced and carries
Skinner fed eight pigeons less than normal so that they
would be hungry. Then he put them in the Skinner box as
depicted in Figure 8. A few days later, he found that six of
them became superstitious.
We programmed eight simulated pigeon with Skinner
automata. They have the same internal energy function as
E(S)(s(i))=25i (i=0,1) that means they are hyperepithymia,
and the same operant energy function as E(O)(o(k))=0.5k (0k
6). They take only 1 sample paired data of operant-con-

Figure 9 The state diagram of the Skinner Automata simulating the su-
perstitious pigeons.

Table 2 The state transition process of the Simulated Superstitious Birds

s(0): being satisfied s(1): longing for food

(0) (1)
o : to do nothing s s(1)
o(1) : to flap wings – s(1)
o(2) : to spin – s(1)
(3)
o : to twist – s(1)
(4)
o : to turn the neck – s(1)
(5)
o : to sing – s(3)
(6)
o : to bow – s(1)
The outside event # – s(0)
Note: The dispenser releases a food pellet every 15 min, which may Figure 10 The operant probability graphs: (a) Results from a well-bal-
occur at the same time when an o(k) (0k6) occurs. anced simulated bird, (b) results from a hyperactive simulated bird.
 Ruan X G, et al. Sci China Tech Sci
sequence at each fixed temperature, which means that they
are biased, and lack sufficient cognition and learning.
However, the simulated birds have different behaviors
because the simulated experiments of operant conditioning
are computationally stochastic processes. The simulated
results are some accidental and unpredictable. As shown in
Figure 11, the simulated birds exhibit different behaviors.
Among eight simulated birds, six of them have become su-
perstitious owing to POC. To pray food, two flaps its wings
(see Figures 11(a), (b)), two spins (see Figures 11(c), (d)),
one turns its neck (see Figure 11(e)), one bows (see Figure
11(f)), and the rest two do nothing (see Figures 11(g), (h)).
As shown in the simulated experiments, it seems that the
higher the state s(1) and the fewer the sample paired data of
operant-consequence at each fixed temperature, the more
the simulated bird is apt to be superstitious.
We acquire three significant suggestions from the super-
stitious Skinner automata: (i) Being hyperactive is apt to be
superstitious; (ii) Hyperepithymia is apt to be superstitious;
(iii) Being biased or lack of sufficient cognition and learn-
ing is apt to be superstitious.
Figure 11 The operant probability graphs: results from the POC of the simulated birds being biased and hyperepithymia.
November (2013) Vol.56 No.11 2755
7 A 3D Wienerian worm
Norbert Wiener, the father of Cybernetics, was very inter-
ested in how machines work like animals. He described a
thought experiment in his book “Cybernetics” [43] and
conceived a mechanism for imitating aphototropic worm,
which we later name as the Wienerian worm. Wiener sup-
posed that an electric bridge could serve as a photosenso-
ry-motor system so that the Wienerian worm was able to
search the darkness as a real aphototropic worm was. Influ-
enced by Wiener, Braitenberg also designed a series of
thought experiments in which a vehicle with simple internal
structure acts in unexpectedly complex ways [44], including
the phototropic and the aphototropic. Both Wienerian worm
and Braitenberg’s vehicle are simple but most significant
because it demonstrates that it is possible for machines to
behave like animals. However, the mechanisms of Wiener
and Braitenberg act like animals just externally but not in-
ternally. Their behaviors are being designed, but not being
shaped by autonomous development, cognition or learning.
The Skinner automaton is able to serve as sensorimotor
2756 Ruan X G, et al.
systems of autonomous agents and robots so that they can
gradually develop their motor skills like animals. We pro-
gram a 3D Wienerian worm shown in Figure 12(a) in a vir-
tual reality environment. It is 4cm in length and 2.4 cm in
width, and equipped with the receptors of two photosensors
PSL and PSR, and the effectors of two motors ML and MR
that respectively drive the left wheel WL and the right wheel
WR.
As depicted in Figure 12(b), a Skinner automaton
(SAUTO) is used to serve as the photosensory-motor system
of the 3D Wienerian worm, which receives state signals
from the receptors PSL and PSR, and sends operant signals
to the effectors ML and MR. Each state of the SAUTO is a
vector of s  (∆ρ, ρ ) where =1 if L>R or else =1,
L and R are the luminances measured by PSL and PSR,
respectively, and ρ  ( ρL  ρR ) / 2 that is divided into 9
levels. There are altogether 18 states in the SAUTO. The
internal energy function of the SAUTO is defined by
E (S )
( s )  250  ρ .
2
Each operant of the SAUTO is a vector of o=(,), where
 is the instructive signal of velocity and  is the instruc-
tive signal of direction;  is divided into 4 levels, i.e., 0, 1,
2, and 3. If =0, we have =0, else =0.157. Accord-
ingly, there are altogether 7 optional operants for the 3D
Wienerian worm. The operant energy function is defined by
E (O ) (ο)  0.025  , (  {0,1,2,3}).
The 3D Wienerian worm is confined to a small ring of
radius 26 cm where a light lies in the center so that the cen-
tre is bright and the edge is dark (see Figure 13). The oper-
ant learning process of the 3D Wienerian worm is shown in
Figure 14, in which an obstacle avoidance scheme will start
up when the 3D Wienerian worm is too close to one of the
walls of the ring. By operant learning, the 3D Wienerian
worm gradually develops its motor skills of negative photo-
tropism to search the darkness. Figure 14 depicts the foot-
prints of the 3D Winerian worm in the ring at different
Figure 12 (a) A 3D Winerian worm. (b) The sensorimotor system of the 3D Wienerian worm.
Sci China Tech Sci November (2013) Vol.56 No.11
temperatures, in which the case at the artificial temperature
TA=10000C takes 3000 discrete times, the others takes
1000 discrete times. Figure 14 implies that the behaviors of
the 3D Winerian worm gets organized more and more as the
SAUTO cools. The 3D Wienerian worm is totally unor-
ganized at the artificial temperature TA=10000C, but gets
highly organized at TA=0.001C. After having finished op-
erant learning, the 3D worm is able to adroitly search the
darkness in practice, and walk from any jumping-off point
to the darkness. Lastly it moves very slowly or stays at the
foot of the walls.
It is most interesting that, by operant learning, the higher
the luminance of the environment, the faster the 3D Wie-
nerian worm moves. The Skinner automaton makes the
Wienerian worm behave like animals not only externally
but internally as well.
8 A robot learning to balancing
(17)
Two-wheeled robots are a class of balancing robots, which
are much absorbing due to their inherent unstable dynamics.
These robots are characterized by the ability to balance on
its two wheels and spin on the spot [45] whereas such abil-
ity is generally man-made and designed in advance.
In a sense, two-wheeled robots are a class of bionic sys-
tems, which imitate human upright posture and attempt to
exhibit some balancing skills to balance their postures. The
(18)
posture balancing skills of a human are developed and
shaped during operant learning, in which operant condi-
tioning plays an important role. Without a doubt, it is sig-
nificant for balancing robots to learn balancing skills like
human and animals.
We have built a physical two-wheeled robot [46] called
Hominid 3 that is 58 cm in height and 22.5 kg in weight
(see Figure 15(a)). Hominid 3 is a flexible balancing robot
with complicated dynamics. It has a flexible lumbar made
of a spring so that it is more bionic and more challenging to
posture balancing. As shown in Figure 15(b), Hominid 3
 Ruan X G, et al. Sci China Tech Sci
Figure 13 The room of a ring that the 3D Winerian lives in.
has the receptor of an inertial measurement unit (IMU), and
the effectors of two motors ML and MR that respectively
drive the left wheel WL and the right wheel WR. The IMU is
used for measuring the tilt angle  of the robot as well as the
angular velocity d/dt.
A Skinner automaton (SAUTO) serves as a sensorimotor
system related posture and movement for Hominid 3 to de-
velop its balancing skills (see Figure 15(b)), which is pro-
grammed and embodied in a digital signal processor (DSP).
The SAUTO’s state is a vector of s  ( ,) measured
by the IMU. The tilt angle and the angular velocity are, re-
spectively, divided into 11 levels as showing in Table 3, and
therefore, there are 121 states in all in the SAUTO’s state set.
Here s= 0 is the upright state of the robot of Hominid 3. The
internal energy function that represents the propensity or
tropism of Hominid 3 is defined by
E ( S ) ( s )  (150 ) 2  1.8  (4500)  (30) 2 ,
where the first item represents the propensity to regulate the
tilt angle, the last to do the tilt velocity, and the middle to do
the coupling between the tilt angle and the tilt velocity.
The SAUTO’s operant that will operate on the effectors
of two motors ML and MR is a scalar o=UPWM ([3000,
+3000]), the value of Pulse Width Modulation (PWM). The
PWM value is divided into 15 levels as listed in Table 4,
Figure 14 The footprints of the 3D worm searching the darkness during the process of operant learning.
November (2013) Vol.56 No.11 2757
and therefore, there are 15 different operants in all in the
SAUTO’s operant set. The operant being selected will be
operated on the servos to control the speed and direction of
the wheels. No operant energy is considered in Hominid 3,
i.e., oO E(O) (o)=0.
We have conducted physical experiments with the robot
of Hominid 3 to demonstrate how a balancing robot
equipped with the Skinner automaton to learn balancing
skills in autonomous operant conditioning way. The initial
artificial temperature TA of the SAUTO is 10000C, the
cooling schedule of the SAUTO follows eq. (16), the coeffi-
cient  is smaller and equals to 0.75 so that the physical
experiment gets a little fast. There is a protector fixed on the
chassis of Hominid 3 to prevent it from tumbling. At the
beginning of each fixed temperature, the robot of Hominid 3
is at rest with a tilt angle of about 28. The discrete time
interval of the SAUTO t is 10 ms. It takes the robot 180 s
(18000 intervals) to do operant learning at each given tem-
perature.
During the physical experiment, if the absolutely value of
the tilt angle is larger than 25, operant learning will stop
and a Proportional-Integral-Derivative (PID) controller will
start up to make the robot upright. Then, if the absolutely
value of the tilt angle is less than 5, the PID control will
stop and the robot will perform operant learning again.
The physical experiment results of the robot’s operant
learning are shown in Figures 1618. The sensorimotor
system of the robot of Hominid 3 and its motor skills are
developing as operant conditioning. Figure 17 shows that,
as the SAUTO cools during the process of operant condi-
(19)
tioning, the robot of Hominid 3 is gradually acquiring bal-
ancing skills and lastly becomes the first two-wheeled robot
that has autonomously learned to balance. As depicted in
Figure 17, the mean of the internal energy and the operant
entropy at each temperature TA decrease as the SAUTO
cools, which implies the self-organizing property of Homi-
nid 3. We have recorded a video of the physical experiment
and only some truncation charts are shown in Figure 18.
2758 Ruan X G, et al. Sci China Tech Sci November (2013) Vol.56 No.11

Figure 15 The robot of Hominid 3: (a) the physical configuration of the

robot, (b) the sensorimotor system made with the Skinner automaton for
the robot to develop its balancing skills.

Table 3 The state levels of the tilt angle and the angular velocity

Level k  (°) d /dt (°/s)

5 (, 15] (, 60]
4 (15, 10] (60, 35]
Figure 16 The tilt angle graphs at different temperature TA: the tilt angle
3 (10, 6] (35, 20] versus the time sensorimotor system made with the Skinner automaton.
2 (6, 3] (20, 10]
1 (3, 0.75] (10, 2.5]
0 (0.75, +0.75) (2.5, +2.5)
1 [+0.75, +3) [+2.5, +10)
2 [+3, +6) [+10, +20)
3 [+6, +10) [+20, +35)
4 [+10, +15) [+35, +60)
5 [+15, +) [+60, +)

Table 4 The operant levels of the PWM value Figure 17 The operant entropy and the mean of the internal energy ver-
sus log10(TA(C)) where TA (=KBT) is the artificial temperature.
Level k UPWM
7 2500
6 1850 9 Conclusion
5 1250
4 750 The Skinner automaton makes the psychological theory of
3 450 operant conditioning ‘take the form of computer programs’.
2 250 The Skinner automaton develops the Monte Carlo method
1 100 from the Metropolis algorithm to Skinner algorithm, in
0 0 which operant (active and voluntary behavior) is introduced
1 +100
as an essential element. As we state in Section 3, although
2 +250
the Monte Carlo method can be used to investigate the rela-
tion between operants (behaviors) and consequences in the
3 +450
process of response-stimulus (RS) conditioning of an or-
4 +750
ganism, it can select neither operants (behaviors) nor con-
5 +1250
sequences. The Metropolis algorithm has modified the
6 +1850
Monte Carlo method so that it is able to select (accept or
7 +2500
refuse) the states or configurations of a system. However,
 Ruan X G, et al. Sci China Tech Sci November (2013) Vol.56 No.11 2759

Figure 18 The robot of Hominid 3 is learning balancing skills as it swaying.

The Metropolis algorithm cannot be directly used to bio-
logical systems. For an organism, the states or configura-
tions imply its behavioral consequence. During the RS con-
ditioning process of an organism, once a consequence
comes into being, it is impossible to refuse it. Now, the
Skinner automaton expands the algorithm to biological sys-
tems, and moreover, it is able to represent behavior selec-
tion mechanism in operant conditioning way. The Skinner
algorithm selects operants (behaviors) via operant condi-
tioning, and it selects consequences via selecting the oper-
2760 Ruan X G, et al. Sci China Tech Sci
ant that induce the consequences.
The work on the Skinner automaton and the Skinner al-
gorithm advances our understanding of animal cognition
and learning. The study on why Skinnerian pigeons become
superstitious provides a significative instance for us to un-
derstand animal behaviors in the way of psychology com-
puting and thermodynamics computing. As demonstrated
with the 3D Winerian worm and the robot Hominid 3, this
work also contributes to robotics, especially to cognitive
developmental robotics (CDR). The aim of CDR is to pro-
vide understanding of how animals’ higher cognitive func-
tions develop and how machines developmentally acquire
cognitive functions in animals’ way [47]. A model of cogni-
tive development starts from a sensorimotor system, in
which operant learning plays an important role. As demon-
strated in the simulated and physical experiments, the Skin-
ner automaton is able to serve as the sensorimotor systems
of autonomous agents. Moreover, this work is beneficial to
machine learning, especially to autonomous machine learn-
ing. Operant conditioning is one of the fundamental mecha-
nisms of animal learning. The Skinner automaton provides
artificial lives and autonomous agents including mobile
robots with replicated neural mechanisms of operant condi-
tioning and makes them learn in animals’ way, so that ma-
chines act like animals not only externally but internally as
well. As a behavioral psychologist, Skinner believed that
the causes of behavior are in the environment and do not
result from inner mental events such as thoughts, feelings,
or perceptions. Rather, Skinner claimed that these inner
mental events are themselves behaviors, and like any other
behaviors, are shaped and determined by environmental
forces [48]. Of course, cognition and learning including
self-learning and autonomous learning are also some be-
haviors.
The behaviors of an organism may be divided into two
classes: the first-order and the second-order. The ones we
call them actions may belong to the first-order behaviors,
and the sort of thoughts, feelings, and perceptions may be-
long to the second-order behaviors. The second-order be-
haviors are the behaviors of the first-order behaviors, and
the causes of the first-order behaviors. The environmental
forces shape the second-order behaviors, and the sec-
ond-order behaviors shape the first-order behaviors. We
claim that the second-order behaviors play important roles
in self-learning and autonomous learning. It is most signifi-
cative and necessary for autonomous systems, such as arti-
ficial lives, autonomous agents, and mobile robots, to learn
autonomously or self-directedly, and become autonomous
learners.
The Skinner automaton could make autonomous agents
including robots become autonomous learners, in which the
operants are the first-order behaviors, the energy function
represents a certain biological propensity or tropism related
to the second-order behaviors. We call the replicated
mechanism of operant conditioning in the Skinner automa-
November (2013) Vol.56 No.11
ton autonomous operant conditioning.
The second-order behaviors and propensity of an organ-
ism are related to the problem of intrinsic motivation [49,
50]. In the Skinner automaton, the energy function related
to the second-order behaviors is in advance designed and
unalterable, but not shaped by ‘environmental forces’. As
the second-order behaviors, how the internal energy func-
tion or propensity is developed in the Skinner automaton
way will be our future work.
This work was supported by the National Natural Science Foundation of
China (Grant Nos. 61075110, 60774077, 61375086), the National Basic
Research Program of China (“973” Project) (Grant No. 2012CB720000),
the National High-Tech Research and Development Program of China
(“863” Project)(Grant No. 2007AA04Z226), the Beijing Natural Science
Foundation (Grant No. 4102011), the Key Project of S&T Plan of Beijing
Municipal Commission of Education (Grant Nos. KM2008- 10005016,
KZ201210005001) and the Specialized Research Fund for the Doctoral
Program of Higher Education (Grant No. 20101103110007).
1 Skinner B F. The Behavior of Organisms. New York: Appleton-
Century-Crofts, 1938. 61–116
2 Skinner B F. Science and Human Behavior. New York: Macmillan,
1953. 45–128
3 Thorndike E L. Animal Intelligence: Experimental Studies. Edison:
Transaction Publishers, 1911. 241–282
4 Watson J B. Behaviorism. New York: People’s Institute, 1924.
141–232
5 Watson J B. Psychology as the behaviorist views it. Psychol Rev,
1913, 20: 158–177
6 Pavlov I P. Conditioned Reflexes: An Investigation of the Physiolog-
ical Activity of the Cerebral Cortex. London: Oxford University
Press, 1927. 219–300
7 Grossberg S. On the dynamics of operant conditioning. J Theor Biol,
1971, 33: 225–255
8 Grossberg S. Classical and instrumental learning by neural networks.
In: Rosen R, Snell F, eds. Progress in theoretical biology. New York:
Academic Press, 1974. 51–141
9 Chang C, Gaudiano P. Application of biological learning theories to
mobile robot avoidance and approach behaviors. Advs Complex Syst,
1998, 1: 79–114
10 Touretzky D S, Saksida L M. Operant conditioning in Skinnerbots.
Adapt Behav, 1997, 5: 219–247
11 Saksida L M, Raymond S M, Touretzky D S. Shaping robot behavior
using principles from instrumental conditioning. Rob Auton Syst,
1997, 22: 231-249
12 Daw N D, Touretzky D S. Operant behavior suggests attentional gat-
ing of dopamine system inputs. Neurocomputing, 2001,38: 1161–
1167
13 Itoh K, Miwa H, Matsumoto M, et al. Behavior model of humanoid
robots based on operant conditioning. In: Proceedings of the 5th
IEEE–RAS International Conference on Humanoid Robots, Tsukuba,
Japan, 2005. 220–225
14 Itoh K, Onishi Y, Takahashi S, et al. Development of face robot to
express various face shapes by moving the parts and outline. In: Pro-
ceedings of the 2nd Biennial IEEE/RAS–EMBS International Con-
ference on Biomedical Robotics and Biomechatronics, Scottsdale,
AZ, USA, 2008. 439–444
15 Sutton R S, Barto A G. Reinforcement Learning: An Introduction.
Cambridge, MA: MIT Press, 1998. 1–86
16 Narendra K S, Thathachar M A L. Learning automata: A survey.
IEEE Trans Syst Man Cybern, 1974, SMC-14: 323–334
17 Thathachar M A L, Sastry P S. Varieties of learning automata: An
Overview. IEEE Trans Syst Man Cybern B Cybern, 2002, 32: 711–722
 Ruan X G, et al. Sci China Tech Sci November (2013) Vol.56 No.11 2761

18 Thathachar M A L, Sastry P S. A new approach to designing rein- 33 Palm W J. System Dynamics. 2nd ed. London: McGraw-Hill Sci-
forcement schemes for learning automata. IEEE Trans Syst Man Cy- ence/Engineering/Math, 2009. 172–283
bern, 1985, SMC-15: 168–175 34 Kiese-Himmel C. Verstärkungslernen: Operante Konditionierung.
19 Lanctot J K, Oommen B J. Discretized estimator learning automata. Sprache-Stimme-Gehör, 2010, 34: 1
IEEE Trans Syst Man Cybern, 1992, 22: 1473–1483 35 Dayan P, Belleine W. Reward, motivation and reinforcement learning.
20 Thathachar M A L, Phansalkar V V. Learning the global maximum Neuron, 2002, 36: 285–298
with parameterized learning automata. IEEE Trans Neural Netw, 36 Oudeyer P Y, Kaplan F, Hafner V V. Intrinsic motivation systems for
1995, 6: 398–406 autonomous mental development. IEEE Trans Evolut Comput, 2007,
21 Phansalkar V V, Thathachar M A L. Local and global optimization 11: 265–286
algorithms for generalized learning automata. Neural Comput, 1995, 37 Brucke E W. Lectures on Physiology. Vienna: Braumuller, 1874.
7: 950–973 38 Haynie D. Biological Thermodynamics. Cambridge: Cambridge
22 Hauwere Y-M De, Vrancx P, Nowé A. Generalized learning automa- University Press, 2001. 293–330
ta for multi-agent reinforcement learning. AI Commun, 2010, 23: 39 Nicholls D G, Ferguson S J. Bioenergetics. 4th ed. Europe: Academic
311–324 Press, 2013. 1–52
23 Viswanathan R, Narendra K S. A note on the linear reinforcement 40 Hopfield J J. Networks, computations, logic, and noise. In: Proceed-
scheme for variable-structure stochastic automata. IEEE Trans Syst ings of IEEE First International Conference on Neural Networks,
Man Cybern, 1972, SMC-2: 292–294 California, USA, 1987. 109–141
24 Poznyak S, Najim K. On nonlinear reinforcement schemes. IEEE 41 Neumann J von. Various techniques used in connection with random
Trans Automat Contr, 1997, 42: 1002–1004 digits, in Monte Carlo Method. Applied Mathematics Series, vol. 12,
25 Stoica F, Popa E M. An absolutely expedient learning algorithm for Washington D.C.: U.S. Department of Commerce, National Bureau
stochastic automata. WSEAS Trans COMPUTERS, 2007, 6: 229–235
of Standards, 1951. 36–38
26 Stoica F, Popa E M. A new evolutionary reinforcement scheme for
42 Skinner B F. ‘Superstition’ in the pigeon. J Exp Psychol, 1948, 38(2):
stochastic learning automata. In: Mastorakis N E, Mladenov V,
168–172
Bojkovic Z, et al., eds. The Proceedings of the 12th WSEAS Interna-
43 Wiener N. Cybernetics: Or Control and Communication in the Ani-
tional Conference on Computers, Stevens Point, Wisconsin, USA,
mal and the Machine. New York: J. Wiley, 1948. 60–132
2008. 268–273
44 Braitenberg V. Vehicles: Experiments in Synthetic Psychology. USA:
27 Simian D, Stoica F. A new nonlinear reinforcement scheme for sto-
The MIT Press, 1986. 95–144
chastic learning automata. In: The Proceedings of 12th WSEAS In-
ternational Conference on Automatic control, Modeling & Simulation, 45 Ooi R C. Balancing a two-wheeled autonomous robot. Dissertation of
Catania, Sicily, Italy, 2010. 450–454 Masteral Degree. Perth: University of Western Australia, 2003. 1–7
28 Metropolis N, Rosenbluth A W, Rosenbluth M N, et al. Equation of 46 Ruan X G, Li X Y, ZHAO J W, et al. A flexible two-wheeled
State Calculations by Fast Computing Machines. J Chem Phys, 1953, self-balancing robot system and its motion control method. China
21: 1087–1092 Patent 200910084259.8, 2010-10-9
29 Jorgensen W L. Perspective on ‘Equation of state calculations by fast 47 Asada M, Hosoda K, Kuniyoshi Y, et al. Cognitive developmental
computing machines’. Theor Chem Acc, 2000, 103: 225–227 robotics: A survey. IEEE Trans Auton Ment Dev, 2009, 1: 12–34
30 Kirkpatrick S, Gelatt C D, Vecchi M P. Optimization by Simulated 48 Wood S E, Wood E G, Boyd D. Mastering the World of Psychology.
Annealing. Science, 1983, 220: 671–680 Boston: Allyn & Bacon, 2004. 333–354
31 Černý V A. Thermodynamical approach to the travelling salesman 49 Baranès A, Oudeyer P Y. R-IAC: Robust intrinsically motivated ex-
problem: An efficient simulation algorithm. J Optim Theory Appl, ploration and active learning. IEEE Trans Auton Ment De, 2009, 1:
1985, 45: 41–51 155–169
32 Horowitz M J. Introduction to Psychodynamics: A New synthesis. 50 Oudeyer P Y, Kaplan F. What is intrinsic motivation? A typology of
New York: Basic Books, 1988. 17–243 computational approaches. Front Neurorobot, 2007, 1: 1–14