VOL. 76, No.

2 MARCH 1969

PSYCHOLOGICAL REVIEW
OPTIMAL BEHAVIOR IN FREE-OPERANT
EXPERIMENTS 1
CHARLES P. SHIMP
University of Utah

Expected Utility Theory successfully predicts the steady-state relative fre-

quencies of the two alternatives in two-key, discrete-trial probability-learning
experiments, in concurrent variable interval schedules, and in concurrent
variable interval schedules modified to study magnitude and delay of rein-
forcement and conditioned reinforcement. It also successfully predicts the
relative frequencies of interresponse times in one-key variable interval
schedules. Different interresponse times between pecks on a single key and
pecks on different keys are treated as different instances of concurrent
operants that have basically the same functional relationships with reinforce-
ment variables. The theory states that an 5" chooses whichever alternative
momentarily has the greatest weighted probability of reinforcement.

In recent years the increasingly powerful
techniques for behavioral control provided by
operant conditioning have enabled experi-
menters to discover quantitative relationships
in the steady-state performance of individual
organisms. This paper shows how several
of these quantitative relationships, obtained
from a wide variety of experimental situa-
tions, can be organized under a small set of
assumptions that constitute a descriptive the-
ory of operant behavior. The basic assump-
tion holds that an organism behaves optimally
in the sense that each of its choices is the
alternative momentarily having the greatest
value, where the value attached to an alterna-
tive equals its momentary objective reinforce-
ment probability weighted by an associated
value. The purpose of this theory is to or-
ganize and relate the data from a maximum
number of experiments under a minimum
1
This work was done at Stanford University and
was supported in part by the author's United
States Public Health Service postdoctoral fellow-
ship. Access to the computer, a PDP-8, was
kindly provided by the Stanford Department of
Preventive Medicine. An abbreviated version of
this paper was presented at the Symposium on
Concurrent Operants at the Washington, D. C.,
meeting of the American Psychological Association,
September 1967.
97
i 1969 by the American Psychological Association, Inc.
number of assumptions. The model treats
only those variables having the largest and
best-known effects on behavior and is, there-
fore, only a first-order approximation. Some
implications of this approach are discussed
below.
It will be shown that each of the experi-
ments discussed below is an instance of the
following paradigm. A subject (S) chooses
repetitively from among a given set of alter-
natives. Each alternative is denoted by an At,
where i ranges over the number of alterna-
tives. Sometimes when A{ occurs it is fol-
lowed by a reinforcing event £4. The momen-
tary probability of reinforcement for response
At, that is, the probability that At, if it occurs,
is reinforced, is called P(Ei}. Each rein-
forcing event £4 has some quantity associated
with it, such as its magnitude, or a delay
superimposed between At and Et, which de-
termines for it a value, V(Et). The specific
functional relationships between these quan-
tities and the F(£«) are best deferred to a
later section, but it is assumed that greater
magnitudes, or shorter delays, will determine
higher values.
It is assumed that an 5" chooses the alterna-
tive for which the momentary objective prob-
98 CHARLES P. SHIMP
ability of reinforcement times the associated
value is greatest, or, in shorter terms, the A±
for which P(Ei)'V(El) is greatest. This
largest weighted reinforcement probability
will be denoted by
MAX[P(£i) • F(E,-)]. [1]
Nonreinforcement of a response is assumed
to have zero value. The assumption em-
bodied in Equation 1 is the invariant core of
the optimal process that will appear through-
out this paper in several different forms, each
of which is determined by a different experi-
mental procedure. It will be noted in passing
that the present theory formally resembles
Expected Utility Theory (Edwards, 1954,
1961; Luce & Raiffa, 1957; Luce & Suppes,
1965). That is, each one of an 6"s choices
is assumed to be of that alternative At for
which
V(E3)
3-1
is greatest. This sum is taken over the /
possible outcomes of At. Since these out-
comes here consist of just reinforcement or
nonreinforcement, and since the latter out-
come has zero value, the largest sum is pre-
cisely Formula 1. This formal relationship
may help to place the present theory into
proper perspective, among the various pos-
sible sorts of quantitative theories, as to aims,
limitations, and so forth. In particular, the
aim of the present theory is to describe suc-
cinctly as much data as possible, and the
theory is limited to steady-state behavior.
The formal assumptions have broad applica-
bility and intuitive appeal, and their merits
and limitations already have been discussed
at length (e.g., Edwards, 1961; Luce &
Suppes, 1965).
Quantitative predictions for a particular
experiment can be made as soon as rules of
correspondence are made between the empiri-
cal variables and the formal terms of the
theory. Although the rules of correspond-
ence for a particular experiment are best
postponed to the discussion of that experi-
ment, it is appropriate to mention here that
the A^ sometimes will be choices of different
keys and sometimes will be choices of differ-
ent interresponse times. In other words, the
present theory predicts both relative and ab-
solute response rates. The first two experi-
ments to be described by Formula 1 are the
discrete-trials probability learning experi-
ment and the concurrent variable interval
schedule of reinforcement. Previous work
(Shimp, 1966) has already indicated that the
behavior of pigeons in these experiments is
very nearly optimal, and thus to show that
this behavior is predicted by a special case
of Formula 1 now will be a simple matter.
The reinforcement contingencies in a con-
current variable interval schedule will be
discussed in detail because the principles
involved are basic to all that follows. Then
the next three experiments to be described
are modified concurrent variable interval
schedules designed to study conditioned rein-
forcement (Herrnstein, 1964a), delay of re-
inforcement (Chung & Herrnstein, 1967),
and magnitude of reinforcement (Catania,
1963b; Neuringer, 1967). Finally, it will be
shown how performance in a one-key variable
interval schedule can be described in terms
of the same formal model. This variable inter-
val schedule is one of the most common base-
line schedules in operant conditioning and
there is a decided need for a useful account
of the behavior it produces. The data typi-
cally recorded from this schedule, and thus
the data the present theory must describe,
are response rates measured either by cumu-
lative records or by relative frequencies of
interresponse times. The other kind of
schedule analyzed here, the concurrent vari-
able interval schedule, is one in which two
separate variable interval schedules are pro-
grammed independently and at the same time
on two separate response keys. An alternative
method of programming this schedule em-
ploys one key on which one or the other of
two variable interval schedules is operative
at any moment, and a second key, called the
change-over key, on which a peck changes
the schedule in effect on the first key (e.g.,
Catania, 1966; Findley, 1958; Shull & Plis-
koff, 1967). In the present paper, these
two methods are treated the same. For this
schedule, programmed either way, as for
most discrete-trials probability-learning pro-
cedures, the most commonly reported data
 OPTIMAL BEHAVIOR IN FREE OPERANT EXPERIMENTS
are choice probabilities. The present analysis
will emphasize, for both one-key and two-
key experiments, the need for other and
more detailed measures of behavior, such as
sequential statistics.
PROBABILITY-LEARNING EXPERIMENTS AND
CONCURRENT VARIABLE INTERVAL
SCHEDULES
Much of the original interest in proba-
bility-learning experiments was due, of
course, to the counter-intuitive prediction
from Statistical Learning Theory (Estes &
Straughan, 1954) that, under certain condi-
tions, an organism would match its choice
probabilities to the reinforcement probabili-
ties. This prediction was counter-intuitive
precisely because it contradicted the predic-
tion that an organism would respond opti-
mally. As it turns out, a pigeon will, after
prolonged training, nearly always choose the
alternative having the highest probability
of reinforcement in discrete-trial probability-
learning experiments (Graf, Bullock, & Bit-
terman, 1964; Shimp, 1966). This alterna-
tive is the same from trial to trial so that
its probability of being chosen eventually
approximates unity. Such behavior is called
maximizing because it maximizes the time
rate of reinforcement. An equivalent de-
scription is that each choice is of the alterna-
tive momentarily having the greatest proba-
bility of reinforcement. Obviously, this
maximizing is a special case of the behavior
predicted by the optimal process described
above, with AI and A2 equal to a choice of
the left or right key, with EI and E2 equal
to a feeder presentation after an AI or an A%,
respectively, and with V(E-L) equal to V(E2)
since there is no difference in reinforcement
delay or magnitude.
Since the time rate of reinforcement thus
appears precisely to control behavior in
probability-learning experiments, does it
similarly control behavior in any other ex-
periments in which reinforcements are also
programmed according to a random sched-
ule? One such experimental procedure is
the concurrent variable interval schedule of
reinforcement. The question of whether a
bird's behavior is optimal here is especially
intriguing because in this schedule the rela-
tive frequency of one response approximately
equals, or matches, the relative frequency of
reinforcement of that response (Catania,
1963a; Herrnstein, 1961; Reynolds, 1963).
This matching has been described as a conse-
quence of
a plausible view of response strength: Rate of re-
sponding is a linear measure of response strength,
which is itself a linear function of frequency of
reinforcement. . . . According to this point of view,
the animals match relative frequency of responding
to relative frequency of reinforcement not because
they take into account what is happening on the
two keys, but because they respond to the keys in-
dependently [Herrnstein, 1961, p. 270].
To differentiate between this account for
matching in terms of response strength and
an account in terms of Formula 1, it is first
necessary to determine the reinforcement
contingencies in a concurrent variable inter-
val schedule. Then the variables P(Et) and
V(Et) in Formula 1 may be computed.
Customarily in this schedule, the probability
that reinforcement is programmed for one of
two alternatives increases almost linearly
with the time since the last choice of alterna-
tive. The approximation to linearity is espe-
cially close over the range of interresponse
times that usually occurs. Thus the proba-
bility of reinforcement for a choice of one
alternative increases while a bird makes
other choices, and nonreinforcement of one
choice resets the probability of reinforcement
of that choice back to its lowest value. In
short, reinforcement probabilities change over
time as a function of both the schedule and a
bird's behavior. Jji general, at any moment,
P(Et) = Rt-ti where Rt is the probability
of reinforcement per unit time for response
AI, t = l , 2 (determined by the schedule),
and ti is the time since the last At (deter-
mined by the bird). For all tt greater than
\/Ri, Ri't( is set equal to 1.0 since it is a
probability. [In arithmetic variable interval
schedules, P(Ei) also depends on the time
since the preceding reinforcement. This de-
pendency affects absolute response rates (Ca-
tania & Reynolds, 1968) but as explained
below, the absolute response rate does not
affect the predictions of relative response
rates in two-key schedules.] Since there are
no delays and since reinforcement magnitudes
100 CHARLES P, SHIMP
are equal, F(#i) = V(E2). Therefore, ac-
cording to Formula 1, maximizing would
consist of a very strong tendency to choose
the alternative that momentarily happens to
have the highest probability of reinforcement,
that is, the alternative which at any moment
has a reinforcement probability equal to
MAX[P(E<)] = [2]
While Equation 2 gives us the optimal
response at any moment, it is not immediately
obvious what the resulting data, averaged
over an entire session, would resemble. For-
tunately, quantitative predictions may be
obtained by letting a high-speed computer
simulate behavior that is at each moment
described by Equation 2. But notice that
one necessary feature of behavior is so far
omitted from the theory. As yet, there is no
provision for establishing an absolute re-
sponse rate. That is, the theory up to this
point describes only which one of two keys
is to be chosen, not when a choice is to occur.
Thus, an additional process is required to
generate a base-line response rate. Unfor-
tunately, the current literature is of little help
here since much less is known about the
absolute response rate than about the relative
response rate in a concurrent variable inter-
val schedule. In fact, there is almost nothing
known about the interresponse-time distribu-
tions and the interresponse-time sequential
statistics. What little is known concerns the
effects of the procedural device known as a
change-over delay, frequently used to reduce
the frequency of adventitiously reinforced
switching behavior. A change-over delay
appears to generate short bursts of responses
that consume somewhat more time than the
change-over delay (cf. Catania, 1961 ; 1963a,
Fig. 7, top cumulative record). However,
the distribution of numbers of responses in
these bursts, and the distributions of inter-
response times at different times after such a
switch from one key to another, are not
known; therefore such local properties of
behavior are omitted from consideration here.
Thus the theory describes an ideal situation
where choice behavior is unaffected by posi-
tion biases and superstitious switching, yet
the effects on the absolute response rate of
the change-over delay, usually required to
avoid these effects, are also omitted. In
short, without quantitative data to guide the
selection of a base-line response rate, a geo-
metric process was selected primarily on the
grounds of convenience. The probability of
a time interval of length k between succes-
sive responses, on either key, was set equal
to q*(\ — q), where 1 — q was the proba-
bility of an occurrence of either response in
some small time interval. Different values
of the single parameter q determining the
constant response rate were found not to
produce different relative frequencies of
choices, at least for parameter values corre-
sponding to realistic response rates.
The computer program can be summarized
now. The probability of reinforcement for
Ai, P(Ei}, equaled at each moment the prod-
uct equal to Ri, a constant determining the
reinforcement rate for Alt multiplied by £«,
the time since the last "occurrence" of A^
When the geometric process assigned an oc-
currence of a response, the computer calcu-
lated P(-Ei) and P(E2), determined which
was larger, and then "chose" the alternative
having the larger one. A programmed re-
sponse occurrence was canceled if P(£i)
equaled F(£ 2 ). A change-over delay was
programmed so that the occurrence of EI
or EZ was never preceded by an AZ or an
A\, respectively, by less than a certain inter-
val of time. This interval, the change-over
delay, appeared to have little effect on the
resulting stat-data and so it was omitted
from the programs for the experiments de-
scribed later in this paper. The programmed
change-over delays were equivalent to ap-
proximately 1 second, and were equal for
both alternatives. Recent data (Shull &
Pliskoff, 1967) indicate that relative fre-
quencies of choices depend on change-over
delays if the delays are unequal and unusually
long. Unequal change-over delays affect the
momentary expected times to reinforcement
on the two keys. These momentary values
affect the predictions of the present theory,
as explained below in the context of delay of
reinforcement. The data of Shull and Plis-
koff seem to be in qualitative agreement with
the predictions given below for the effects of
delay of reinforcement. In short, the simu-
lation program arranged that whenever a
 OPTIMAL BEHAVIOR IN FREE OPERANT EXPERIMENTS
choice occurred, it was the alternative having
the greater reinforcement probability, and
this probability was determined as it is in a
concurrent variable interval schedule.
The computer program just described was
used, then, to simulate maximizing in con-
current variable interval schedules (Shimp,
1966). The resulting stat-data revealed that
maximizing produces a relative frequency of
choices of one alternative that closely ap-
proximates the relative frequency of rein-
forcement of that alternative. In short,
maximizing produces an approximate match-
ing. Thus Equation 2 predicts the correct
overall choice probability for concurrent vari-
able interval schedules. The same computer
program, minus the geometric process, also
was used to simulate maximizing in a dis-
crete-trials choice experiment equivalent to a
concurrent variable interval schedule, except
that the absolute response rate was controlled
by the experimenter, not by the bird. The
stat-data revealed not only matching but close
approximations to two kinds of data unavail-
able from concurrent variable interval sched-
ules : sequential statistics (the probabilities of
a choice given different sequences of preced-
ing choices), and the probability of a choice
as a function of the length of time since the
preceding reinforcement. These data are badly
needed from concurrent variable interval
schedules themselves. The fit between pre-
dicted and observed values appears all the
more striking in these instances since the
present model is deterministic and provides
for no variability. Close fits in such cases
are possible only because of the extraordi-
narily good control provided by the schedules
of reinforcement.
Originally it was implied (Shimp, 1966)
that responding was based on preceding se-
quences of choices. More recent data
(Shimp, 1967a) indirectly suggest that pre-
ceding choices are unlikely to control choice
behavior. These data, plus the demand for
parsimony between the analysis for concur-
rent variable interval schedules here and the
analysis for variable interval schedules de-
scribed below, suggest the present formula-
tion, Equation 2, that is, a formulation stated
only in terms of lengths of time since pre-
ceding choices. Since predictions here de-
101
pend on times since the last At's, it is very
likely that effects of induction across different
interresponse times (cf. Shimp, 1967b) ulti-
mately will have to be considered.
The adequacy of a model is perhaps best
understood in comparison with that of alter-
native models. In the present case, the only
competing account of matching is provided
by the response strength notion, as succinctly
described by Herrnstein in the above quota-
tion. The present model would seem to pro-
vide the better account on several grounds.
First, maximizing provides accurate, quanti-
tative predictions for low-order sequential
statistics. It is not clear how to generate
these statistics from response strength ideas.
Second, as we shall try to show below, the
present model can be generalized to account
for much additional data. It remains to be
seen whether the response strength idea can
be so generalized.
In summary, the one most important con-
sequence of the present analysis is the impli-
cation that matching in concurrent variable
interval schedules results from averaging
over many choices, no one of which obeys
any matching principle. The data from this
schedule are therefore consistent with those
from probability-learning experiments. The
consistent relationship is between momentary
choice probabilities and momentary rein-
forcement probabilities, rather than between
average choice probabilities and average
reinforcement probabilities. This relation-
ship is summarized by the optimal process
defined by Formula 1. Skinner's well-
known warning that a curve averaged over
5s may be unrepresentative of individual Ss
is, in generalized form, appropriate here_^ A
curve averaged over responses from a single
5 may be unrepresentative even of that 5"'s
individual responses.
MODIFIED CONCURRENT VARIABLE INTERVAL
SCHEDULES OF REINFORCEMENT
Although Formula 1 accounts nicely for
the matching between relative rates of re-
sponding and of reinforcement in unmodified
concurrent variable interval schedules, still
other evidence for a more general kind of
matching has been obtained from various
modified concurrent variable interval sched-
102 CHARLES P. SHIMP
ules. The existence of this more general
matching has made the relationship between
maximizing and matching in the unmodified
schedules appear coincidental. In a study by
Catania (1963b), the relative frequency of a
choice approximately equaled the relative
magnitude of reinforcement of that choice,
when magnitude of reinforcement was mea-
sured by the duration of access to food and
when the two variable interval schedules
were equal. Also, Chung and Herrnstein
(1967) showed that the relative frequency
of a choice approximately equaled the relative
immediacy of reinforcement (defined as the
complement of the relative delay of rein-
forcement) ; again, the two variable interval
schedules were equal. The delay of rein-
forcement equaled the duration of a black-
out programmed between response and rein-
forcement. In addition, Herrnstein (1964a)
performed an experiment on conditioned re-
inforcement. He programmed a two-link
schedule in which the first link was a con-
current schedule with equal variable interval
components. The second link was either a
variable interval or a variable ratio schedule
programmed on just one or the other of the
two keys and the remaining key was non-
functional. Choices of one key in the first
link were reinforced by the presentation of
the stimulus associated with primary rein-
forcement on that key in the second link. The
relative frequency of a response to a key in
the first link approximately equaled the rela-
tive frequency of primary reinforcement on
the same key in the second link. Herrnstein
concluded that the relative reinforcing
strength of a conditioned reinforcing stimulus
approximates the relative rate of primary
reinforcement in the presence of that
stimulus.
According to the following analysis, these
experiments are all conceptually identical,
and Formula 1 can account for the data
from each of them. To show this, we first
must provide the rules of correspondence
between the terms in Formula 1, that is, be-
tween P(Et) and the V(E^, and the em-
pirical variables in each of the experiments.
The rules are basically the same for each
experiment so we shall discuss in detail only
Herrnstein's experiment on conditioned rein-
forcement. Obviously, an optimal choice in
the first link would be guided both by the
probability of conditioned reinforcement and
by the delay between conditioned reinforce-
ment and primary reinforcement. More pre-
cisely, a low probability would be offset by
a sufficiently short delay. If a pigeon's be-
havior here maximized the rate of primary
reinforcement, then V(El) would equal l/Dt
and the maximized variable would be
P(Et) • V(Et) = CR, • WO* [3]
where P(Et) is determined exactly as in
Equation 2 since again the schedule is a
concurrent variable interval, and Dt is the
time interval that must elapse between onset
of the conditioned reinforcing stimulus and
primary reinforcement. The intervals DI for
each alternative were actually variables in
Herrnstein 's experiment but for convenience
in the computer simulation of this experi-
ment, each D{ was set equal to the reciprocal
of the arithmetic average rate of primary
reinforcement for alternative At. (Herrn-
stein, 1964b, has shown that the arithmetic
average is not exactly right for this purpose.
However, there is no general agreement on
what should be used. For example, see
Fantino, 1967; McDiarmid & Rilling, 1965.)
When the computer program described above
simulated maximizing, with Equation 2 re-
placed by Equation 3, the resulting predicted
curve was never more than .07 from the
main diagonal representing perfect matching.
The chief discrepancy between theoretical
and obtained curves was that they lay on
opposite sides of the main diagonal.
Equation 3, where the value of a delay is
assumed to equal the reciprocal of that delay,
is a special case of Formula 1. Only in this
special case will the predicted behavior opti-
mize the rate of reinforcement. In general,
the value of a given delay is assumed not to
be so simply related to the length of that
delay. Instead, this relationship clearly ought
to be as consistent as possible with the vast
literature on the effects on behavior of a
delay superimposed between a response and
the succeeding reinforcement. The present
model depends on delay of the reinforcement
only because such a delay may affect the
rate of reinforcement. Therefore, no dis-
 OPTIMAL BEHAVIOR IN FREE OPERANT EXPERIMENTS
tinction is made between a delay that post-
pones reinforcement and must be terminated
by a response, and a delay of equal duration
that postpones reinforcement but terminates
independently of behavior. "Delay of rein-
forcement" usually refers, of course, only to
the second procedure, but here the term will
apply to both unless specifically stated other-
wise. Much of the data on delays of either
type, for example, see Bower, McLean, and
Meacham (1966), Chung (1965), Fantino
(1967), Herrnstein (1964b), Logan (1960),
Perin (1943), Prokasy (1956), Pubols
(1962), Wyckoff (1959), can be sum-
marized, at least qualitatively, by either a
power function or an exponential function.
For present purposes then, we may say that
the literature suggests that the value of a
given delay is either a power function or an
exponential function of that delay, with in-
creasingly long delays having less and less
value.
Both the power function and the exponen-
tial function were employed at different times
to simulate maximizing in Herrnstein's ex-
periment on conditioned reinforcement. Even
though the exponential function gave slightly
better fits, only the power function will be
discussed here since it required but one esti-
mated parameter to give an adequate fit and
therefore gave a more parsimonious account.
An informal parameter search indicated that
a power equal to 2.0 for each bird would
provide an acceptable fit to the data. The
available computational facilities limited the
possible values for this parameter and so
there may be values giving even better fits.
Figure 1 compares the original data from
Herrnstein's experiment with the results of
simulating maximizing where each choice
was of the alternative having the greatest
weighted reinforcement probability, that is,
MAX JZ, • fc • TT; . [4]
The stat-data compare favorably enough with
the real data to suggest that the assumptions
leading to Equation 4 may account for the
major portion of the effect of the independent
variable. Therefore, the data may describe
not a relation between rate of primary rein-
forcement and the conditioned reinforcing
103
1.0
• REAL DATA
0.9 STAT DATA
BEST-FITTING
STRAIGHT LINE
fOA
n 0-7
% 0.6
O 0.1 0.8 0.3 0.4 0.8 0.6 0.7 0.8 0.9 1.0
RELATIVE FREO. OF PRIMARY REINFORCEMENT FOR RESPONSE A,
FIG. 1. Relative frequencies of response Ai dur-
ing the first link as a function of the relative rate
of primary reinforcement on that key in the sec-
ond link. (The real data are from the left panels
of Figure 4 in Herrnstein's, 1964a, paper on con-
ditioned reinforcement. The straight line was
plotted by Herrnstein. The curved line was ob-
tained by computer simultation of the optimal proc-
ess defined by Equation 4. The predicted curve
closely approximates the data and the best-fitting
straight line.)
strength of a conditioned reinforcing stimu-
lus, but perhaps instead, the degree of control
by the probability of primary reinforcement
weighted by a delay-of-reinforcement gra-
dient.
The application of the model to the experi-
ment by Chung and Herrnstein (1967) on
delay of reinforcement requires only that the
appropriate changes be made in the rules of
correspondence between formal terms and
empirical variables. Thus all the rules re-
main the same except that Dt is now the
length of the black-out between an occur-
rence of A{ and the following EL With this
single change, Equation 4 becomes the basis
for the simulation of maximizing for this
second modified concurrent variable interval
schedule. Figure 2 shows the resulting fit
between the stat-data and the real data for
the group of birds that had a standard delay
of 8 seconds. The real data for which the
standard delay was 16 seconds were too
greatly influenced by position preferences to
be suitable for analysis here. The fit revealed
in Figure 2 is fairly good, especially since
104 CHARLES P. SHIMP
i.O
REDRAWN FROM CHUNG AND HERRNSTEIN
0.9
ft REAL DATA REAL DATA
0.8 STAT-DATA STAT-DATA
0.7 MATCHING

0.6

0.5

0.4

; 0.3
0.2

0.1

0 4 8 12 16 20 24 28 32 0 0.1 O.2 0.3 0.4 0.5 0.6 0.7 O.8 0.9 1.0
DELAY OF REINFORCEMENT FOR RESPONSE A, RELATIVE DELAY OF REINFORCEMENT
FOR RESPONSE A,
FIG. 2. The left panel shows the relative frequency of response Ai as the function of the de-
lay of reinforcement for Ai. (The delay of reinforcement for response A* was always 8 sec-
onds. The right panel shows the same data as a function of the relative delay of reinforcement.
The real data are from Figure 2 in the paper by Chung and Herrnstein, 1967. The predicted
curve in each panel was obtained by computer simulation of the optimal process denned by
Equation 4. The predicted curves roughly approximate the obtained curves.)

the parameter was held equal to 2.0, its value
in the different context of Herrnstein's ex-
periment on conditioned reinforcement.
The present analysis suggests that of the
experiments by Chung (1965) and by Chung
and Herrnstein (1967), only the former
directly studied delay of reinforcement, ac-
cording to the usual definition given above.
In Chung's experiment, maximum reinforce-
ment rates on the two keys were equated by
nondelay black-outs. Therefore, the present
model would predict a relative frequency of
.50 for responding on either key. The devia-
tions from this prediction presumably reflect
genuine effects of delay of reinforcement.
The curvilinear function Chung obtained is
consistent with most of the data on delay of
reinforcement. However, the nondelay black-
outs were omitted from the experiment by
Chung and Herrnstein, with the result that
the reinforcement rate, rather than delay of
reinforcement, in the usual sense, was the
controlling variable. That is, maximum rein-
forcement rates were not equated on the two
keys and the present model for optimal be-
havior predicts matching if a curvilinear
delay-of-reinforcement function is assumed.
(The present account therefore shows that
the matching function obtained by Chung and
Herrnstein is consistent with the literature.)
In other words, the present analysis suggests
that the data from the experiment by Chung
and Herrnstein would have been essentially
unchanged if a response had been required,
at the end of the delay interval, to produce
reinforcement.
This prediction from the model recently
has been tested by the author. A paper
currently in preparation will provide the de-
tails of this experiment and only an outline
of the procedure and results will be given
here. The procedure of the experiment
duplicated that of the experiment by Chung
and Herrnstein except that reinforcement
was not automatically presented at the end
of a black-out. Thus, the formal structure
of the experiment was identical to the experi-
ment by Chung and Herrnstein in terms of
reinforcement probabilities and lengths of
black-outs, that is, in terms of the present
model; yet reinforcements were always pre-
sented immediately after a key peck. In
short, there was no delay of reinforcement in
the sense presumably intended by Chung and
 OPTIMAL BEHAVIOR IN FREE OFERANT EXPERIMENTS
Herrnstein. Nevertheless, the data of three
birds replicated the results of their experi-
ment. That is, a bird matched the relative
frequency of a response to the relative imme-
diacy of reinforcement of that response.
Thus, the Chung and Herrnstein data would
seem not to establish a matching function for
delay of reinforcement, as defined tradition-
ally. Instead, the present analysis suggests
that their data actually increase the generality
of a curvilinear function such as that found
by Chung and used in Equation 4. This con-
firmation of an a priori prediction of the
theory reveals that, although the theory is
primarily intended as a means of describing
data, it nevertheless also has predictive power
by virtue of its implicating certain variables
and their relationships as the controlling
factors in a wide class of experiments.
In Catania's experiment on magnitude of
reinforcement, the duration of reinforcement
associated with one alternative increased the
value of that alternative, and so the value
equivalent to Equation 4 is
MAXC& • ti • Df] [5]
where RI and U are as in Equation 4, and
A is the feeder duration for response At.
Catania used feeder durations of 3, 4.5, and
6 seconds to obtain relative feeder durations
of .33, .50, and .67. The relative frequencies
of choices predicted from Equation 5 are
.27, .50, and .73 and approximately match
the relative feeder durations. Thus the same
power function that determined the values of
different delays also determined the values
of different reinforcement magnitudes. How-
ever, this identity between functions pre-
sumably is coincidental. More generally,
when reinforcement magnitude is determined
not by feeder duration but instead by, say,
sucrose concentration, one would imagine
that at least the parameter value in the power
function would differ from the one used here.
The account provided here for the match-
ing behavior in modified concurrent schedules
is parsimonious. First, there is but one esti-
mated parameter in the power function. Sec-
ond, the matching behavior in each of the
above two-key experiments is different from
the behavior generated in one-key experi-
ments. In fact, these matching relationships
105
for delay of reinforcement, magnitude of
reinforcement, and conditioned reinforcement
seem restricted to these specific procedures.
Therefore, equations that describe behavior
in these procedures fail to describe behavior
in analogous one-key procedures or in any
other two-key procedures (Catania, 1963a;
Herrnstein, 1964a). From the present ac-
count we can see why this is so. These equa-
tions are based on the assumption that
matching is the invariant relationship. But
it now appears that matching may be only
a mathematical consequence of maximizing.
The real invariant in these experiments ap-;
pears to be an optimal process. '
Clearly, Equations 4 and 5 are not re-
stricted to the special case for which the
variable interval components are equal. That
is, the model could predict data for the modi-
fied concurrent variable interval schedules
discussed above even if RI were not equal to
R2. The testing of such predictions is a
possible direction for further research in this
area. But the rest of this paper travels still
another route. In the experiments by Chung
and by Chung and Herrnstein, only certain
responses initiated delays. But now suppose
that each response initiated a delay such that
a reinforcement, if programmed, could not
occur until the end of that delay. We shall
show in the next section that, with appro-
priate rules of correspondence, the present
model describes behavior on a one-key vari-
able interval schedule of reinforcement in
terms of this special case in which each re-
sponse initiates a delay.
VARIABLE INTERVAL SCHEDULES
OF REINFORCEMENT
An important assumption behind the de-
scription of variable interval behavior by
Formula 1 was shown some years ago by
Anger (1954) to have empirical validity.
This idea is simply that an ordinary, one-key
variable interval schedule is a concurrent
schedule of reinforcement for different inter-
response times. To show this, Anger devised
a synthetic variable interval schedule in
which reinforcements were programmed
separately for different bands of inter-
response times. He used a different inter-
mittent schedule, typically a fixed interval
106 CHARLES P. SHIMP
schedule, for each of the different bands.
Thus a response terminating an interresponse
time in a given band was reinforced if a cer-
tain length of time had elapsed since the
preceding reinforcement of that band of inter-
response times. Anger selected these lengths
of time, that is, the values of the fixed interval
schedules, so that the rate of reinforcement
in each band approximated the rate of rein-
forcement in that band in an ordinary varia-
ble interval schedule. To obtain the latter
rates, Anger ran his animals on a variable
interval schedule until they stabilized. He
then shifted the animals to his synthetic
schedule and discovered that it maintained
the behavior previously established by the
variable interval schedule. In short, Anger
found that his synthetic variable interval
schedule was equivalent to an ordinary varia-
ble interval schedule, with respect to the
relative frequencies of interresponse times
that each schedule generated.
The essential difference between Anger's
schedule and an ordinary variable interval
schedule is that only the former controls the
relative frequencies of reinforcement of the
different interresponse times. Therefore he
avoided the troublesome interaction normally
prevailing in variable interval schedules be-
tween relative frequencies of interresponse
times and relative frequencies of reinforce-
ment. To show that these relative reinforce-
ment frequencies actually controlled relative
frequencies of interresponse times, Anger
varied the reinforcement frequencies for cer-
tain bands of interresponse times and found
appropriate changes in the interresponse-time
distribution. He also found that long inter-
response times were much less sensitive to a
given change in reinforcement frequency than
were short interresponse times.
In a recent experiment with a simplified
synthetic variable interval schedule, the num-
ber of reinforced interresponse-time bands
was reduced to two, and the problems result-
ing from induction across reinforced inter-
response times were removed (Shimp, 1968).
These controls enabled the results to be more
easily interpreted. The simplified schedule
showed that the relative frequency of an
interresponse time is an orderly function of
both the relative frequency and relative mag-
nitude of reinforcement for that interresponse
time. Indeed, the approximately linear func-
tion appeared similar to the function expected
on the basis of the linear functions in the
two-key concurrent variable interval sched-
ules discussed above. It should be empha-
sized that these functions are for steady-state
behavior and are from a schedule that con-
trols relative frequencies of reinforcement.
Presumably, the functions could be obscured
if (a) data were from acquisition or transi-
tion periods, or, (5) any other relationship
between relative frequencies of interresponse
times and the corresponding relative fre-
quencies of reinforcement were proscribed by
the schedule (such as in variable interval and
variable ratio schedules. See Revusky,
1962; Shimp, 1967b). Thus Malott and
Cumming (1964, 1966) may have obtained
data at variance with Shimp's for the second
reason, and the failure of Blough and Blough
(1968) to find any obvious relationship may
be attributable to both reasons, as well as to
the effects of induction.
In summary, the present theory assumes
the existence of functional relationships be-
tween various reinforcement variables and
relative frequencies of interresponse times.
The main support for this assumption is the
dependency obtained by Anger, and more
importantly, the orderly functions obtained
by Shimp (1967b, 1968). At the present
level of analysis, the mere existence of the
orderly functions obtained from synthetic
variable interval schedules is an adequate
basis upon which to construct quantitative
models. There is no doubt, however, that
identification, and control of, the fundamental
behavioral events occurring between recorded
key pecks would permit analysis on a more
satisfactory level. Presently, some readers
may wish to imagine that behavioral chains
of different lengths "mediate" interresponse
times of different lengths. These chains
would be differentially reinforced by variable
interval schedules.
It is clear then, that Anger's synthetic
schedule is a concurrent schedule of rein-
forcement for different interresponse times
(also see Catania, 1966). More specifically,
it is assumed here that Anger's schedule is
behaviorally equivalent to a concurrent sched-
 OPTIMAL BEHAVIOR IN FREE OPERANT EXPERIMENTS
ule in which each component is a variable
interval schedule for a particular inter-
response time. Therefore the arguments de-
veloped above for choice behavior in two-key
concurrent variable interval schedules apply
to Anger's schedule, and by the equivalence
Anger demonstrated, to variable interval
schedules. Of course, the choices are now
among interresponse times instead of between
two response keys. Since the choice behavior
itself now generates the interresponse-time
distribution, it is no longer necessary to
retain the geometric process previously used
to determine when to respond in the simula-
tions.
The theoretical sequence of events for a
variable interval schedule can now be out-
lined. Imagine a concurrent variable interval
schedule for different interresponse times.
The arbitrarily selected bands of inter-
response times form the different operants
Ai. The probability of reinforcement for one
interresponse time may be said to increase
linearly as a function of the time since the
last response terminating that interresponse
time. Thus, P(Ei) is computed as before.
The event Et is still, of course, a presentation
of the feeder after an At. An 5 chooses the
next interresponse time when he responds
and ends the preceding interresponse time.
When .9 chooses, he behaves as if he scans
the times since he ended the various inter-
response times and then computes values.
The interresponse time corresponding to the
greatest of these values is chosen. In short,
the chosen interresponse time satisfies Equa-
tion 4 where now £« is the time since last
ending an interresponse time in the ith band
of interresponse times, and Rt is obtained
from the data and gives the reinforcement
rate for the «th band. (This length is not to
be confused with the width of the band.)
Unlike earlier theories for operant behav-
ior (Bush & Hosteller, 1951, 1955; Estes,
1950), the present theory does not assume
that S chooses, in every small time interval
h, whether or not to respond. Here, 5"
chooses how long to wait before responding.
Or, in other terms, S1 chooses which medi-
ating behavioral chain to initiate. The as-
sumption that a bird chooses the length of
an interresponse time at its beginning, not at
107
its termination, is critical for the present
model for variable interval behavior. Here,
the length of time between a choice initiating
a certain interresponse time and a succeeding
reinforcement for the response terminating
that interresponse time, is precisely the
length of that interresponse time. Therefore,
the choice is reinforced only after a delay
equal to the chosen interresponse time. The
machinery developed above for two-key con-
current variable interval schedules with de-
lays of reinforcement (Dt~) therefore applies
to one-key variable interval schedules with
the different delays being the different inter-
response times. Notice that this assumption
also makes one-key and two-key schedules
similar in terms of availability of the oper-
ants. In the two-key schedules, a bird may
at any moment choose either key. Thus,
both operants are equally available. Simi-
larly, in the one-key schedule a bird, when-
ever he makes a choice, may choose any
interresponse time. Thus, all the operants
again are equally available. It is emphasized
that the interresponse times are not equally
available if it is assumed that a choice may
occur at any moment. In that case, a long
interresponse time is as available as a short
one only if the short one never occurs. The
reason for making the present assumption is
to allow the same formal model developed
above for choice to apply also to response
rate. Thus, a considerable gain in theoreti-
cal parsimony results from an untraditional
assumption. No relevant data appear incon-
sistent with this assumption. In particular,
the data providing perhaps the best evidence
that an interresponse time is chosen at its
end, not at its beginning, comes from very
early performance in interval schedules
(Anger, 1956; Mueller, 1950). That is,
a flat interresponse-times-per-opportunity
curve is more easily accounted for by the
traditional assumption than by the one made
here. But early in training, the effects of
differential reinforcement of different inter-
response times are not obvious, and there is
yet no particular reason for a bird to attend
to time as a relevant dimension. But in
asymptotic behavior, which is the only be-
havior addressed by the present model, either
108 CHARLES P. SHIMP
or both of these circumstances could con-
ceivably necessitate the present assumption.
Notice that if the termination of an inter-
response time were to reset its probability
of reinforcement to zero and if decision time
were the moment of responding, an inter-
response time could never succeed itself.
But interresponse times, especially short
ones, surely do succeed themselves and thus
the following provision was made for such
response perseveration. A choice of one
interresponse time selectively predisposed an
animal to choose the interresponse time again
because the reinforcement probability for the
second successive choice of an interresponse
time was based on a moment of time in the
very near future. That is, when a response
terminated an interresponse time in the fth
band, ti in Equation 4 was not reset to zero.
Instead, it was reset to a short interval A£
which was estimated from the data. In other
words, the times tt in Equation 4 were always
greater, by A£, than the actual times.
In the two-key experiments analyzed pre-
viously, the experimenter controlled the rela-
tive frequencies of reinforcement. But in
variable interval schedules, the steady-state
reinforcement rate for each interresponse
time depends on 5" and so must be obtained
from the data. Consequently, momentary
maximizing no longer predicts a priori asym-
totic choice probabilities. All the theory
can do is predict the relative frequencies of
the interresponse times, given the relative
frequencies of reinforcement. However, the
model can make a priori predictions for
schedules that are similar to variable interval
schedules but that, unlike variable interval
schedules, do not have a deterministic rela-
tionship between relative frequencies of inter-
response times and of reinforcements. If an
estimate of the parameter Af were available,
one could predict steady-state relative fre-
quencies of interresponse times for such
schedules, including synthetic schedules, be-
fore obtaining the data. Also, when maxi-
mizing successfully predicts behavior in a
variable interval schedule, it also predicts
behavior in the corresponding synthetic
schedule.
The model is used below to predict behav-
ior in a variable interval schedule with an
average interreinforcement interval of 1 min-
ute. The experimental procedure, data, and
other relevant materal are reported elsewhere
in detail (Shimp, 1967b). Figure 4 in this
earlier paper provides the relative numbers
of reinforcements per unit time, that is, the
Rf. The Dl were set equal to the lower
bounds of the interresponse time intervals
shown in Figure 3 of the present paper.
However, for the shortest interval, that from
0 to .6 second, the delay was set equal to .3
second. The rationale for using .3 second
was as follows. It is often the shortest inter-
response time ended by a response with a
topography similar to that of responses end-
ing longer interresponse times (Blough,
1963, 1966; Ray & McGill, 1964). Some
special precautions, described in the earlier
paper, successfully reduced the frequency of
shorter interresponse times to virtually zero.
Since the minimum delay from one response
to the next response was about .3 second, the
minimum delay from the supposed decision
time to the nearest possible reinforcement
was also about .3 second.
The parameter A£ for response persevera-
tion was estimated, purely on the basis of the
resulting fit between observed and predicted
relative frequencies of interresponse times,
to be .5 second for each of the three birds.
Ideally, perhaps, this parameter would be
selected by comparing predicted and obtained
sequential statistics. But as stated earlier,
there are unfortunately no sequential statis-
tics from variable interval schedules. As was
also said earlier, there is no clear consensus
on the appropriate function relating values
to delays of reinforcement. Until such a con-
sensus is established, the present model
should be flexible enough so that at least
power and exponential functions provide
reasonable fits. Such actually was the case
earlier for two-key data and such also is the
case here. One-parameter power functions
were found to provide fairly good fits and
exponential functions did slightly better.
Figure 3 shows the fit between data and
theoretical predictions obtained from com-
puter simulation of maximizing defined by
MAX[U, [6]
 OPTIMAL BEHAVIOR IN FREE OPERANT EXPERIMENTS 109
0.80

0.70 -

0.60 -

0.50

0.40

0.30 -

0.20 -

0.10 -

(0,0.6)
(0.6,0.9) (1.2,1.8) INTERRESPONSE TIMES (SECONDS)

FIG. 3. The relative frequencies of interresponse times in a variable interval schedule with
an average interreinforcement interval of 1 minute. (The predicted curves were obtained by
computer simulation of the optimal process defined by Equation 6, and approximately equal the
obtained curves.)

where e is the base of the natural logarithms,
and C is a parameter estimated by an infor-
mal search method to be 1.5 for Bird 1 and
2.0 for Birds 2 and 3. 'Figure 3 shows that
an optimal process is capable of generating
an acceptable first-order description of varia-
ble interval behavior. An important conse-
quence of the good fit is that the predicted
behavior is stable over a fairly long time,
as it is in variable interval schedules. With-
out a good fit, the predicted distribution of
interresponse times would change the distri-
bution of reinforcements. The changed rein-
forcement distribution would in turn, accord-
ing to Equation 6 with the appropriately
revised reinforcement rates, generate an in-
terresponse-time distribution even further
from the original one, and so forth.
The present model for variable interval
behavior provides an account for a phenome-
non variously termed a response bias in favor
of short interresponse times (Malott & Gum-
ming, 1966), or a greater susceptibility to
reinforcement by short interresponse times
(Millenson, 1966). Such terms are intended
to explain or to describe the tendency for
shorter interresponse times to be more fre-
quent than longer ones, even when rein-
forcement frequencies are equal (e.g., Shimp,
1968). According to the present model,
this tendency follows directly from the non-
linearity of delay-of-reinforcement functions
such as power and exponential functions.
CONCLUSIONS
The greatest virtue of the present optimal
process is its capacity to organize numerous
experiments which at first glance seem un-
related. This process, that of optimizing
weighted reinforcement probabilities of dif-
ferent concurrent operants, provides an alter-
native way to look at behavior in many
familiar schedules of reinforcement. And in
fact, at the present time, no other way ap-
pears to be consistent with the steady-state
behavior of pigeons in probability-learning
experiments, concurrent variable interval
schedules, more complex two-key schedules
based on concurrent variable interval sched-
ules, and one-key variable interval schedules.
110 CHARLES P. SHIMP
Besides organizing data already in the litera-
ture, the model also successfully predicted
data from a new experiment and did so
without any newly estimated parameters.
An alternative account based on the idea
of response strength appears less successful.
That account presently seems restricted to
the very special case of matching in concur-
rent variable interval schedules. If response
strength is a linear function of reinforcement
frequency, and response rate is a linear func-
tion of response strength, then response rate
is a linear function of reinforcement fre-
quency. But neither maximizing in proba-
bility-learning experiments nor behavior in
variable interval schedules can be described
as linear functions of reinforcement fre-
quency. The relative frequencies of inter-
response times in a variable interval schedule
cannot possibly be a linear function of the
relative frequencies of reinforcement (Re-
vusky, 1962; Shimp, 1967b). This impossi-
bility is due to a unique property of a
variable interval schedule. This property
and corresponding properties in other sched-
ules pose difficulties for other theories as well
as for the response strength theory. The
difficulty for the present theory is that the
function relating value, F(£{), to reinforce-
ment delay, Dtl will not be the same, for
example, in variable ratio and variable inter-
val schedules. At present this dissimilarity
cannot be accounted for. But the problem
will have to be faced if the present model is
to be generalized to the full range of classical
free-operant schedules of reinforcement.
While the present model seems quite dif-
ferent from the unitary response strength
notion, it is markedly similar in some of its
formal aspects to other recently developed
models for operant behavior. The assump-
tion that behavior may be described by
decision-making processes, and especially by
optimal processes, is a common one. For
example, Boneau and Cole (1967) success-
fully applied such notions to the behavior of
pigeons in some psychophysical experiments.
In addition, recent work by Logan (1965a,
196Sb) makes use of this assumption. Many
of the differences between his work and the
present paper seem attributable to differences
in experimental procedures. His discrete-
trials procedures allow relatively few mea-
surements per S (measured in the tens in-
stead of in the thousands), tend more to
confound individual differences with other
experimental effects, and employ responses
such as running speeds or choices in T mazes
(instead of interresponse times or choices in
Skinner boxes). A feature common to these
models assuming maximizing is an emphasis
on steady-state behavior. One of the most
challenging problems is to relate such models
to others that attempt to describe behavior in
transition (e.g., see Norman, 1966; Suppes
&Donio, 1967).
The reinforcement schedules analyzed in
this paper, and indeed virtually all free-
operant reinforcement schedules, are not
designed, of course, with their amenability to
formal analyses the foremost consideration.
They are quite complicated behaviorally, and
a complete description of the behavior they
produce will necessarily involve an extremely
large number of variables. This complexity,
together with the primary purpose of this
paper, implies that the fits between predicted
and obtained data shown here are not as good
as those obtainable from an alternative ap-
proach. That is, it would be possible even
now to concoct parameters and functions for
the effects of several variables omitted from
the present model, such as, for example,
temporal discrimination, complexity, and ab-
solute reinforcement rate. [Perhaps the most
important omission is a treatment of Ca-
tania's experiments (Catania, 1962, 1963a)
showing that overall response rate may re-
main constant while local features of respond-
ing vary. But according to one of Catania's
tentative accounts, (Catania, 1962, p. 184)
this invariance may only be a special case of
the effects of the absolute rate of reinforce-
ment.] However desirable the resulting im-
provement in fit might be by itself if the
model provided for such variables, the ap-
proach followed here is to postpone their
addition to the model until more precise,
quantitative data on their effects exist.
REFERENCES
ANGER, D. The effect upon simple animal be-
havior of different frequencies of reinforcement.
Report PLR-33, Office of the Surgeon General,
19S4. (Document #7779, ADI Auxiliary Pub-
 OPTIMAL BEHAVIOR IN FREE OPERANT EXPERIMENTS
lications Project, Photoduplication Service, Li-
brary of Congress, Washington, D. C.)
ANGER, D. The dependence of interresponse times
upon the relative reinforcement of different in-
terresponse times. Journal of Experimental
Psychology, 1956, 44, 145-161.
BLOUGH, D. S. Interresponse time as a function
of continuous variables: A new method and some
data. Journal of the Experimental Analysis of
Behavior, 1963, 6, 237-246.
BLOUGH, D. S. The reinforcement of least-fre-
quent interresponse times. Journal of the Ex-
perimental Analysis of Behavior, 1966, 9, 581-591.
BLOUGH, P. M., & BLOUGH, D. S. The distribution
of interresponse times in the pigeon during vari-
able-interval reinforcement. Journal of the Ex-
perimental Analysis of Behavior, 1968, 11, 23-27.
BONEAU, C. A., & COLE, J. L. Decision theory,
the pigeon, and the psychophysical function.
Psychological Review, 1967, 74, 123-135.
BOWER, G., McLEAN, J., & MEACHAM, J. Value
of knowing when reinforcement is due. Journal
of Comparative and Physiological Psychology,
1966, 62, 184-192.
BUSH, R. R., & HOSTELLER, F. A mathematical
model for simple learning. Psychological Re-
view, 1951, 58, 313-323.
BUSH, R. R., & MOSTELLER, F. Stochastic models
for learning. New York: Wiley, 1955.
CATANIA, A. C. Behavioral contrast in a multiple
and concurrent schedule of reinforcement. Jour-
nal of the Experimental Analysis of Behavior,
1961, 4, 335-342.
CATANIA, A. C. Independence of concurrent re-
sponding maintained by interval schedules of re-
inforcement. Journal of the Experimental Analy-
sis of Behavior, 1962, S, 175-184.
CATANIA, A. C. Concurrent performances: Rein-
forcement interaction and response independence.
Journal of the Experimental Analysis of Be-
havior, 1963, 6, 253-263. (a)
CATANIA, A. C. Concurrent performances: A
baseline for the study of reinforcement magni-
tude. Journal of the Experimental Analysis of
Behavior, 1963, 6, 299-300. (b)
CATANIA, A. C. Concurrents operants. In W. K.
Honig (Ed.), Operant behavior: Areas of re-
search and application. New York: Appleton-
Century-Crofts, 1966.
CATANIA, A. C., & REYNOLDS, G. S. A quantita-
tive analysis of the responding maintained by
interval schedules of reinforcement. Journal of
the Experimental Analysis of Behavior, 1968, 11,
327-383.
CHUNG, S. -H. Effects of delayed reinforcement
in a concurrent situation. Journal of the Ex-
perimental Analysis of Behavior, 1965, 8, 439-
444.
CHUNG, S. -H., & HERRNSTEIN, R. J. Choice and
delay of reinforcement. Journal of the Experi-
mental Analysis of Behavior, 1967, 10, 67-74.
EDWARDS, W. The theory of decision making.
Psychological Bulletin, 1954, 51, 380-417.
111
EDWARDS, W. Behavioral decision theory. In P. R.
Farnsworth, O. McNemar, & Q. McNemar
(Eds.), Annual review of psychology. Palo
Alto: Annual Reviews, 1961.
ESTES, W. K. Toward a statistical theory of learn-
ing. Psychological Review, 1950, 57, 94-107.
ESTES, W. K., & STRAUGHAN, J. H. Analysis of
a verbal conditioning situation in terms of sta-
tistical learning theory. Journal of Experimental
Psychology, 1954, 47, 225-234.
FANTINO, E. Preference for mixed- versus fixed-
ratio schedules. Journal of the Experimental
Analysis of Behavior, 1967, 10, 35-43.
FINDLEY, J. D. Preference and switching under
concurrent scheduling. Journal of the Experi-
mental Analysis of Behavior, 1958, 1, 123-144.
GRAF, V., BULLOCK, D. H., & BITTERMAN, M. E.
Further experiments on probability-matching in
the pigeon. Journal of the Experimental Analy-
sis of Behavior, 1964, 7, 151-157.
HERRNSTEIN, R. J. Relative and absolute strength
of response as a function of frequency of rein-
forcement. Journal of the Experimental Analysis
of Behavior, 1961, 4, 267-272.
HERRNSTEIN, R. J. Secondary reinforcement and
rate of primary reinforcement. Journal of the
Experimental Analysis of Behavior, 1964, 7, 27-
36. (a)
HERRNSTEIN, R. J. Aperiodicity as a factor in
choice. Journal of the Experimental Analysis of
Behavior, 1964, 7, 179-182. (b)
LOGAN, F. A. Incentive: How the conditions of
reinforcement affect the performance of rats.
New Haven: Yale University Press, 1960.
LOGAN, F. A. Decision making by rats: Delay
versus amount of reward. Journal of Compara-
tive and Physiological Psychology, 1965, 59, 1-12.
(a)
LOGAN, F. A. Decision making by rats: Uncer-
tain outcome choices. Journal of Comparative
and Physiological Psychology, 1965, 59, 246-251.
(b)
LUCE, R. D., & RAIFFA, H. Games and decisions:
Introduction and critical survey. New York:
Wiley, 1957.
LUCE, R. D., & SUPPES, P. Preference, utility, and
subjectivity probability. In R. D. Luce, R. R.
Bush, & E. Galanter (Eds.), Handbook of mathe-
matical psychology. Vol. III. New York:
Wiley, 1965.
MALOTT, R. W., & GUMMING, W. W. Schedules
of interresponse time reinforcement. Psycho-
logical Record, 1964, 14, 211-252.
MALOTT, R. W., & GUMMING, W. W. Concurrent
schedule of IRT reinforcement: Probability of
reinforcement and the lower bounds of the rein-
forced IRT intervals. Journal of the Experi-
mental Analysis of Behavior, 1966, 9, 317-325.
McDiARMio, C. G., & RILLING, M. E. Reinforce-
ment delay and reinforcement rate as determi-
nants of schedule preference. Psychonomic Sci-
ence, 1965, 2,195-196.
112 CHARLES P. SHIMP
MILLENSON, J. R. Probability of response and
probability of reinforcement in a response-de-
fined analogue of an interval schedule. Journal
of the Experimental Analysis of Behavior, 1966,
9, 87-94.
MUELLER, C. G. Theoretical relationships among
some measures of conditioning. Proceedings of
the National Academy of Sciences, 1950, 36, 123-
130.
NEUSINGEH, A. J. Effects of reinforcement mag-
nitude on choice and rate of responding. Journal
of the Experimental Analysis of Behavior, 1967,
10, 417-424.
NORMAN, M. F. An approach to free-responding
in schedules that prescribe reinforcement prob-
ability as a function of interresponse times.
Journal of Mathematical Psychology, 1966, 3,
235-268.
PERIN, C. T. The effect of delayed reinforcement
upon the differentiation of bar responses in white
rats. Journal of Experimental Psychology, 1943,
32, 95-109.
PROKASY, W. F. The acquisition of observing re-
sponses in the absence of differential external re-
inforcement. Journal of Comparative and Physi-
ological Psychology, 1956, 49, 131-134.
PUBOLS, B. H. Constant versus variable delay of
reinforcement. Journal of Comparative and
Physiological Psychology, 1962, 55, 52-56.
RAY, R. C., & McGiLL, W. Effects of class-in-
terval size upon certain frequency distributions
of inter-response times. Journal of the Experi-
mental Analysis of Behavior, 1964, 7, 125-127.
REVUSKY, S. H. Mathematical analysis of the
duration of reinforced interresponse times during
variable-interval reinforcement. Psychometrika,
1962,27, 307-314.
REYNOLDS, G. S. On some determinants of choice
in pigeons. Journal of the Experimental Analy-
sis of Behavior, 1963, 6, 53-59.
SHIMP, C. P. Probabilistically reinforced choice
behavior in pigeons. Journal of the Experi-
mental Analysis of Behavior, 1966, 9, 443-455.
SHIMP, C. P. Reinforcement of least-frequent se-
quences of choices. Journal of the Experimental
Analysis of Behavior, 1967, 10, 57-65. (a)
SHIMP, C. P. The reinforcement of short inter-
response times. Journal of the Experimental
Analysis of Behavior, 1967, 10, 425-434. (b)
SHIMP, C. P. Magnitude and frequency of rein-
forcement and frequencies of interresponse times.
Journal of the Experimental Analysis of Beha-
vior, 1968, 11, 525-535.
SHULL, R. L., & PLISKOFF, S. S. Changeover de-
lay and concurrent schedules: Some effects on
relative performance measures. Journal of the
Experimental Analysis of Behavior, 1967, 10, 517-
527.
SUPPES, P., & DONIO, J. Foundations of stimulus-
sampling theory for continuous-time processes.
Journal of Mathematical Psychology, 1967, 4,
202-225.
WYCKOFF, L. B. Toward a quantitative theory of
secondary reinforcement. Psychological Review,
1959,66, 68-78.
(Received January 15, 1968)