Journal of Experimental Psychology: Copyright 1983 by the

Animal Behavior Processes American Psychological Association, Inc.

1983, Vol. 9, No. 4,401-411

The Role of Marking When Reward is Delayed

Glyn V. Thomas David A. Lieberman
University of Birmingham, University of Stirling,
Birmingham, England Stirling, Scotland
Donald C. Mclntosh Peter Ronaldson
University of Birmingham, University of Stirling,
Birmingham, England Stirling, Scotland

Two-choice spatial discrimination by rats is enhanced if a salient stimulus, marker

occurs immediately .after every choice response and again after a delay interval
(Lieberman, Mclntosh & Thomas, 1979). Three experiments further explore this
effect. Experiment 1 found that the second marker is unnecessary. Experiment 2
found that a marker presented before a response is as effective as one presented
after. Both effects could be explained in terms of markers focusing attention on
subsequent cues. Experiment 3, however, found that markers after choice enhance
learning even when no discriminative cues are present following the marker. Markers
thus appear to initiate both a backward search through memory and attention to
subsequent events; both processes help to identify events that might be related to
the unexpected marking stimulus.

Research into the effects of delayed rein-
forcement on animal learning has shown that
even small delays in the presentation of a re-
inforcer can have dramatic effects on learning.
In a classic study by Grice (1948), for example,
delays of .5 sec caused a five-fold increase in
the number of trials required to solve a simple
visual discrimination, whereas delays of 10 sec
prevented learning altogether. In a recent
maze-learning experiment by Lieberman,
Mclntosh, & Thomas (1979), however, pre-
sentation of a brief but salient stimulus im-
mediately following the choice response—a
flash of light, a burst of noise, or handling by
the experimenter—resulted in substantial
learning even when reinforcement was delayed
for 2 min (see also Lett, 1973, 1975).
Why should presentation of a salient stim-
ulus following a choice response facilitate
learning? One possibility is that the stimulus
Experiments 1 and 2 were based on a dissertation by
D. C. Mclntosh, submitted to the University of Bir-
mingham in partial fulfillment of the requirements of a
master's degree. Experiment 3 was based on a dissertation
by Peter Ronaldson, submitted to Stirling University in
partial fulfillment of the requirements for a baccalaureate
degree.
Requests for reprints should be sent to Glyn Thomas,
Department of Psychology, University of Birmingham, P.O.
Box 363, Birmingham B15 2TT, England.
401
is in some way reinforcing, but this would not
explain the differential strengthening of the
correct choice response because the stimulus
follows incorrect choices as well as correct
ones. To account for differential strengthening,
the authors put forward a marking hypothesis
based on Leon Kamin's "surprise" analysis of
classical conditioning (Kamin, 1969). When
an unexpected unconditioned stimulus is pre-
sented, Kamin suggested, subjects initiate a
search through memory to identify possible
predictive cues. Lieberman et al. hypothesized
that any salient and unexpected stimulus ini-
tiates such a search and that during this search
subjects examine previous responses as well
as stimuli. If a response is identified as the
best predictor of the unexpected event, the
memory trace of this response would be
strengthened (perhaps as a result of the extra
rehearsal it receives during the search process)
so that it is more likely to be recalled during
any subsequent memory searches. When a
light flash is presented to a rat in a maze,
according to this analysis, the rat will im-
mediately search its memory to identify pos-
sible causes or predictors of this unexpected
event, and because the light flash immediately
follows the rat's choice response it will be
especially likely to attend to this response and
thereby mark it in memory. When food is sub-
sequently discovered, this event will also trigger
402 THOMAS, LIEBERMAN, McINTOSH, AND RONALDSON

a search through memory for possible causes. after they made their choice response but also
On the basis that salient events are particularly just before receiving food. The;second marker
memorable (cf. the von Restorff effect), might have been critical; a marker presented
marked choice responses are likely to be iden- only after a response might be insufficient to
tified by such a search. Of course, other events produce learning. If so, the paradoxical effects
(very salient ones, such as being placed in the of punishment and of quasi-reinforcement
apparatus or very recent ones immediately could not be attributed to marking because
prior to reward) may also be considered as only a single marker was presented in those
possible causes of reward. It is assumed, how- situations. As to why this second marker
ever, that these other events ultimately do not should be so critical, one possibility is that the
become as strongly associated with reward as first marker does not strengthen the memory
a marked correct response does, because only trace of the preceding response but instead
the latter has a good correlation with reward. becomes associated with it. Then, when the
If this analysis is correct, it could have marker is presented before food, this second
important implications for learning in a wide presentation reactivates the memory trace of
range of situations. Marking could, for ex- the first marker, which, in turn,: reactivates the
ample, help us to understand why contingent memory trace of the original response. Cronin
electric shock sometimes strengthens respond- (1980), Lett (1979), Roberts (1976) and Spear
ing rather than punishes it. In a study by (1978) all suggested that such memorial rein-
Muenzinger (1934), for example, rats were statement might underlie learning with pro-
trained in a T-maze with food as the reinforcer, cedures that resemble those used by Lieber-
Muenzinger found that subjects given a mild man et al.
electric shock immediately after entering the Like the first explanation of marking, this
correct arm learned faster than did subjects second analysis emphasizes memory and re-
not shocked at all. Marking readily explains trieval, but it differs in assuming that two
this paradoxical effect: Shock, a highly salient markers are necessary for learning. Support
event, marks the choice response in the rat's for such an analysis comes from studies of
memory and thus increases that response's human learning by Endel Tulving and his as-
likelihood of recall when food is subsequently sociates (Tulving & Osier, 1968; Tulving &
discovered in the goal box. Pearlstone, 1966). Using a free-recall para-
In a similar way, marking could help to ex- digm, they found that target words were better
plain the phenomenon of "quasi-reinforce- recalled if they were accompanied by weak
ment", first identified by Neuringer and Chung associates during training (e.g., leg-MUTTON)
(1967). They trained pigeons to peck a key, but ony if these associates were also present
but instead of reinforcing a pigeon every time during testing. One possible explanation is that
it satisfied a particular reinforcement schedule the target word and its associate become as-
(e.g., Fixed Ratio 11), they presented only food sociated during training but that this associ-
on a percentage of these occasions. Responding ation does not itself increase the retrievability
under these conditions was initially low, but of the target word: Only if the associate is
if a brief blackout intervened every time the again present during testing will it reactivate
schedule requirement was satisfied, the average its earlier trace and thus lead 'the subject to
rate of responding more than doubled. Because the target word.
controls established that the blackout was not Experiment 1 was designed to test these
a reinforcer, this facilitory effect was difficult possibilities by investigating whether two
to explain, but it would be expected if the markers are necessary to facilitate learning or
blackout served as a marker identifying the only one.
correct response sequence (for further discus-
sion, see Lieberman et al., 1979). Experiment 1
Thus, marking could be a fundamental pro- Method
cess of learning, with implications in a wide
variety of settings (see also D'Amato, Fazzaro, Subjects
& Etkin, 1968). However, Lieberman et al. The subjects were 24 male hooded rats, about 140 days
presented their subjects with a marker not only old and experimentally naive at the start of the experiment.
 MARKING AND DELAYED REWARD
They were housed individually in a different rqanv
the experimental apparatus and were maintained at 80%
of their previously determined free-feeding body weights
for the duration of the experiment.
Apparatus
.The apparatus used was a maze similar to that employed
by Ueberman et al. (1979, Experiment 4). A ground plan
of this maze is shown in Figure 1, together with details
of its dimensions. The walls and floor of the maze were
made of wood. The entire maze, except the side arms,
was painted gray. The right-hand side arm was painted
white, the left-hand side arm black. Vertically sliding guil-
lotine doors made of opaque gray plastic separated the
different sections of the maze. The guillotine doors from
the choice box to the side arms could be opened simul-
taneously by means of a pulley system. All the sections
of the maze were 15 cm high and were covered with re-
movable clear plastic lids. A 35-ohm miniature loudspeaker
was mounted on the center of the lids of each of the two
side arms and the goal box. These loudspeakers were con-
nected to a white-noise source via an electronic timer set
to deliver a 2-sec burst of noise at approximately 90 dB
whenever a hand switch was operated by the experimenter.
Reinforcement consisted of 2.5 gm (dry weight) of pow-
dered rat diet mixed with water into a wet mash and
presented in a small dish placed in the goal box. A small
barrier of clear plastic was placed across half of the goal
box and in front of the dish so that the rats had to enter
the goal box completely to investigate its contents.
Procedure
Pretmining, All the subjects were handled for 1 wk.
before the start Of the experiment. During this period the
food deprivation regimen was established, and the rats
were introduced to the food dishes and the wet mash to
be used as reinforcement in-the experiment.
Delayed reward twining. In view of the overall black
preference initially displayed by the rats studied by Lie-
berman et al. (1979) in mazes similar to the one used in
the present experiment, it was decided to reward all subjects
for running to the white (right) side arm. The subjects
were randomly assigned to three equal groups of eight
subjects each. The first group was designated the "two-
marker group"; the second, the "one-marker group";'and
the third, the "no«marker group."
On training trials, all Subjects were treated in exactly
the same way except for the delivery of noise as a marking
stimulus. Each subject was placed into the start box of
the maze, and after 10 sec the door to the choice box was
opened. If the subject had not entered the choice jbox
within a further interval of 10 sec, it was gently guided
SIDE ARM
^-^
I BLACK 1
START CHOICE -• • DELAY GOAL
BOX BOX - - BOX:
BOX
SIDE ARM
( WHITE 1
-^^ Ocm i 1
DOORS 4-—)-
Figure I. Ground plan of the maze.
through the door by hand. When the subject was in the
choice box, the door from the start box was lowered behind
it and 10 sec later the doors to the side arms were raised
simultaneously and the subject allowed to enter either side
arm. Once the animal's entire body,'excluding the tail,
was within the side arm, the door was lowered to prevent
retracing.
Exactly 2 sec after a choice response, the door at the
far end of the side arm was raised and the animal was
allowed to walk into the delay box, where it was confined
for 120 sec with all doors closed. Following the delay in-
terval, the door to the goal box was raised to allow the
subject to enter. As soon as the subject was in the goal
box, the door was lowered behind it to prevent retracing.
If the previous choice response had been correct, the food
dish in the goal box contained wet mash. If the previous
choice had been incorrect, then the food dish was empty
and the subject was confined in the goal box before being
returned to the start box and run again. The food disk-
(empty or full) was placed in the goal box just before the
end of the delay interval to eliminate food odor as a cue
during the delay.
Subjects in the two-marker group were presented with
a 2-sec burst of noise in the side arms immediately after
each choice response and again immediately after they
had entered the goal box. Subjects in the one-marker group
were presented with a 2-sec burst of noise after each choice
response but not on entry into the goal box. After their
choice responses and entries into the goal box, subjects
in the no-marker group were treated in exactly the same
way as the two-marker group and the one-marker group
subjects except, of course, that no noise was presented on
either occasion.
A correction procedure was employed so that each an-
imal was run repeatedly until a correct choice was made
or until it made eight perseverative errors. In this event,
the subject was given a forced choice on the next run in
which only the door of the correct side arm was opened,
but the subject was otherwise treated exactly as if the
correct choice had been made freely. In either case, after
reward was presented and consumed, the trial was ter-
minated, and the animal was returned to its home cage.
If on any run an animal failed to make a choice by
entering one of the side arms within 120 sec, it was guided
by hand into, one of the side arms. After such a guided
choice, the animal was treated exactly as if the choice Had
been made freely. If a subject required more than one
guided choice, it was guided alternately to the correcfand
the incorrect side arms. Guided and forced choices were
disregarded in the analysis of the data.
Initially, all subjects were given, one trial per day. As
progressively fewer incorrect choices were made and la-
tencies shortened, the number of trials for each subject
was increased to two per day, but with at least 1 hr. between
trials. To help minimize the possibility that odor trails left
by preceding subjects could influence subjects' choices,
the entire maze was wiped with a solution of soap and
disinfectant after each trial (see Lieberman et al., 1979;
Roberts, 1976). The order in which subjects were run was
varied from day to day.
Following the completion of each day's trials, each sub-
ject was given sufficient dry food in its home cage to main-
tain it at 80% of its free-feeding weight. The experiment
was run until each subject had received 50 trials. :
404 THOMAS, LIEBERMAN, McINTOSH, AND RONALDSON
100i
00
LU
oo
z The ANOVA also revealed that the improve-
o
CL
00
cc. 50-
O O ONE MARKER GROUP
• • TWO MARKER GROUP
ex,
LU O—D NO MARKER GROUP
o.
1 - 2 3 4
BLOCKS OF TEN TRIALS
Figure 2. Percentage of correct first responses for blocks
of 10 trials for each group in Experiment 1.
Results
The principal data from this experiment
are the number of correct first responses over
trials for each group. These data are presented
in Figure 2 as percentages over blocks of 10
trials. It can be seen that all groups started at
approximately the same level: around 45%
correct over the first block of 10 trials. It can
also be seen in Figure 2 that all groups showed
some improvement in correct responding over
the 50 trials of the experiment, although the
no-marker group showed only slight improve-
ment. Both the marked groups showed con-
siderable improvement, increasing from ap-
proximately 45% correct on the first block of
10 trials to approximately 85% correct on the
final block of trials. No obvious differences
between the data for the two-marker and the
one-marker groups are apparent in Figure 2.
A two-way analysis of variance (ANOVA) us-
ing the factors of experimental treatment and
trial blocks revealed that there were significant
differences between the groups in overall cor-
rect responding, F(2, 21) = 13.36, p < .01.
Post hoc t tests showed that there was no sig-
nificant difference in overall correct responding
between the two-marker and the one-marker
groups, f(14) = .89, p > .05. However, the
overall correct responding for the no-marker
group did differ significantly from the com-
bined scores of the other two groups, t(22) =
1.91, p<. 05.
ment in correct first responses over trials was
significant, F(4, 84) = 13.34, p< .01. A trend
analysis showed that the overwhelming trend
of the improvement over trials was linear, F( 1,
84) = 52.25, p < .01, and that there were
significant differences between the groups in
their, linear trends over trials, F(2, 6) = 12.63,
p<.0l.
Post hoc t tests were then carried out to
compare the linear trends of each of the
marked groups separately with that of the no-
marker group, using the procedure devised by
Dunnett (1955). In both cases the difference
was significant: two marker versus no marker,
5 t(2l) = 3.31, p < .01; one marker versus no
marker, /(21) = 2.83, p < .05.
As suggested by Figure 2, then, the two-
marker and the one-marker groups showed
significantly more improvement in correct re-
sponding over trials than did the no-marker
group. The rationale of Dunnett's test did not
permit us to directly compare the linear trends
of the marked groups, but these two groups
did not differ significantly from each other in
terms of overall correct responses.
The mean latencies of choice responses, av-
eraged over trials, were as follows: one-marker
group, 4.6 sec; two-marker group, 4.6 sec;
and no-marker group, 4.2 sec. An ANOVA in-
dicated that these scores did not differ signif-
icantly, F(2, 21) = 1, p > .05.
Discussion
The results of Experiment 1 show clearly
that it is not necessary to present a marker
twice for it to be effective: Even a single pre-
sentation, providing it follows the response
closely (see Lieberman et al., 1979), may fa-
cilitate learning. It is less clear, whether re-
peating the marker before food enhances
learning. On several counts, such enhancement
might be expected. One possibility already
discussed is that presentation of a second
marker would remind the subject of the first,
and thus of the choice response. Alternatively,
presenting the marker immediately before food
might increase its salience by endowing it with
secondary reinforcing properties. This in-
 MARKING AND DELAYED REWARD
creased salience would then make the marker
more effective because subjects would be more
likely to attend to it and would expend more
effort in searching for its cause. There are,
therefore, grounds for expecting two markers
tq,be more effective than one, and indeed sub-
jects in the two-marker group did learn faster.
The difference between the two groups* how-
ever, was statistically insignificant. Whether the
second marker has any effect, it is clearly not
necessary for learning.
Experiment 2
On the surface, the phenomenon of marking
seems closely analogous with a common ex-
perience in human memory: When people ex-
perience an event of special importance, they
may later be able to recall not only the event
itself but also many of its circumstances. In
the case of John F. Kennedy's assassination,
for example, many people can vividly remem-
ber what they were doing when they heard the
news, and Brown and Kulik (1977) have la-
beled these memories "flashbulb memories."
Although Brown and Kulik focused on mem-
ory for events that preceded the news of Ken-
nedy's assassination, anecdotal evidence sug-
gests that people also have vivid memories for
what they did after hearing the news. One of
the authors, for example, remembers hearing
the news while returning from a physical ed-
ucation class and then spending the rest of the
afternoon standing on the mall of Columbia
University, talking with other students and
waiting in a slightly stunned state for further
news.
If the analogy between marking and flash-
bulb memories is meaningful, we might expect
that a salient event such as a bright light or a
loud noise would enhance memory not only
for preceding events but also for those that
follow, and the primary purpose of Experiment
2 was to investigate this hypothesis. Three
groups were run, with subjects receiving a
marker either before making their choice re-
sponse, afterward, or not at all. The procedure
for subjects in the marked-after group was es-
sentially identical to that used for the one-
marker group in Experiment 1, with a single
marker presented immediately after they made
their choice response and entered one of the
side arms. In the marked-before group, on the
405
other hand, the marker was presented just be-
fore the doors to the side arms were raised. If
markers enhance recall for subsequent re-
sponses as well as preceding ones, subjects in
the marked-before group should also be more
likely to recall their choice response (it is the
first response they make after the marker is
presented), so that learning in both groups
should be roughly comparable.
As a further check on the assumption that
markers improve learning only when tem-
porally contiguous with the choice responses,
a fourth group was included. Subjects in this
group progressed through the maze in exactly
the same way as those in the marked-after
group, but instead of receiving the marker as
soon as they made their choice response, it
was not presented until 30 sec later. If the
effectiveness of a marker depends on its tem-
poral proximity to the response, then learning
in this group should be significantly weaker
than in the marked-before and the marked-
after groups.
Method
Subjects
The subjects were 32 male hooded rats. They were ap-
proximately 140 days old and experimentally naive at the
start of the experiment. Housing and maintenance con-
ditions were the same as in Experiment 1.
Apparatus
The maze used in this experiment was the same as in
Experiment 1 with the following modifications. For subjects
marked before their choice response, an additional loud-
speaker was mounted on the center of the lid of the choice
box (see Figure 1). This loudspeaker was connected in
parallel with the loudspeakers over the side arms and goal
box and allowed a 2-sec burst of noise to be presented to
subjects in the choice chamber. All other details of the
apparatus were exactly the same as in Experiment 1.
Procedure
Pretraining. Pretraining was the same as in Experi-
ment 1.
Delayed reward training. The subjects were randomly
assigned to four groups of eight,subjects: marked-after,
marked-before, delayed-marker, and no-marker groups. The
subjects were taken individually in a small transfer cage
to the experimental room for testing.
Other than changes in the scheduling of the marker's
presentation, the only change from the procedure, of Ex-
periment 1 was that all subjects were confined in the ap-
propriate side arm for 30 sec following both correct and
incorrect choices rather than for 2 sec. After this period
406 THOMAS, LIEBERMAN, McINTOSH, AND RONALDSON

had elapsed the door to the delay box was opened and the trials to about 80% correct at the final block.
subjects were allowed to enter and spend the remaining In contrast, the performance of the delayed-
90 sec of the delay there before they were given access to
the goal box. Thus, the total delay was 2 rain as in Ex- marker and the no-marker groups showed no
periment 1, but the first 30 sec of the delay were spent inclear change over the course of the experiment,
the side arm rather than in the delay box. remaining close to 50% correct throughout.
Subjects in the marked-after group received a 2-sec burst The data shown in Figure 3 were subjected
of noise immediately after their choice response. Subjects
in the marked-before group received a similar 2-sec burst to a two-way ANOVA, the factors being exper-
of noise, but it was presented immediately before the doorsimental treatment and' blocks of trials. The
to the side arms were raised to allow subjects to make analysis revealed that there were significant
their choice responses. Subjects in the delayed-marker group
differences in overall correct responding ac-
received a 2-sec burst of noise immediately before they cording to experimental treatment, F(3,28) =
were allowed to enter the delay box, that is, 30 sec after
their choice responses. Subjects in the no-marker group 3.59, p < .05, and that there was also a sig-
nificant effect of trials, F(4, 112) = 9.59,
did not experience white noise at any point in their training
trials. p < .01.
The correction procedure and the procedures for guided To determine the source of the significant
and forced choices, control of odor trials and reinforcement
were exactly the same as in Experiment 1. All subjects treatment effects, t tests were carried out on
the scores of the different groups averaged over
were initially given one trial daily, but as errors declined
trials. No significant difference was found be-
and latencies shortened, the number of daily trials for each
subject was increased to two per day, with at least 1 hr. tween the marked-after and the marked-before
between successive trials for each subject. groups, f(14) = .57, p > .05. There was also
no significant difference between the delayed-
Results marker and the no-marker groups, t( 14) = .29,
Figure 3 shows the percentage of correct p > .05, but the combined scores of the
first responses averaged over blocks of 10 trials marked-before and the marked-after groups
for each of the four treatment groups. It is did differ significantlyj from-the combined
apparent from this figure that the performance scores of the delayed marker and the no-
of the marked-before and the marked-after marker groups, f(30) = 3:68, p < .01. Thus,
groups improved steadily over blocks of trials the averaged scores for correct responding of
from about 45% correct at the first block of the marked-before and the marked-after
groups were significantly higher than those of
the delayed-marker and the no-marker groups.
As in Experiment 1, a trend analysis was
100 then carried out to examine changes in per-
formance over trials. The trend analysis re-
vealed that the only significant trend over trials
was linear, F(l, 112) = 36.57, p < .01, and
that there were significant differences between
the linear trends of the groups, F(3, 9) - 5.49,
p < .05.
Post hoc t tests were then carried out to
50- compare the linear trends of each of the
marked groups with that of the no-marker
(control) group (Dunnett, 1955). Both the
• NO MARKER GROUP markedrbefore and the marked-after groups
O D MARKED BEFORE GROUP were found to differ significantly from the no-
• MARKED AFTER GROUP marker group in terms of their linear trends,
O—O DELAYED MARKER GROUP r(28) = 27.56 and *(28) = 23.08, respectively,
both with p < .01. The linear trend of the
delayed-marker group did not differ signifi-
1 2 3 U 5 cantly from that of the no-marker group,
BLOCKS OF TEN TRIALS Z(28) = 2.01, p < .05.
Figure 3. Percentage of correct first responses for blocks The mean latencies of choice responses av-
of 10 trials for each group in Experiment 2. eraged over trials Were marked-before group,
 MARKING AND DELAYED REWARD
3.8 sec; marked-after group, 4.0 sec; delayed-
marker group, 4.9 sec; and no-marker group,
3.6 sec. The differences between these groups'
means were not significant, F(3, 28) =1.53,
p > .05.
Discussion
One purpose of this experiment was to eval-
uate, under more tightly controlled conditions,
Lieberman et all's (1979) finding that the ef-
fectiveness of a marker depends on its conti-
guity with the response to be learned. This
conclusion was confirmed: Subjects receiving
a marker immediately after choice responses
showed significant learning, whereas those for
whom the marker was delayed 30 sec did not.
Because both groups spent identical periods
in the black and white side arms, moreover,
this result cannot be attributed to differential
exposure to the discriminative cues.
The second major purpose of this experi-
ment was to test the intuition that markers
may enhance memory for subsequent events
as well as preceding ones. This prediction was
also confirmed: Subjects that received a marker
immediately before making their choice re-
sponse learned as well as those receiving one
immediately after. Although this result con-
firmed the intuition that generated the exper-
iment, it poses, however, difficulties for the
backward search mechanism proposed by
Lieberman et al. Because the noise in the
marked-before condition preceded the choice
response, any memory search it initiated could
not have detected the choice response. Indeed,
any such search could have interfered with
learning, because it would mark whatever the
rat was doing just before the marker was pre-
sented. Facilitated recall of this irrelevant be-
havior would presumably interfere with learn-
ing of the current choice.
Because learning in the marked-before
group cannot be attributed to a backward
search mechanism, how can it be explained?
One possibility involves the effects of a salient
stimulus on attention. One such effect appears
to be a focusing of attention (Easterbrook,
1959; Telegdy & Cohen, 1971). Thus rats that
receive a noise marker before making their
choice response may be more likely to focus
their attention on the black and white; cues
facing them and thus be better able to recall
4Q7
which cues they chose when they later receive
food.
If we assume that markers presented after
a choice response also focus attention in this
manner, then learning in the marked-after
group could be explained by attention without
invoking any memory search. When the rat
enters one of the side arms, presentation of
the marker will focus the rat's attention on
the dominant cues in its environment. Because
the most salient of these is likely to be the
chamber's color (black or white), the rat would
be especially likely to attend to this brightness
cue and thus would be likely to recall it when
later fed in the goal box.
The learning that occurs when a marker
follows a choice response, then, could be due
to a backward memory search for possible
causes (Lieberman et al., 1979), but it could
equally be explained by increased attention to
subsequent events. The purpose of Experiment
3 was to obtain further evidence concerning
the relative plausibility of these explanations.
Experiment 3
If the backward search hypothesis is correct,
a marking stimulus triggers a search through
memory to identify preceding events that
might have caused it. If the attention hypoth-
esis is correct, markers focus attention on the
dominant cues present immediately following
the marker. In Experiment 2, with distinctive
side arm cues present both before and after a
choice, these hypotheses predict equivalent
learning. If the white and black discriminative
cues could somehow be removed as soon as
the rat made its choice response, however, then
the two hypotheses would lead to very different
predictions.
To achieve this aim, in Experiment 3 the
doors leading to the side arms were painted
black or white, but the side arms themselves
were an identical shade of gray. To eliminate
other possible cues, the two side arms were
randomly interchanged over trials, and the po-
sition of the maze within the room was rotated.
Once a rat entered a side arm, therefore, no
cues were available to indicate whether or not
it had made a correct response.
Because these changes might make the dis-
crimination problem considerably more dif-
ficult, the delay of reinforcement was reduced
408 THOMAS, LIEBERMAN, McINTOSH, AND RONALDSON

from 120 sec to 30 sec. Additional discrimi- open, followed, on the next day, by a single rewarded place-
native cues were also made available in the ment in the goal box.
Delayed reward training. The subjects were randomly
choice box. The doors leading to the side arms assigned to two groups of eight subjects: the marked group
and their surrounds were painted black or and the unmarked group. Subjects were taken individually
white, and a small light was mounted im- in a small transfer cage to the experimental room for
mediately above the white door. testing. For all subjects a correct response was denned as
an entry into the right-hand side arm (entered via a white
A number of discriminative cues were thus door and surround).
available at the moment of choice, but none The procedure for delayed reward training was the same
were still present when the marker occurred. 'as in Experiment 1 with the following modifications. The
If a marker triggers a memory search, learning delay interval was shortened to 30 sec. A light above the
should still be possible; whereas if it only in- right-hand, white side arm door was turned on 2 sec before
the doors to the side arms were raised to allow the subject
creases attention to subsequent events, learning to make its choice response. Subjects in the marked group
should not occur. only were presented with a burst of white noise at 90 dB
immediately after their choice response. Subjects in the
Method unmarked group were treated in exactly the same way as
the marked subjects, except that noise was not presented
Subjects following their choice responses.
After every trial the maze was rotated on a random
The subjects were 16 male rats, experimentally naive basis, and the side arms were interchanged. All other pro-
and approximately 110 days old at the start of the ex- cedures were the same as in Experiment 1. All subjects
periment. Each subject was maintained at 80% of its free- were given two trials daily until each had received SO
feeding body weight and housed individually in a different trials.
room from the one in which the experiment was run.

Apparatus Results and Discussion

The layout and dimensions of the maze used in this
experiment were similar to those of the maze used in
Experiments 1 and 2. The present experiment, however,
required a number of modifications to the details of the
maze design.
The maze was constructed entirely of clear plastic that
was channeled to accept colored plastic sheets on the outer
walls so that the color of these walls could be easily changed.
The walls and floor of all compartments of the maze were
colored gray. The guillotine doors between the various
compartments of the maze were made of opaque gray
plastic that was left unpainted except for the doors from
the choice box to the two side arms. These two doors and
their surrounds were visually differentiated but only on
the sides that faced the choice box. The left-hand door
and surround were painted black; the right-hand door and
surround were painted white. A-small 6-W lamp was
mounted centrally on top of the door surround of the
white, right-hand door. The door surfaces facing the side
arms were left gray, as were the walls and floor surfaces
of the side arms. The two side arms themselves were in-
terchangeable.
The goal box in the present maze was made slightly
larger than that used previously. This enlargement ensured
that even without a barrier (see above) a rat would have
to completely enter the goal box to investigate the contents
of a dish placed against its far wall.
All other particulars of the apparatus were the same as
in Experiment 1.
Procedure
Pretraining. ^retraining was the same as in Experiment
1 except that each subject was first given a single 10-rain.
session of free exploration in the maze with all doors held
Figure 4 shows the percentage of correct
first responses over blocks of five trials for each
of the two treatment groups. There was some
increase in correct first responses over trials
in both groups, but the increase was much
greater for the marked group.
A two-way ANOVA was carried out on the
data presented in Figure 4 for the factors of
experimental treatment and trials. This anal-
ysis showed a significant effect of treatments,
F(\, 14) = 6.2, p < .05, and a significant effect
of trial blocks, F(9, 126) = 2.8, p < .01. In
addition, there was a significant interaction
between treatments and trials, F(9, 126) =
2.2, p < .05, confirming that the marked group
showed significantly more improvement over
trials than did the unmarked group.
The mean latency of choice responses for
subjects in the marked group was 5.16 sec
(SD = 4.6 sec), and for subjects in the un-
marked group, it was 3.4 sec (SD = 3.2 sec).
The difference between the group means was
not significant, f(14) = 1.83,/> > .05. Whereas
unmarked control subjects showed little learn-
ing, the performance of subjects in the marked
group improved substantially (and signifi-
cantly) over trials. Thus despite the fact that
no discriminative cues were present following
the marker, its presentation still enhanced
 MARKING AND DELAYED REWARD
learning substantially. A marker, it is thus clear,
does not simply focus attention on subsequent
events: It also produces learning about events
that precede it.
General Discussion
The results of the present experiments have
implications for our understanding of marking
at both empirical and theoretical levels. One
important finding is that a marker presented
immediately after a response can enhance
learning even if it is not repeated when food
is presented subsequently. Marking might thus
play a role in any delayed-reinforcement sit-
uation in which the response produces dis-
tinctive feedback. Consider, for example, the
standard practice in operant experiments of
presenting a click immediately after every bar
100-
Q.
00
50
LU
ce
QC
O
MARKED GROUP
UNMARKED GROUP
2 4 6 8 10
BLOCKS OF FIVE TRIALS
Figure 4. Percentage of correct first responses for Mocks
of five trials for each group in Experiment 3.
409
press. It has long been recognized that this use
of feedback stimuli facilitates shaping (Skinner,
1938), and this facilitation has traditionally
been attributed to secondary reinforcement
because the click is closely followed by food.
This analysis may well be correct, but the
present results raise the interesting possibility
that some or all of this facilitory effect could
be the result of a marking process: When the
click occurs it may help the rat to remember
the bar press as a functional unit, or alter-
natively, it may direct attention to the critical
segment of the response (i.e., the bar's down-
ward movement). Rather than directly
strengthening the response, in other words, the
click may be helping the rat to remember the
critical units in its execution so that when
food is presented subsequently the rat will be
better able to identify these units and thus
reproduce them. One important function of
feedback stimuli, in other words, may be to
mark the preceding response in memory.
A,second important finding was the dem-
onstration in Experiment 2 that a marker may
enhance learning about subsequent events as
well as preceding ones. The authors are not
aware of any precedent in the animal learning
literature, but a related finding in human
learning may be the effects of noise on paired-
associate learning. Hamilton, Hockey, and
Quinn (1972), for example, gave two groups
of subjects a list of paired associates to mem-
orize. Subjects whose initial exposure to the
list occurred in the presence of a continuous
85 dB white noise were found to recall sig-
nificantly more words during subsequent test-
ing (see also Craik & Blankenstein, 1975).
Together, these findings pose interesting
theoretical questions. The; most obvious is how
a marker can enhance learning about events
that follow as well as precede it. Our current
belief is that a salient stimulus elicits two com-
plementary coping strategies: (a) a backward
search through memory in order to identify
events that might have produced the marker
and (b) changes in attention to external events,
perhaps as part of the same search for causes.
The memory search would lead to recall of
preceding events, whereas changes in attention
would lead to better recall of those which
follow.
A second issue concerns the apparent con-
flict between the marked-after result and Wag-
410 THOMAS, LIEBERMAN, McINTOSH, AND RONALDSON
ner's rehearsal model (Wagner, 1978, 1981;
Wagner, Rudy, & Whitlow, 1973). According
to this model, an event must be rehearsed in
short-term memory before a permanent rep-
resentation of it can be formed in long-term
memory. Because rehearsal capacity is as-
sumed to be limited, presentation of a salient
stimulus will normally reduce rehearsal of
preceding events and thus reduce the likeli-
hood of their being remembered. Support for
this analysis comes from Wagner's own re-
search on classical conditioning and also from
operant experiments (e.g., Cook, 1980; Pearce
& Hall, 1978; Tranberg & Rilling, 1980). The
model predicts that a marker should hamper
learning because it will interfere with rehearsal
of the preceding response.
Marking is not, however, the only instance
in which a salient stimulus enhances rather
than disrupts learning about contemporaneous
events, and thus appears to contradict the re-
hearsal model: A parallel finding in taste-aver-
sion conditioning has been termed potentia-
tion. In a study by Palmerino, Rusiniak, and
Garcia (1980), for example, an odor was pre-
sented either by itself or in compound with a
taste and then followed after a delay by illness.
Not only did the presence of the taste not
interfere with conditioning to the odor, it sig-
nificantly enhanced it.
A possible theoretical reconciliation of en-
hancement and interference effects, however,
can be achieved with only a slight change in
the rehearsal model. Interference occurs, ac-
cording to the model, because a novel stimulus
reduces the amount of processing available for
contemporaneous events. Granting this, we
can allow for enhancement if we assume that
the reduced amount of available processing is
not necessarily redistributed equally over the
set of events concerned. In particular, if an
event with some special relationship to the
salient stimulus (e.g., contiguity, similarity)
receives a larger share of the reduced amount
of processing, then this event could actually
receive more processing rather than less fol-
lowing a salient stimulus.
Further theoretical speculation is probably
premature. The important point is that al-
though a salient stimulus may interfere with
learning about contemporaneous events under
some conditions, in others it may enhance it.
A pressing problem for future research will
be to clarify the conditions under which each
of these effects is obtained.
References
Brown, R., & Kulik, J. Flashbulb memories. Cognition,
1977, 5, 73-99.
Cook, R. G. Retroactive interference in pigeon short-term
memory by a reduction in ambient illumination. Journal
of Experimental Psychology: Animal Behavior Processes,
1980, 6, 326-338.
Craik, F. I. M, & Blankenstein, K. R. Psychophysiology
and human memory. In P. H. Venables & M. J. Christie
(Eds.), Research in psychophysiology. London: Wiley,
1975.
Cronin, P. B. Reinstatement of postresponse stimuli prior
to reward in delayed-reward discrimination learning by
pigeons, Animal Learning & Behavior, 1980, 8, 352-
358.
D'Amato, M. R., Fazzaro, J., & Etkin, M. Anticipatory
responding and avoidance discrimination as factors in
avoidance conditioning. Journal of Experimental Psy-
chology, 1968, 77, 41-47.
Dunnett, C. W. A multiple comparison procedure for
comparing several treatments with a control. Journal
of the American Statistical Association, 1955,50, 1096-
1121.
Easterbrook, J. A. The effect of emotion on cue utilization
and the organization of behavior. Psychological Review,
1959, 66, 183-201.
Grice, G. R. The relation of secondary reinforcement to
delayed reward in visual discrimination learning. Journal
of Experimental Psychology, 1948, 38, 1-16.
Hamilton, P., Hockey, G., & Quinn, J. Information se-
lection, arousal and memory. British Journal of Psy-
chology, 1972,65, 181-189.
Kamin, L. J. Predictability, surprise, attention and con-
ditioning. In B, A. Campbell & R. ;M. Church (Eds.),
Punishment and aversive behavior. New \brk: Appleton-
Century-Crofts, 1969.
Lett, B. T. Delayed reward learning: Disproof of the tra-
ditional theory. Learning and Motivation, 1973,4, 237-
246.
Lett, B. T. Long delay learning in the T-maze. Learning
and Motivation, 1975, 6, 80-90.
Lett, B. T. Long-delay learning: Implications for learning
and memory theory. In N. S. Sutherland (Ed.), Tutorial
essays in psychology: A guide to recent advances (Vol.
2). New York: Halstead Press, 1979.
Lieberman, D. A., Mclntosh, D. C., & Thomas, G. V.
Learning when reward is delayed: A marking hypothesis.
Journal of Experimental Psychology: Animal Behavior
Processes, 1979, 5, 224-242.
Muenzinger, K. F. Motivation in learning: I. Electric shock
for correct responses in the visual discrimination habit.
Journal of Comparative Psychology, 1934,17, 267-277.
Neuringer, A. J., & Chung, S. H. Quasi-reinforcement:
Control of responding by a percentage-reinforcement
schedule. Journal of the Experimental Analysis of Be-
havior, 1967, 10, 45-54.
Palmerino, C. C., Rusiniak, K. W., & Garcia, J. Flavor-
illness aversions: The peculiar roles of odor and taste
in memory for poison. Science, 1980, 208, 753-755.
Pearce, J. M., & Hall, G. Overshadowing the instrumental
 MARKING AND DELAYED REWARD 411

conditioning of a lever-press response by a more valid
predictor of the reinforcer. Journal of Experimental
Psychology: Animal Behavior Processes, 1978, 4:, 356-
367.
Roberts, W. A. Failure to replicate visual discrimination
learning with a 1-min delay of reward. Learning and
Motivation, 1976, 7, 313-325.
Skinner, B. F. The behavior of organisms. New York: Ap-
pleton-Century-Crofts, 1938.
Spear, N, E. The processing of memories: Forgetting and
retention. New York: Erlbaum, 1978.
Telegdy, G. A., & Cohen, J. S. Cue utilization and drive
level in albino rats. Journal of Comparative and Phys-
iological Psychology, 1971, 75, 248-253.
Tranberg, D. K,, & Rilling, M. Delay-interval illumination
changes interfere with pigeon short-term memory, /ow-
nal of the Experimental Analysis ofBehavior, 1980,33,
39-49.
Tulving, E., & Osier, S. Effectiveness of retrieval cues in
memory for words. Journal of Experimental Psychology,
1968, 77. 593-601.
Tulving, E., & Pearlstone, Z. Availability versus accessibility
of information in memory for words. Journal of Verbal
Learning and Verbal Behavior, 1966, 5, 381-391.
Wagner, A. R. Expectancies and the priming of STM. In
S. H. Hulse, H. Fowler, & W. K. Honig (Eds.), Cognitive
processes in animal behavior. Hillsdale, N.J.: Erlbaum,
1978.
Wagner, A. R. SOP: A model of automatic memory pro-
cessing in animal behavior. In N, E, Spear & R. R.
Miller (Eds.), Information processing in animals. Hills-
dale, N.J.: Erlbaum, 1981.
Wagner, A. R., Rudy, J. W, & Whitlow, J. W. Rehearsal
in animal conditioning. Journal of Experimental Psy-
chology, 1973, 97, 407-426. (Monograph)
Received May 17, 1982
Revision received February 2, 1983 •

Third Edition of the Publication Manual

APA has just published the third edition of the Publication Manual. This new
edition replaces the 1974 second edition of the Manual. The new Manual updates
APA policies and procedures and incorporates changes in editorial style and practice
since 1974. It amplifies and refines some parts of the second edition, reorganizes
other parts, and presents new material. (See the March issue of the American Psy-
chologist for more on the third edition.)
AH manuscripts to be published in the 1984 volumes of APA's journals will be
copy edited according to the third edition of the Manual. Therefore, manuscripts
being prepared now should be prepared according to the third edition. Beginning
in 1984, submitted manuscripts that depart significantly from third edition style will
be returned to authors for correction.
The third edition of the Publication Manual is available for $12 for members of
APA and $15 for nonmembers. Orders of $25 or less must be prepaid. A charge of
$1.50 per order is required for shipping and handling. To order the third edition,
write to the Order Department, APA, 1400 N. Uhle Street, Arlington, VA 22201.