See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/333568505

The effects of different mastery criteria on the skill maintenance of children

with developmental disabilities

Article  in  Journal of Applied Behavior Analysis · June 2019

DOI: 10.1002/jaba.580

CITATIONS READS

3 169

3 authors:

Sarah Richling Larry Williams

Auburn University University of Nevada, Reno
11 PUBLICATIONS   54 CITATIONS    83 PUBLICATIONS   518 CITATIONS   

SEE PROFILE SEE PROFILE

James Edward Carr

Behavior Analyst Certification Board
172 PUBLICATIONS   3,569 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Representation of Women in Behavior Analysis: An Empirical Investigation View project

Guided notes View project

All content following this page was uploaded by James Edward Carr on 15 July 2019.

The user has requested enhancement of the downloaded file.

JOURNAL OF APPLIED BEHAVIOR ANALYSIS 2019, 52, 701–717 NUMBER 3 (SUMMER)

The effects of different mastery criteria on the skill maintenance

of children with developmental disabilities
SARAH M. RICHLING AND W. LARRY WILLIAMS
UNIVERSITY OF NEVADA, RENO

JAMES E. CARR
BEHAVIOR ANALYST CERTIFICATION BOARD

The acquisition of skills by individuals with developmental disabilities typically includes the
attainment of a certain mastery criterion. We conducted a survey of practitioners who indicated
the most commonly used mastery criterion as 80% accuracy across three consecutive sessions.
Based on these results, we conducted a series of three experiments to evaluate the relation
between mastery criterion and subsequent skill maintenance with 4 individuals with various
developmental disabilities. Results suggest that 80% accuracy across three consecutive sessions
may be insufficient for producing maintenance in some cases.
Key words: accuracy, developmental disabilities, discrete-trial teaching, maintenance, mastery
criteria, skill acquisition

A number of recent applied behavior-analytic acquisition. Several clinical manuals recommend

studies have evaluated the effects of different
instructional variables during discrete-trial teach-
ing (DTT) on subsequent skill maintenance.
These variables have included intertrial intervals
(e.g., Cariveau, Kodak, & Campbell, 2016),
measurement intensity (e.g., Cummings & Carr,
2009), and task interspersal (e.g., Majdalany,
Wilder, Greif, Mathisen, & Saini, 2014), among
others. Another important variable that is clearly
related to skill maintenance is the specific acqui-
sition criterion, often termed the “mastery crite-
rion” (Luiselli, Russo, Christian, & Wilczynski,
2008). Luiselli et al. (2008) indicated that prac-
titioners generally have used percentage correct
or accuracy as the predominant measure of skill
This study was conducted by the first author, under
the supervision of the second author, in partial fulfillment
of the requirements for the PhD degree at University of
Nevada, Reno. We thank Amy Gonzales, Terralyn Tiffer,
and Kendall Temple for their assistance with data collec-
tion and Matthew Locey, Linda Hayes, MaryAnn Dem-
chak, and Stephen Rock for their helpful comments on
the dissertation.
Correspondence should be addressed to the first author
at Department of Psychology, 226 Thach Hall, Auburn
University, AL 36849-5214, email: smr0043@auburn.edu
doi: 10.1002/jaba.580
© 2019 Society for the Experimental Analysis of Behavior
701
80% or 90% accuracy for skill mastery
(e.g., Anderson, Taras, & O’Malley Cannon,
1996; Leaf & McEachin, 1999). Luiselli et al.
recommended clinicians require learners to meet
these accuracy percentages across multiple ses-
sions, days, or technicians. A survey of clinical
practices by Love, Carr, Almason, and Pet-
ursdottir (2009) reported that 62% of respon-
dents serving as program supervisors of early
and intensive behavioral intervention programs
indicated using percentage correct measures
across multiple sessions to determine the
achievement of mastery. The authors did not
provide detailed information about common
mastery criterion practices such as the specific
accuracy percentages used or the application
of these percentages across a certain number
of sessions.
The literature includes reports of a number
of mastery criteria such as 80% correct for a
single session (McCormack, Arnold-Saritepe, &
Elliffe, 2017), 80% correct across two or three
sessions (Najdowski et al., 2009), 90% correct
across two sessions (Toussaint, Kodak, &
Vladescu, 2016), 90% correct across three
702 SARAH M. RICHLING et al.
sessions (Wunderlich & Vollmer, 2017), 100%
for a single session (Grow, Kodak, & Clements,
2017), and 100% across three sessions (Belisle,
Dixon, Stanley, Munoz, & Daar, 2016). Mas-
tery criteria vary across multiple dimensions
including percentage correct, number of
sessions/trials, and additional variables such as
applying those criteria across multiple thera-
pists, to name a few. Given the ubiquity of the
use of mastery criteria in skill acquisition pro-
grams, there is a surprising lack of research
empirically validating different mastery criteria
on training outcomes. To date, only one publi-
shed study has explicitly evaluated the impact
of mastery criteria on skill maintenance for
individuals with developmental disabilities
(Fuller & Fienup, 2018).
Fuller and Fienup (2018) conducted a pre-
liminary analysis of the effects of the accuracy
level dimension of mastery on response mainte-
nance with three individuals diagnosed with
autism spectrum disorder. This experiment
involved teaching sight and spelling words until
various mastery criteria levels were met
(i.e., 50%, 80%, and 90% accuracy during a
single 20-trial session). Thereafter, we assessed
response maintenance at weekly intervals across
one month. The 90% accuracy criterion
resulted in the highest levels of maintained
responding for all three participants, whereas
the 80% accuracy criterion resulted in mainte-
nance performance below 70% for two of the
three participants. The 50% accuracy criterion
resulted in the lowest levels of response mainte-
nance. This preliminary research suggests that
an 80% mastery criterion across 20 trials may
be insufficient for producing acceptable levels
of maintained responding. However, given the
large number of variations in the dimensions of
mastery criteria, additional research is
warranted to determine which mastery criteria
practitioners utilize most frequently.
We may consider the mastery criterion a
practitioner establishes for a skill acquisition
program an independent variable, as various
criteria (e.g., 80% vs. 100% correct) might
result in differential maintenance outcomes. An
experimental evaluation of different mastery-
criterion arrangements is warranted, to provide
behavior analysts with empirical evidence to
guide their selection of mastery criteria. Thus,
the purpose of the current investigation was to
determine commonly used mastery criteria via
a survey of practitioners and subsequently to
evaluate the effects of different mastery criteria
on the skill maintenance of individuals with
developmental disabilities.
EXPERIMENT 1: A SURVEY OF
CLINICAL PRACTICES
To acquire more specific information about
mastery criteria used by practitioners in
advance of experimental research on the topic,
the first author conducted a survey of Board
Certified Behavior Analysts® (BCBAs®) who
supervise behavior-analytic services for individ-
uals with developmental disabilities. The mas-
tery criteria dimensions evaluated in
Experiments 2 through 4 were obtained from
the results of this survey.
Method
Participants and materials. Participants for
this phase of the experiment were recruited by
emailing invitations to participate to 15,677
individuals through the Behavior Analyst Certi-
fication Board® (BACB®) mass email service
who: (a) resided within the United States;
(b) were certified at the doctoral (BCBA-D®)
or master’s level (BCBA); (c) indicated a pri-
mary emphasis of work in behavior analysis,
behavior therapy, education, or positive behav-
ior support; (d) indicated a primary area of
work in intellectual disabilities, autism, special
education, or college education; and
(e) indicated their primary age group of clients
as children or adolescents. The initial email
invitation (see Supporting Information) indi-
cated that individuals who are involved in
 MASTERY CRITERIA
service delivery in the area of intellectual dis-
abilities and autism should complete the sur-
vey. The survey (see Supporting Information)
included 23 multiple-choice and fill-in-the-
blank questions about respondents’ certifica-
tion status and practice area, as well as their
clinical practices related to mastery criteria
(e.g., dependent variable used, number of ses-
sions applied, source of procedures). One hun-
dred and ninety-nine individuals completed
the survey. The section below describes the
results for each of the areas addressed by the
survey. Note that the percentages reported are
based on the number of responses for a spe-
cific question, which did not necessarily equal
the total number of survey respondents.
Results and Discussion
Five questions addressed general characteris-
tics of the respondent and the clients served.
Eighty-four percent (n = 168) of respondents
indicated that they were currently certified as a
BCBA®, and 16% (n = 31) of respondents
indicated that they were currently certified as a
BCBA-D®. Seventy-nine percent (n = 158) of
respondents indicated that the highest degree
held in behavior analysis was a master’s degree,
and 21% (n = 41) of respondents indicated that
the highest degree held in behavior analysis was
a doctoral degree. With respect to the year in
which the highest degree was earned, responses
ranged from 1985 to 2016 (n = 149). Sixty-six
percent (n = 98) of respondents indicated they
received their degree between the years of 2007
and 2016, 18% (n = 37) between the years of
1997 and 2006, and 9% (n = 14) between the
years of 1985 and 1994. The highest percent-
age (74%) of respondents (n = 147) indicated
their primary role in behavior analysis was that
of a practitioner, 16% (n = 32) indicated their
primary role was that of an administrator, 9%
(n = 18) indicated their primary role was that
of a faculty member, and 1% (n = 1) indicated
their primary role was that of a student.
703
Regarding the age distribution of the clients
served, results showed that across settings the
average proportion of clients served between
the ages of 0-6 years was 32%, between the
ages of 7-12 years was 28%, between the ages
of 13-17 years was 12%, and 18 years or older
was 5%.
Sixteen questions addressed clinical practices
related to the mastery criteria used within
respondents’ clinical practice. Survey results
indicated that 68% of respondents (n = 132 of
194) use mastery criteria based on a certain
percentage of correct trials (i.e., session based),
28% (n = 55 of 194 respondents) use a certain
number of correct trials in a row, and 4%
(n = 7 of 194 respondents) use a number of
responses per unit of time (rate). For individ-
uals who indicated they use a certain percent-
age of correct trials, 57% (n = 75 of
131 respondents) require the percentage correct
to be met across multiple sessions in combina-
tion with other variables (e.g., in the presence
of two or more technicians), and 35% (n = 46
of 131 respondents) of this sample require the
percentage correct to be met across multiple
sessions without other variables. Only 8% of
participants (n = 10 of 131) indicated they
apply a percentage of correct trials during one
session (i.e., a single trial block). Additional
results from respondents indicating the utiliza-
tion of a certain number of correct trials in a
row or a rate of response per unit of time have
not been included in this report.
Most notably, 54% of individuals (n = 70 of
130 respondents) reported using an 80% crite-
rion across one or more sessions, 28% (n = 36
of 130 respondents) reported using a 90% cri-
terion, and 7% (n = 9 of 130 respondents)
reported using a 100% criterion. In addition,
6% of individuals (n = 8 of 130 respondents)
reported using between an 81% and 89% crite-
rion across one or more sessions, 5% (n = 6 of
130 respondents) reported using between a
91% and 99% criterion, and 1% (n = 1 of
130 respondents) reported using below an 80%
704 SARAH M. RICHLING et al.
criterion. Respondents reported applying these
criteria across three (50%, n = 65 of 130
respondents), two (22%, n = 28 of 130 respon-
dents), more than four (13%, n = 17 of
130 respondents), or four (8%, n = 10 of
130 respondents) consecutive sessions. As pre-
viously mentioned, the remaining 8% of
respondents (n = 10 of 130) reported applying
these criteria during a single session. The survey
also asked participants to indicate the primary
information source contributing to the use of
the specific mastery criteria. The highest per-
centage (44%, n = 83 of 188 respondents)
specified their previous supervised experience as
the source of their mastery criteria. Twenty per-
cent (n = 37 of 188 respondents) identified
their employer policies or requirements, 16%
(n = 30 of 188 respondents) indicated their
graduate school, 10% (n = 19 of 188 respon-
dents) indicated continuing education experi-
ences (e.g., workshops), 9% (n = 16 of
188 respondents) indicated regulatory require-
ments (e.g., Individualized Education Plans),
and 2% (n = 3 of 188 respondents) indicated a
funding source.
Results from this phase of the experiment
confirm the aforementioned conjectures of vari-
ous authors (i.e., Anderson et al., 1996; Leaf &
McEachin, 1999; Luiselli et al., 2008), that cli-
nicians often use a percentage of correct
responding as a measure of skill mastery. The
findings of this phase of the experiment also
extend previous survey findings of Love et al.
(2009) by demonstrating that the highest per-
centage of BCBAs® providing services to indi-
viduals diagnosed with autism spectrum
disorder and intellectual disabilities report using
80% correct trials across three sessions as a
measure of mastery. In addition, results indicate
that the majority of participants have adopted
the use of this mastery criterion level from their
supervised experience. Finally, results show the
largest number of clients served is between
0 and 12 years of age. Given the common
use of the 80%-for-three-consecutive-sessions
mastery criterion with this pool of clients and
the lack of research evaluating the impact of
mastery criteria on skill maintenance, the fol-
lowing experiments were designed to empiri-
cally evaluate the effects of the most commonly
reported mastery criterion dimensions on skill
maintenance.
EXPERIMENT 2
We designed Experiment 2 to evaluate the
effects of an 80% mastery criterion across three
sessions on skill maintenance. We selected this
criterion as the data from Experiment 1 indi-
cated it was the most commonly reported mas-
tery criterion. We compared this criterion to
both a 60% mastery criterion across three ses-
sions and a 100% mastery criterion across three
sessions. We included the 60% and 100%
criteria to identify whether a parametric rela-
tion between mastery criteria and skill mainte-
nance existed.
Method
Participants and setting. Four children with
developmental disabilities who had normal
vision and hearing and were between the ages
of 6 and 9 years old participated in this experi-
ment. We selected participants from a compre-
hensive life-skills classroom for students with
mild to moderate support needs within a public
elementary school. All participants had passed
Level 6 on the Assessment of Basic Learning
Abilities-Revised (DeWiele, Martin, Martin,
Yu, & Thomson, 2011), indicating they dem-
onstrated discriminated responding toward a
combination of auditory and visual stimuli.
Evan was a 6-year-old boy diagnosed with
autism spectrum disorder who could respond
to multiple-step instructions, speak in short
sentences, exhibit all verbal operants, and
engage in simple conversational verbal
exchanges. He received a standard score of
51 (average range = 85-115) on the Peabody
Picture Vocabulary Test (PPVT; Dunn &
 MASTERY CRITERIA
Dunn, 1997). Sandy was a 7-year-old girl diag-
nosed with health impairment and intellectual
disability, who could respond to multiple-step
instructions, speak in short sentences, exhibit
all verbal operants, and engage in simple con-
versational verbal exchanges. She received a
standard score of 52 on the PPVT. Cyril was a
9-year-old boy diagnosed with health impair-
ment and intellectual disability who could
respond to multiple-step instructions, speak in
short sentences, exhibit all verbal operants, and
engage in conversational verbal exchanges. He
received a standard score of 66 on the PPVT.
Adam was a 6-year-old boy diagnosed with
health impairment, Williams Syndrome, and
intellectual disability who could respond to
multiple-step instructions, speak in complex
sentences, exhibit all verbal operants, and
engage in conversational verbal exchanges. He
received a standard score of 89 on the PPVT.
An experimenter conducted sessions in a 3.5
x 4.6 m therapy room at the school during nor-
mal school hours. The room contained a table,
chairs for the participant and the experimenter,
and materials relevant to the experiment. The
experimenter placed a video camera in the
room in an unobtrusive manner and videotaped
most sessions for subsequent data scoring. A
secondary observer (one of two trained assis-
tants) was present for some sessions to collect
interobserver agreement and treatment
integrity data.
Stimulus preference assessments. The experi-
menter provided contingent access to preferred
food or tangible items as a consequence for cor-
rect responses during training sessions. The
experimenter provided noncontingent access to
these items during baseline and weekly follow-
up probes. Prior to the experiment, the first
author asked the classroom teacher to identify
potential preferred food or tangible items. The
experimenter then conducted a single-array
multiple-stimulus-without-replacement prefer-
ence assessment (DeLeon & Iwata, 1996) prior
to each 10-trial session. Each assessment
705
included either seven edible items or seven tan-
gible items (this varied across participants). The
experimenter then used the first three stimuli
selected by each participant during these assess-
ments for that session. Prior to the onset of the
first trial, the experimenter asked the partici-
pant which item they would like to work for
from the array of the top three items. After
each consequence delivery following correct
responding, the therapist again asked the partic-
ipant to choose an item from the array of three.
The first author instructed the teacher to
restrict access to all items included in the assess-
ment on session days for the duration of the
experiment, as much as possible.
Response definitions and measurement. We col-
lected data on the percentage of discrete trials in
which the participant made a correct response
during each 10-trial session. Correct responses
included independent responses that occurred
within 3 s following the presentation of the rele-
vant discriminative stimulus. We only recorded
responses for trials during which a participant
made a response to the instruction within the
relevant response class. For example, we only
recorded a correct or incorrect response if the
child pointed to one of the three comparison
cards in the receptive identification program and
did so without prompting. If a response was not
made within 3 s, the trial would have been
repeated until a response was made. The pur-
pose of this protocol was to ensure that nonre-
sponses due to noncompliance were not
included in the dependent measure. The re-
presentation of trials due to nonresponding was
never required during the experiment.
Target acquisition program. The acquisition
program taught during this experiment was
receptive identification, an auditory–visual con-
ditional discrimination. We taught participants
three sets of three target stimuli. The stimuli
used for this program were nine 15.2 x
15.2 cm cards with color-printed pictures of
animals, plants, and food items (see Table 1)
on a white background. The first author and
706 SARAH M. RICHLING et al.
Table 1
Target Stimuli, Experiments 2-4
Stimulus Set Experiment 2
Willow, Magnolia, Succulent
Set 1
Set 2 Lemur, Walleye, Spookfish
Set 3 Armadillo, Gibbon, Wallaby
Non-Experimental Target Set 1 N/A
Non-Experimental Target Set 2 N/A
the classroom teacher collaboratively selected
target stimuli for Studies 2-4. The teacher
aided in identifying target words that included
sound blends and numbers of syllabi that all
participants had previously demonstrated the
ability to pronounce and respond to differen-
tially. In addition, the stimuli depicted things
and included words the classroom teacher indi-
cated she was unlikely to teach the participant
outside of the experimental sessions. For each
participant, each stimulus set was assigned to
one of the three mastery criteria (i.e., 100%,
80%, or 60% correct for three consecutive ses-
sions). We counterbalanced mastery criteria
across stimulus sets and across participants. We
used the same stimuli across all participants to
allow for counterbalancing of stimulus sets and
criterion levels. We assigned stimulus set 1 to
the 100%, 60%, 80%, and 80% criteria for
participants 1, 2, 3, and 4, respectively. We
assigned stimulus set 2 to the 60%, 80%,
100%, and 60% criteria for participants 1, 2,
3, and 4, respectively. We assigned stimulus set
3 to the 80%, 100%, 60%, and 100% criteria
for participants 1, 2, 3, and 4, respectively.
During a given session, the experimenter placed
three stimulus cards from one of the sets on
the table in front of the child. The experi-
menter then provided an instruction
(e.g., “Point to lemur”). The experimenter
recorded a correct response if the participant
touched the card as instructed. The experi-
menter randomly rotated the position of target
stimuli and corresponding instruction through-
out each 10-trial session. Training among the
Experiment 3 Experiment 4
Kiwi (animal), Ginger, Octagon Alaska, Geyser, %
Blowfish, Star Fruit, Rhombus Florida, Canyon, &
Sloth, Dragon Fruit, Spiral Texas, Rainforest, @
N/A Raccoon, Cockatoo, Fiddleheads
N/A Daffodil, Paw Paw, Wombat
three sets rotated randomly across sessions for
each participant.
Experimental design. We used a nonconcurrent
multiple-baseline design across participants with
an embedded modified alternating treatments
design to permit within-subject treatment com-
parisons and between-subject replications.
We evaluated the maintenance of skills fol-
lowing teaching until participants met a mas-
tery criterion of 80% correct responding for
three consecutive sessions as compared to
both a mastery criterion of 100% and 60%
correct responding for three consecutive
sessions. The experiment involved three con-
ditions: baseline, teaching, and weekly follow-up
probes. For each participant, we conducted
three to five training sessions per day, 3 to
5 days per week until the onset of weekly
follow-up probes.
Baseline. The experimenter did not provide
reinforcement for correct responses or prompt-
ing for incorrect responses. The experimenter
provided access to preferred stimuli for 15 s
(food items) or 30 s (tangible items) noncon-
tingently on a 1-min schedule. The purpose of
this protocol was to decrease the probability of
noncompliant behavior associated with escape
from demands.
Teaching. Teaching sessions were similar to
baseline sessions except the experimenter deliv-
ered praise on a continuous schedule for correct
responses (i.e., a fixed-ratio [FR] 1 schedule)
and 15 s (food items) or 30 s (tangible items)
of access to the preferred stimuli on a variable-
ratio (VR) 3 schedule. All participants used a
 MASTERY CRITERIA
token system for their normal classroom acqui-
sition tasks, which required five tokens to
exchange for preferred items/activities (i.e., an
FR-5 schedule). Given this history, we selected
a VR-3 schedule for tangible/food items and an
FR-1 schedule of praise because they likely
would be sufficiently dense, protect against sati-
ation, and would not disrupt compliance with
demands in the classroom setting. In addition,
incorrect responses resulted in the use of a
least-to-most prompting procedure that
included gestural, partial-physical, and full
physical prompts. No participants required pro-
mpts beyond the gestural level during the
experiment. Prompted responses did not result
in reinforcer delivery. Correct responses after a
prompt resulted in the experimenter saying
“okay” in a neutral tone. We conducted train-
ing sessions for a given set of stimuli until
responding met the designated mastery crite-
rion (i.e., 60%, 80%, or 100% for three con-
secutive sessions). After the participant met the
designated mastery criterion for each set,
the weekly follow-up probe phase began. Thus,
the experimenter conducted weekly follow-up
probes with some of the sets of stimuli while
some of the other sets were still in the teaching
phase.
Weekly follow-up probes. Weekly follow-up
probes were identical to baseline. These main-
tenance probes occurred at approximately one,
two, three, and four weeks following acquisi-
tion for each stimulus set. Each maintenance
probe consisted of a single 10-trial session.
Interobserver agreement and procedural integ-
rity. A second observer independently collected
data in situ or from video recordings of sessions.
We used these data to calculate interobserver
agreement (IOA) and procedural integrity
scores. We assessed IOA on a trial-by-trial basis
by dividing the number of trials in agreement
by the total number of trials and then multiply-
ing by 100%. We assessed IOA for at least 33%
of baseline sessions (M = 42%), 33% of teach-
ing sessions (M = 37%), and 33% of follow-up
707
sessions (M = 42%) for each participant. IOA
was 100% across all conditions for each
participant.
Procedural integrity and IOA on procedural
integrity were assessed for at least 33% of
teaching sessions (M = 37%) for each partici-
pant. To assess procedural integrity, the experi-
menter’s behavior was measured during each
trial by a second observer to assess whether
(a) the instruction was delivered correctly,
(b) the experimenter accurately recorded correct
or incorrect responses, and (c) the experimenter
followed the designated reinforcement schedule
or prompting hierarchy (if needed) immediately
following the participant’s response. The first
author assessed the procedural integrity score
based on the number of correct responses
observed divided by the total number of
responses. Procedural integrity was 100% for
each participant. The first author determined
the IOA on procedural integrity by comparing
the overall procedural integrity scores between
two independent observers for each session.
IOA on procedural integrity was 100% for each
participant.
Results and Discussion
Figure 1 displays the results of Experiment
2 for each participant. All participants achieved
the designated mastery criterion for all three
receptive identification task sets during the
teaching condition. For the 100% criterion,
response accuracy was at or above 80% correct
for all four participants across all weekly follow-
up sessions. For the 80% criterion, response
accuracy maintained near or slightly below
mastery performance levels for two of the four
participants (Sandy, third panel, and Adam,
bottom panel) during the weekly follow-up
condition. For one of the four participants
(Cyril, second panel), the 80% criterion
resulted in an immediate decrease in response
accuracy to an average of 47.5% correct across
weekly follow-up sessions. For one of the four
708 SARAH M. RICHLING et al.

Baseline Teaching Weekly Follow-up

100 100

80 80
100% Criterion
60 60
80% Criterion
40 40
60% Criterion
20 20
Evan
0 0
0 5 10 15 20
100 100

80 80

60 60
Percentage Correct

40 40

20 20
Cyril
0 0
10

20
15
0

100 100

80 80

60 60

40 40

20 20
Sandy
0 0
15
10

20
0

100 100

80 80

60 60

40 40

20 20
Adam
0 0
0 5 10 15 20 1 2 3 4
Sessions Week

Figure 1. Percentage of correct responses across participants during the 100%-, 80%-, and 60%-for-three-sessions
mastery criteria series in Experiment 2.

participants (Evan, top panel), the 80% crite-
rion resulted in an immediate drop to baseline
levels during the weekly follow-up condition.
This was similar for the 60% criterion, except
for one participant (Adam, bottom panel), for
whom the 60% criterion resulted in higher
levels of responding in follow-up than did the
80% criterion. In summary, although there was
a parametric effect of mastery criterion level on
maintenance, only skills taught using a 100%
mastery criterion maintained at greater than
80% accuracy.
 MASTERY CRITERIA
The findings with respect to the 80% mas-
tery criterion do not support the consistent use
of this criterion across all individuals when
teaching a receptive identification task to pro-
duce maintained responding. However, these
results are preliminary and warrant further
investigation to determine the extent to which
the observed patterns are replicable. For exam-
ple, it is possible the observed patterns are
unique to specific individuals’ performance on
a specific acquisition task. In addition, the
forced-choice selection arrangement in the cur-
rent experiment allows for chance responding
on individual trials. A nonchoice-based task
would allow for a more pure detection of differ-
ences in maintained responding across mastery
criteria. As such, we conducted the following
experiment, which involved teaching a different
type of acquisition task in a nonforced-choice
selection format.
EXPERIMENT 3
We designed Experiment 3 to systematically
replicate Experiment 2 and to extend the inves-
tigation to a second acquisition task. All other
procedures were identical to Experiment 2 (i.-
e., participants, settings, programmed conse-
quences, session structure, instructional
procedures, and experimental design).
Target Acquisition Program
The acquisition program taught during this
experiment was vocal tacting. Three sets of
three pictures were included in this curricular
program with each set being randomly assigned
to one of the three mastery criteria (i.e., 60%,
80%, or 100% for three consecutive sessions).
As in Experiment 2, we counterbalanced these
assignments across participants. The stimuli
used for this program were nine 15.2 x
15.2 cm cards with color-printed pictures of
animals, plants, food items, and geometric or
abstract shapes (see Table 1) on a white back-
ground. These stimuli were different from
709
those taught in Experiment 2 and depicted
things the teacher would not likely target out-
side of the experimental sessions, as confirmed
by the classroom teacher. During a given ses-
sion, the experimenter presented one of the
three cards from one of the sets to the child
and said, “What is this?” The experimenter
recorded a correct response if the participant
provided the appropriate vocal response
assigned to the presented picture. Incorrect
responses (including the participant saying, “I
don’t know”) resulted in the presentation of
prompts in a least-to-most procedure that
included a partial vocal prompt (i.e., the first
letter sound of each target word) and a full
vocal prompt.
Interobserver Agreement and Treatment
Integrity
The first author assessed IOA for at least
33% of baseline sessions (M = 42%), 40% of
acquisition sessions (M = 47%), and 42% of
follow-up sessions (M = 48%) for each partici-
pant. IOA was 100% across all conditions for
each participant. The first author assessed pro-
cedural integrity for at least 40% of acquisition
sessions (M = 47%) for each participant. Across
all participants, procedural integrity was 100%
and IOA for these data was 100%.
Results and Discussion
Figure 2 displays the results of Experiment
3 for each participant. All participants achieved
the designated mastery criterion for all three
vocal tacting task sets during the teaching con-
dition. With respect to the 100% criterion,
response accuracy was at or above 70% for all
four participants across all weekly follow-up
sessions. With respect to the 80% and 60%
criteria, results were fairly idiosyncratic across
participants. For Sandy (top panel), the 80%
criterion resulted in an immediate decrease to
zero-level responding across all weekly follow-
up sessions, undifferentiated from the 60%
710 SARAH M. RICHLING et al.

Baseline Teaching Weekly Follow-up

100 100

80 80
100% Criterion
60 60
80% Criterion
40 40 60% Criterion

20 20
Sa ndy
0 0
0 5 10 15 20 25 30

100 100

80 80

60 60
Percentage Correct

40 40

20 20
Evan
0 0
0

100 100

80 80

60 60

40 40

20 20
Cyril
0 0
10

15

20

25

30
0

100 100

80 80

60 60

40 40

20 20
Adam
0 0
0 5 10 15 20 25 30 1 2 3 4
Sessions Week

Figure 2. Percentage of correct responses across participants during the 100%-, 80%- and 60%- for-three-sessions
mastery criteria series in Experiment 3.

criterion results. For Evan (second panel), the
80% criterion resulted in an immediate
decrease to 30% during the weekly follow-up
condition and continued decreasing to zero for
the final two sessions. These levels fell below
those observed for the 60% criterion. For Cyril
(third panel), the 80% criterion resulted in
response accuracy that maintained near mastery
levels, on average, and above the levels observed
for the 60% criterion during the weekly follow-up
 MASTERY CRITERIA
condition. In addition, on average, follow up
responding was similar across the 100% and
80% criterion sets. Finally, for Adam (bottom
panel), the 80% criterion resulted in
maintained response accuracy of 80% at the
1-week follow-up sessions and then subse-
quently decreased to 60% across the final
three weekly follow-up sessions. These levels
were only slightly higher than those observed
for the 60% criterion.
The results of Experiment 3 replicate and
extend the findings of Experiment 2. These
data provide further evidence against the con-
sistent use of an 80% mastery criterion. Skill
maintenance followed similar patterns across a
second type of acquisition skill, and response
patterns did not appear to be unique to specific
participants across the two experiments. Again,
the results of the experiment are preliminary
and additional replications are necessary to
determining recommended mastery criteria.
Moreover, the question remains whether mas-
tery criterion needs be 100% to produce
acceptable levels of maintained responding, or
whether the second-most commonly used mas-
tery criterion (i.e., 90%) is sufficient. In order
to provide further evaluation of the 80% crite-
rion and a preliminary analysis of the 90% cri-
terion, we designed the following experiment.
EXPERIMENT 4
Experiment 4 was identical to Experiment
3 (i.e., participants, settings, programmed con-
sequences, session structure, experimental
design, and target acquisition program) except
that we evaluated a new mastery criterion
(i.e., 90%) in lieu of the 60% mastery crite-
rion, as well as four other procedural variations.
These variations (described in more detail
below) included: a) the inclusion of stimulus
targets identified by the classroom teacher as
targets the student would likely learn in future
school curricula, b) the inclusion of non-
experimentally targeted stimulus sets, c) an
711
increased number of sessions conducted per
day, and d) a single maintenance probe session.
Procedures
We evaluated the maintenance of skills fol-
lowing teaching until participants met a mas-
tery criterion of 80% accuracy for three
consecutive sessions compared to mastery
criteria of 90% and 100% accuracy for three
consecutive sessions. Again, vocal tacting was
taught; however, novel stimuli and target
responses were utilized. We included three tar-
get sets of three pictures in this curricular pro-
gram. We randomly assigned each set to the
three mastery criteria (i.e., 80%, 90%, or
100%). These stimuli differed from those used
in Experiments 2 and 3 in that they included
colored pictures of states and natural land for-
mations, and black and white symbols (see
Table 1). We included these stimuli because
the teacher identified them as more representa-
tive of stimuli commonly taught in a classroom
setting. The teacher reported these were not
stimuli currently being taught in any other con-
text but would likely be taught at some point
in future years.
In addition, Experiment 4 differed from the
previous experiments in that after a given set of
stimuli was mastered (per the designated crite-
rion), a new set of nonexperimental target stim-
uli (see Table 1) was introduced and taught
such that the same number of target stimuli
were always in acquisition simultaneously. This
procedural variation was included to control for
possible confounding effects of teaching fewer
total synchronous stimuli as participants mas-
tered previous stimulus sets. We did not apply
any mastery criteria to these two sets, and train-
ing continued regardless of performance.
Graphical displays do not include data from
these sets. Two participants exhibited response
accuracy levels that would have met the 80%
criterion across three sessions for one of the
two nonexperimental target sets each. However,
712 SARAH M. RICHLING et al.
each of those participants met this criterion on
the final teaching session. For the other partici-
pants and nonexperimental target sets,
responding remained below the lowest criterion
level.
This experiment also differed from the previ-
ous two experiments in that it included a
greater number of sessions per day to accom-
modate a short timeline prior to an extended
academic break. As such, each participant com-
pleted six to nine training sessions per day,
5 days per week. Finally, we only assessed
maintenance after 1 week following acquisition
for each set, to accommodate an extended aca-
demic break. However, because the first follow-up
assessment after teaching is the purest represen-
tation of maintenance before a history of extinc-
tion can affect responding, these data maintain
interpretive validity.
Interobserver Agreement and Procedural
Integrity
We assessed IOA for at least 33% of baseline
sessions (M = 53%), 33% of acquisition ses-
sions (M = 38%), and 40% of follow-up ses-
sions (M = 50%) for each participant. IOA was
100% across all conditions for each participant.
We assessed procedural integrity data for at
least 33% of acquisition sessions (M = 38%)
for each participant. Across all participants,
procedural integrity was 100% and IOA for
these data was 100%.
Results and Discussion
Figure 3 displays the results of Experiment
4 for each participant. All participants achieved
the designated mastery criterion for all three
vocal tacting sets during the teaching condition.
For the 100% criterion, response accuracy was
at or above 70% for all four participants during
the follow-up probe. For the 90% criterion,
response accuracy immediately fell to 0% for
three of the four participants (Cyril, second
panel; Adam, third panel; Sandy, bottom
panel). The 90% criterion resulted in a
decrease to 40% accuracy for the remaining
participant (Evan, top panel). Finally, the 80%
criterion resulted in an immediate decrease to
0% response accuracy for two participants
(Evan, top panel; Sandy, bottom panel). The
80% criterion resulted in an immediate
decrease in response accuracy to 20% during
follow-up for one of the four participants
(Cyril, second panel) and to 60% for one of
the four participants, (Adam, third panel).
Figure 4 displays performance on the first
follow-up probe session (i.e., 1 week after the
mastery criterion was met) for all participants
across Experiments 2 (top panel), 3 (middle
panel), and 4 (bottom panel). Across all 12 eval-
uations, the 100% criterion resulted in the
highest maintenance levels in all but one
instance (Sandy, top panel) and was at or above
70% accuracy (M = 86%). Across four compar-
isons (Experiment 4), the 90% criterion
resulted in maintenance at 10% accuracy, on
average. The 80% criterion resulted in variable
maintenance that ranged from 0% to 80%
accuracy (M = 37%). Across eight evaluations
(Experiments 2 and 3), the 60% criterion
resulted in maintenance that averaged 52.5%
accuracy.
These results replicate and extend the results
of Experiments 2 and 3, and provide further
evidence against the use of the 80% mastery
criterion and preliminary evidence against the
use of a 90% mastery criterion. The results of
Experiment 4 are particularly surprising because
for half of the participants, the 80% mastery
criterion resulted in better maintenance than
did the more stringent 90% criterion. How-
ever, in the current preparation, the difference
between obtaining 80% accuracy and 90%
accuracy was only one trial in a given session.
As such, the difference was marginal and spe-
cific patterns for two of only four participants
might be due to chance, although the same
could be said for the difference between the
90% criterion and the 100% criterion.
 MASTERY CRITERIA 713

Baseline Teaching 1 Week Follow-up

100 100

80 80
100% Criterion
60 60
90% Criterion
40 40
80% Criterion
20 20
Evan
0 0
0 5 10 15 20 25 30 35 40 45

100 100

80 80

60 60
Percentage Correct

40 40

20 20
Cyril
0 0
10

15

25

30

40

45
20

35
0

100 100

80 80

60 60

40 40

20 20
Adam
0 0
10

15

20

25

30

35

40

45
0

100 100

80 80

60 60

40 40

20 20
Sandy
0 0
0 5 10 15 20 25 30 35 40 45 1
Sessions Week

Figure 3. Percentage of correct responses across participants during the 100%-, 90%- and 80%- for-three-sessions
mastery criteria series in Experiment 4.

GENERAL DISCUSSION for three sessions) resulted in skill maintenance

We designed the current study to evaluate for children with developmental disabilities.
the extent to which the most commonly This study involved a comparison of this mas-
reported mastery criterion (i.e., 80% accuracy tery criterion with three other mastery criteria
714 SARAH M. RICHLING et al.

100
100% Criterion 80% Criterion 60% Criterion Study 2 average of 13 sessions (range, 4 to 22), the 80%
80 criterion required an average of seven sessions
60 (range, 3 to 16), and the 60% criterion required
40 an average of four sessions (range, 3 to 4). How-
20
ever, although the 80% mastery criterion pro-
0
duced a greater number of reinforcer
Evan Cyril Sandy Adam
presentations than the 60% criterion (as did the
Percentage Correct on First Follow-up

100
80
60
40
20
0
Evan Cyril
100
100% Criterion 90% Criterion
80
60
40
20
0
0 0
Evan Cyril
Figure 4. Percentage correct on the first follow-up
probe session (i.e., 1 week following meeting mastery cri-
terion) for all participants across Experiments 2-4.
(i.e., 60%, 90%, and 100% accuracy for three
sessions). We made comparisons across two
curricular programs and four children for a
total of 12 data sets. Maintenance following
teaching to 80% accuracy for three consecutive
sessions only occurred in approximately half of
the 12 evaluations.
These findings may suggest that the addi-
tional acquisition sessions required to meet a
100% mastery criterion produced higher levels
of maintenance than did the 60%, 80%, and
90% mastery criteria. That is, for 9 of the
12 evaluations, the 100% mastery criterion
resulted in the greatest number of teaching ses-
sions and provided a greater number of rein-
forcer presentations during acquisition. Across
all experiments and participants, the 100% mas-
tery criterion required an average of 14 sessions
(range, 4 to 33), the 90% criterion required an
Study 3
90% criterion relative to the 80% criterion), it
did not always result in superior performance
during follow-up probes.
The most obvious explanation for why the
0
100% mastery criterion resulted in the best
Sandy Adam maintenance is the absence of errors at the end
of the instructional period, which meant that
80% Criterion Study 4
the proper stimuli exerted discriminative control
over the target responses. For the remaining
criteria, it would be possible for the participant
to make an error on the first presentation of a
given stimulus, receive error correction, and then
0 0 0 perform accurately on subsequent presentations
Sandy Adam
of that particular stimulus. This could result in
the individual never responding accurately to
that stimulus in that absence of at least one
prompt for that trial block. This possibility may
provide support for the practice of requiring that
the response to the first presentation of a stimu-
lus be unprompted. In the present study, the
stimulus sets consisted of three target stimuli,
and as such, the first trial data may be insuffi-
cient for capturing this possibility. Rather, it
might be necessary to require an independent
response to the first presentation of each stimu-
lus. We did not design the current investigation
to evaluate the effects of the number of rein-
forcer presentations, independent first
trial/stimulus presentation requirements, or the
number of sessions across which participants
must demonstrate accuracy. Future researchers
should evaluate the extent to which mastery
criteria that require components such as accu-
racy on the first trial or accurate responding
across a greater number of sessions affect the
subsequent maintenance of responding.
Several procedural arrangements varied across
the experiments and the potential limitations of
 MASTERY CRITERIA
these variations warrant further discussion. A
potential limitation of Experiment 4 was the
inclusion of only a single follow-up probe.
Although the first maintenance probe is argu-
ably the purest demonstration of maintenance
prior to the onset of extended extinction
effects, there were several instances in Experi-
ments 2 and 3 in which performance levels
increased on subsequent probes. This pattern
may be indicative of an extinction-related phe-
nomenon (i.e., an increase in the previously
reinforced accurate response in the absence of
the contingent delivery of reinforcers). These
points warrant future research and discussion
regarding what is the most socially valid test of
maintenance within the constraints of empirical
research. In addition, in Experiment 4, investi-
gators conducted more sessions per day than in
Experiments 2 and 3. It is possible that error
correction in early sessions affected responding
in subsequent sessions conducted on the same
day. As such, participants would be more likely
to meet the mastery criteria on a single day
than across multiple days. Future researchers
should evaluate the extent to which the addi-
tion of a time dimension component to the
mastery criteria beyond just sessions affects the
maintenance of response accuracy for longer
durations of time.
On a similar note, all of the mastery criteria
examined in Experiment 4, including the
100% criterion, resulted in lower levels of
maintained responding than in Experiments
2 and 3. One potential explanation for this
outcome may be the inclusion of the non-
experimental target stimulus sets. The inclusion
of these additional sets resulted in teaching
considerably more targets overall. Future
researchers should evaluate the extent to which
the number of targets included in a massed-trial
teaching format affects skill acquisition and
subsequent maintenance of response accuracy.
In addition, it is possible that the inclusion
of these additional sets resulted in increased
probabilities of errors in the form of previously
715
reinforced responses. That is, the entire experi-
mental context may serve as a discriminative
stimulus, in the presence of which investigators
previously reinforced a larger number of
responses. In the current study, there were sev-
eral occasions in which a participant erred by
emitting a response that was a correct tact for a
stimulus in another set or emitted a blend of
tact words across various sets. Interestingly,
there were several instances in which a partici-
pant emitted a tact response in Experiments
3 and 4 identical to a verbal discriminative
stimulus provided by the trainer in the previous
receptive identification program in Experiment
2. The inclusion of the nonexperimental target
stimuli resembles how individuals may teach
skills in a natural environment. Once an indi-
vidual masters a particular skill, the teacher
likely adds a new target skill to the overall pro-
gramming rather than waiting until the individ-
ual has mastered all skills before introducing
new targets. Future empirical researchers utiliz-
ing similar comparative preparations should
consider the adoption of nonexperimental tar-
get sets as a way to more closely emulate typical
teaching situations.
It is worth noting several aspects of the cur-
rent study that might function as barriers to
the generality of findings to other contexts.
First, many of the survey respondents indicated
they often used an 80% mastery criterion in
combination with other variables including
training across two or more technicians, train-
ing across two or more environments, task
interspersal, and reinforcer schedule thinning.
The current study did not include these vari-
ables. Although respondents identified these
variables as strategies utilized to promote main-
tenance, several of those listed only address
generalization. For example, training across two
or more therapists or environments promotes
stimulus generalization rather than mainte-
nance of responding. Further, the use of task
interspersal for the promotion of maintenance
has equivocal empirical support (Rapp &
716 SARAH M. RICHLING et al.
Gunby, 2016). However, there is conceptual
and empirical support for reinforcer schedule
thinning as a means for promoting mainte-
nance (e.g., Heinicke et al., 2016). Future
researchers should compare various mastery
criteria involving combinations of accuracy
levels, numbers of sessions, training days, and
other variables in order to determine the most
efficient and effective strategies for producing
optimal training outcomes.
Second, the current study only evaluated
mastery criteria based on a percentage of cor-
rect responses across sessions. Some practi-
tioners use a trial-based teaching approach,
which requires a trial-based mastery criterion
(e.g., 10 consecutive correct trials). Future
researchers should evaluate mastery criteria in
these arrangements as well.
The current study is one of the first to experi-
mentally evaluate commonly used mastery
criteria for skill acquisition. The evaluation
included four children with varying diagnoses,
two curricular program areas, and four mastery
criterion levels. Additional research is necessary
to extend this line with other practice-relevant
variables, such as populations, settings, skill
types, supplemental maintenance interventions,
an so on. Such activity would enable behavior
analysts to make programming decisions based
on evidence rather than on lore. In the mean-
time, however, results from the current investi-
gation, albeit preliminary, suggest that the 80%
and 90% mastery criteria utilized by a majority
of behavior analysts may not be sufficient to
promote skill maintenance for some individuals.
REFERENCES
Anderson, S., Taras, M., & O’Malley Cannon, B. (1996).
Teaching new skills to children with autism. In
C. Maurice, G. Green, & S. C. Luce (Eds.), Behav-
ioral intervention for young children with autism: A
manual for parents and professionals (pp. 181–194).
Austin, TX: PRO-ED.
Belisle, J., Dixon, M. R., Stanley, C. R., Munoz, B., &
Daar, J. H. (2016). Teaching foundational
perspective-taking skills to children with autism using
the PEAK-T curriculum: Single-reversal “I-you” deic-
tic frames. Journal of Applied Behavior Analysis, 49,
965-969. https://doi.org/10.1002/jaba.324
Cariveau, T., Kodak, T., & Campbell, V. (2016). The
effects of intertrial interval and instructional format
on skill acquisition and maintenance for children
with autism spectrum disorders. Journal of Applied
Behavior Analysis, 49, 809-825. https://doi.org/10.
1002/jaba.322
Cummings, A. R., & Carr, J. E. (2009). Evaluating pro-
gress in behavioral programs for children with autism
spectrum disorders via continuous and discontinuous
measurement. Journal of Applied Behavior Analysis,
42, 57-71. https://doi.org/10.1901/jaba.2009.42-57
DeLeon, I. G., & Iwata, B. A. (1996). Evaluation of a
multiple-stimulus presentation format for assessing
reinforcer preferences. Journal of Applied Behavior
Analysis, 29, 519-533. https://doi.org/10.1901/jaba.
1996.29-519
DeWiele, L., Martin, G., Martin, T. L., Yu, C. T., &
Thomson, K. (2011). The Kerr-Meyerson assessment of
basic learning abilities revised: A self-instructional man-
ual (2nd Ed.). St. Amant Research Centre, Winnipeg,
MB, Canada. Retrieved from http://stamant.ca/
research/abla
Dunn, L. M., & Dunn, L. M. (1997). Peabody picture
vocabulary test, third edition. Circle Pines, MN: Ameri-
can Guidance Service.
Fuller, J. L., & Fienup, D. M. (2018). A preliminary
analysis of mastery criterion level: Effects on response
maintenance. Behavior Analysis in Practice, 11, 1-8.
https://doi.org/10.1007/s40617-0201-0
Grow, L. L., Kodak, T., & Clements, A. (2017). An eval-
uation of instructive feedback to teach play behavior
to a child with autism spectrum disorder. Behavior
Analysis in Practice, 10, 313-317. https://doi.org/10.
1007/s40617-016-0153-9
Heinicke, M. R., Carr, J. E., Pence, S. T., Zias, D. R.,
Valentino, A. L., & Falligant, J. M. (2016). Assessing
the efficacy of pictorial preference assessments for
children with developmental disabilities. Journal of
Applied Behavior Analysis, 49, 848-868. https://doi.
org/10.1002/jaba.342
Leaf, R., & McEachin, J. (Eds.) (1999). A work in pro-
gress: Behavior management strategies and a curriculum
for intensive behavioral treatment of autism. New York:
DRL Books.Love, J. R., Carr, J. E., Almason,
S. M., & Petursdottir, A. I. (2009). Early and inten-
sive behavioral intervention for autism: A survey of
clinical practices. Research in Autism Spectrum Disor-
ders, 3, 421-428. doi:https://doi.org/10.1016/j.rasd.
2008.08.008
Luiselli, J. K., Russo, D. C., Christian, W. P., &
Wilczynski, S. P. (2008). Skill acquisition, direct
instruction, and educational curricula. In
J. K. Luiselli (Ed.), Effective practices for children with
autism (p. 196). New York: Oxford University Press.
 MASTERY CRITERIA
Majdalany, L. M., Wilder, D. A., Greif, A.,
Mathisen, D., & Saini, V. (2014). Comparing
massed-trial instruction, distributed-trial instruction,
and task interspersal to teach tacts to children with
autism spectrum disorders. Journal of Applied Behav-
ior Analysis, 47, 657-662. https://doi.org/10.1002/
jaba.149
McCormack, J., Arnold-Saritepe, A., & Elliffe, D. (2017).
The differential outcomes effect in children with
autism. Behavioral Interventions, 32, 357-369. https://
doi.org/10.1002/bin.1489
Najdowski, A. C., Chlingaryan, V., Bergstrom, R.,
Granpeesheh, D., Balasanyan, S., Aguilar, B., &
Tarbox, J. (2009). Comparison of data-collection
methods in a behavioral intervention program for
children with pervasive developmental disorders: A
replication. Journal of Applied Behavior Analysis, 42,
827-832. https://doi.org/10.1901/jaba.2009.41-827
Rapp, J. T., & Gunby, K. (2016). Task interspersal for
individuals with autism and other neurodevelopmental
disorders. Journal of Applied Behavior Analysis, 49, 730-
734. https://doi.org/10.1002/jaba.319
View publication stats
717
Toussaint, K. A., Kodak, T., & Vladescu, J. C. (2016).
An evaluation of choice on instructional efficacy and
individual preferences among children with autism.
Journal of Applied Behavior Analysis, 49, 170-175.
https://doi.org/10.1002/jaba.263
Wunderlich, K. L., & Vollmer, T. R. (2017). Effects of
serial and concurrent training on receptive identifica-
tion tasks: A systemic replication. Journal of Applied
Behavior Analysis, 50, 641-652. https://doi.org/10.
1002/jaba.401
Received October 26, 2017
Final acceptance April 3, 2019
Action Editor, Jeanne Donaldson
SUPPORTING INFORMATION
Additional Supporting Information may be
found in the online version of this article at the
publisher’s website.