Training and Education in Professional Psychology

© 2018 American Psychological Association 2018, Vol. 12, No. 4, 245–261

1931-3918/18/$12.00 http://dx.doi.org/10.1037/tep0000208

The Science of Training III: Supervision, Competency, and

Internship Training
Jennifer L. Callahan and C. Edward Watkins Jr.
University of North Texas

In this third installment of our review on the science of training, we examine empirical articles associated
with two pan-training areas, supervision and competency, before considering the capstone predoctoral
training experience of internship. Critical issues considered include supervision effectiveness, differences
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

and diversity in supervision, competency assessment, competency problems, internship match, and
This document is copyrighted by the American Psychological Association or one of its allied publishers.

internship training. We offer conclusions and discuss intersecting implications across these areas (e.g.,
changes to the internship application to reflect norm-referenced assessments of competency by super-
visors). We reiterate our call for a decrease in tribalistic rhetoric and a sharpened focus on curating an
evidence-based approach to training. To that end, we advocate for greater stewardship concerning the
science of training. We encourage directors of health services psychology training programs to dissem-
inate program evaluation efforts into the peer-reviewed literature. We also encourage accrediting bodies
to more clearly ground regulations in an empirically derived evidence-base. We conclude by proposing
a recurrent conference designed to provide (a) pragmatic assistance to directors in transforming program
evaluations into submission ready articles for dissemination into the peer-reviewed literature, (b) a venue
for establishing collaborative research that builds on data accumulated across programs, and (c)
opportunities for learning necessary methodological or analytic skills.

Keywords: evidence-based training, supervision, internship, competency, doctoral psychology

This is the concluding installment in our series on the science of
training. We review empirical literature focused on two pan-training
areas associated with doctoral training in health services psychology:
supervision and competency. We close the series with a review of
evidence associated with the capstone training experience of internship.
Supervision
Research on supervision covers a fairly broad swath of issues
including (a) supervision effectiveness, (b) supervisee nondisclosure
JENNIFER L. CALLAHAN, PhD, ABPP, earned her doctoral degree in
clinical psychology from the University of Wisconsin–Milwaukee and
completed her internship and postdoctoral training at Yale University
School of Medicine. She holds board certification in clinical psychology
and is a fellow of Division 29 (Psychotherapy) of the American Psycho-
logical Association. She is currently a professor and the director of clinical
training for the clinical psychology program in the Department of Psychol-
ogy at the University of North Texas, where she directs the Evidence-
Based Training and Competencies Research Lab.
C. EDWARD WATKINS Jr., PhD, is professor of psychology at the De-
partment of Psychology, University of North Texas. His primary profes-
sional interests focus on psychotherapy supervision and psychoanalytic
theory, practice, and research. He is editor of the Handbook of Psycho-
therapy Supervision and coeditor of The Wiley International Handbook of
Clinical Supervision. He is a Fellow of Divisions 29 (Psychotherapy) and
17 (Counseling Psychology) of the American Psychological Association.
CORRESPONDENCE CONCERNING THIS ARTICLE should be addressed to
Jennifer L. Callahan, Department of Psychology, University of North
Texas, 1155 Union Circle 311280, Denton, TX 76205. E-mail: jennifer
.callahan@unt.edu
and unhelpful supervision, (c) individual differences and diversity,
and (d) group supervision. Each of these will be reviewed in turn.1
Does Good Supervision of Trainees Lead to Good
Outcomes Among Their Clients?
An increasingly pressing research and practical issue facing super-
vision is whether or not it is effective (Bernard & Goodyear, 2014).2
Research suggests supervision can have a positive impact on various
samples of supervisees (e.g., in terms of skills acquisition and usage,
treatment knowledge, self-awareness, self-efficacy, and working alli-
ance; Beutler & Kendall, 1995; Goodyear & Guzzardo, 2000; Hollo-
way, 2012; Holloway & Neufeldt, 1995; Inman & Ladany, 2008;
Ladany & Inman, 2011; Wheeler & Richards, 2007), but the effect of
supervision on client outcomes is thought to be the raison d=etre for
the use of supervision as a training mechanism (Rousmaniere, Swift,
Babins-Wagner, Whipple, & Berzins, 2016).
It is perhaps somewhat surprising then that only recently has the
effect of supervision on clients been examined empirically. The
earliest study of this question was conducted in Australia (Bam-
bling, King, Raue, Schweitzer, & Lambert, 2006). That study
provided the first empirical demonstration located in the extant
1
See Callahan and Watkins (2018a) for an overview of the methodology
used in accomplishing this review.
2
In this section, we focus on direct supervision. Outcomes associated
with metasupervision (advanced trainees supervising more novice trainees)
have been rarely examined, but it appears that metasupervision can be
accomplished safely. The primary risk to clients appears to be attenuated
maintenance of gains posttreatment (Keenan-Miller & Corbett, 2015).

245
 246 CALLAHAN AND WATKINS
literature that supervisors can exert a significant effect on client
outcomes. Rather than sample doctoral trainees, that study re-
cruited volunteer therapists from the community who, on average,
were nearly 20 years older than the average doctoral student in
health services psychology and carried years of experience. Fur-
ther, only 5% of supervisors were psychologists. Despite these
differences from our target sample, we mention this study as it
represents seminal work. Bambling and colleagues assigned clients
to one of three supervision conditions: no supervision of their
treatment; process supervision; or clinical skills supervision. Ther-
apist adherence to treatment was intact and did not differ across
conditions. However, the working alliance, rate of attrition, and
symptom improvement were all found to be significantly better
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
among those clients assigned to a supervision condition. The
This document is copyrighted by the American Psychological Association or one of its allied publishers.
supervision conditions did not significantly differ from one an-
other on any of the study metrics.
Within a few years of that study, evidence specific to the target
sample of doctoral trainees in health services psychology began to
emerge. The first empirical documentation of a moderate client
outcome effect by supervisors in a psychology training clinic
appeared nearly 10 years ago (Callahan, Almstrom, Swift, Borja,
& Heath, 2009). In that study, cross-tabulation of supervisors with
client treatment outcome categories (deteriorated; no reliable
change; improved but not recovered; recovered) revealed a Cram-
er’s V of .46. After controlling for initial symptom severity and
therapist attributes, 16.4% of the variance in change scores on a
measure of symptom distress from first to last session was ac-
counted for by supervisors.
Subsequently, a larger sample in a different training clinic
replicated those findings (Wrape, Callahan, Ruggero, & Watkins,
2015). A multinomial logistic regression (that covaried initial
symptom distress scores and entered trainee clinician and super-
visor as predictors) again revealed a moderate effect on client
outcomes (Cox and Snell ⫽ .63; Nagelkerke ⫽ .74; McFadden ⫽
.51). Entry of the supervisor into the equation resulted in a 12.5%
improvement in client outcome classification accuracy. That study
also offered new information on two previously unstudied super-
visor variables: faculty status and length of time since supervisor
degree attainment. With respect to faculty status, the authors
reported that there were no significant differences in client out-
comes as a function of whether the supervisor was classified as an
adjunct versus tenure-track faculty member. However, length of
time since the supervisor’s degree attainment was found to be
significant, improving accuracy in classification of client outcomes
by 8% (Cox and Snell ⫽ .20; Nagelkerke ⫽ .23; McFadden ⫽
.11). The authors speculated that increased emphasis in training on
supervision as a specific competency and/or increasing use of
evidence-based practice may have contributed to the finding.
While promising, future research is needed to determine
whether the above findings are generalizable. This is especially
important because the effect of supervisors has not been demon-
strated in other samples (mental health nurses; White & Winstan-
ley, 2010) or mixed samples with only a minority of the target
population and supervisors represented (Rousmaniere et al., 2016).
Importantly, research in this nascent area appears to be vulnerable
to methodological weaknesses (e.g., not using therapists with
similar training backgrounds, not using psychometrically sound
measures). Future researchers are encouraged to review Freitas
(2002) for an overview of methodological shortcomings that can
be expected prior to conducting studies in this area, but one that is
not mentioned in that review is worth specifically highlighting
here.
As noted in the second installment of this series on the science
of training, the most common outcome among clients being treated
in psychology training clinics is one of no reliable change (Calla-
han & Watkins, 2018c). It is difficult to parse the specific variance
associated with supervisors when there is very little variance in
client outcomes available to deconstruct. For example, Rous-
maniere and colleagues (2016) reported ⬍1% of variance in client
outcomes was associated with supervisors in their sample of train-
ees and supervisors from assorted mental health disciplines. Yet,
for each of the 23 supervisors in that study, the modal client
outcome under their care was no reliable change. Although the
sample was quite large, the variance was fairly low. From that
perspective, prospects seem dim for additional research into the
association of supervision to outcomes in training clinics, where
modal outcomes are widely reported to be reflective of no reliable
change (e.g., Callahan et al., 2009; Callahan, Gustafson, et al.,
2014; Callahan & Hynan, 2005; Nyman, Nafziger, & Smith, 2010;
Okiishi et al., 2006; Rieck & Callahan, 2013; Shimokawa, Lam-
bert, & Smart, 2010). One pathway for potentially productive
exploration in future research may be to focus on identifying
exceptional supervisors and studying them, and their effects, more
closely (coined super-supervisors by Rousmaniere and colleagues,
2016, p. 197).
Exceptional supervisors maximize the variance within a given
sample of supervisors, potentially overcoming limited client out-
come variance that may be apparent at the aggregate level of the
clinic. Review of the results presented in the Callahan and col-
leagues (2009) study is consistent with that description. Client
outcomes, at the aggregate level, indicated no reliable change as
the modal client outcome in that training clinic, but classification
of outcomes at the level of supervisors revealed some exceptional
supervisors with highly (75–100%) positive client outcomes who
differed from less effective supervisors (⬍25% positive out-
comes). In contrast, Rousmaniere and colleagues (2016) findings
revealed no significant variability at the supervisor level. Despite
a large number of supervisors, good design, and careful analyses,
there were no exceptional supervisors within that sample. Taken
together, a reasonable hypothesis from the small body of existing
evidence is that supervision may exert an effect on client outcomes
in the range of ⬍1% to 16%, depending, at least in part, upon
whether the supervisor is a “super-supervisor.” Future research
into this tentative hypothesis is clearly needed.
Some support for the hypothesized variability in supervisor
competency may be gleaned from observations that competency
among doctoral psychologists in the provision of supervision ap-
pears to emerge over time (Milne & James, 2002). As described by
Milne and James, early competency development in supervision
was characterized by frequent use of listening (40%). As supervi-
sors’ competency improved, they continued to engage in listening
behaviors, but at a diminishing rate (37%, 32%, and 24%). As they
became increasingly competent, supervisors evidenced increasing
rates of guided experiential learning techniques (e.g., role-plays,
behavioral exercises, modeling) during supervision (7%, 14%,
19%, and 29%). As a further suggestion for developing compe-
tency, novice supervisors may benefit from training that prepares
them for how to handle incidents that trainees often describe as
 SCIENCE OF TRAINING III
major turning points within supervision: incidents involving the
supervisory relationship, trainee competence, emotional aware-
ness, and autonomy (Ellis, 2006).
At first glance, the observation of greater supervisor compe-
tency progression over time by Milne and James may seem at odds
with the finding from Wrape and colleagues (2015) that recency of
the supervisors’ degree attainment was associated with their effec-
tiveness. However, integrating the findings across the two studies
may suggest a testable hypothesis for future research. Recently
degreed supervisors may have been more likely to have had
specific training to develop their competencies in supervision
during their training years.
In a fairly recent study, additional hypotheses regarding possible
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
qualities of exceptional supervisors emerged. Rieck, Callahan, and
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Watkins (2015) reported more positive client outcomes when
relatively low supervisor agreeableness was present. The authors
suggested that their finding may be an indirect reflection of the
value of constructive criticism during supervision. The authors
also noted the importance of a strong alliance between supervisor
and trainee, speculating that effective supervisors may offer con-
structive criticism successfully because it falls within the safe
confines of a strong alliance. Perhaps supporting that speculation
of a relationship between constructive criticism and supervisory
alliance, a different study reported that satisfaction with supervi-
sion was improved when corrective feedback, in the form of
routine client outcome data, was included in supervision (Grossl,
Reese, Norsworthy, & Hopkins, 2014). Additionally, in a study
reported by Reese and colleagues (2009), trainees who were as-
signed to a supervision with client outcome feedback condition
evidenced significantly better client outcomes than did trainees
assigned to supervision without client outcome feedback. Unfor-
tunately, although the majority of training clinics report use of
routine client outcome monitoring to improve both supervision and
client outcomes, a significant minority of clinics and internship
sites do not yet offer this training due to lack of resources or
supervisor barriers (Mours, Campbell, Gathercoal, & Peterson,
2009; Peterson & Fagan, 2017). Among those sites that do use
outcome tracking, the use of a database for storing the accumulated
data is quite common and may serve as a repository for needed
research (Mours et al., 2009).
Supervisee Nondisclosure and Unhelpful Supervision
Improvements in the emotional bond between trainees and su-
pervisors have been found to be associated with greater trainee
supervision satisfaction (Ladany, Ellis, & Friedlander, 1999). But,
unfortunately, supervisors are not universally helpful. Supervision
can be experienced as hindering, unhelpful, or even harmful to a
trainee’s development as an emerging professional, weaken the
supervisory relationship, and/or negatively impact the trainee’s
work with clients (Enyedy et al., 2003; Gray, Ladany, Walker, &
Ancis, 2001; Henry, Schacht, Strupp, Butler, & Binder, 1993).
Nearly 40% of trainees reported intentionally withholding details
from supervision regarding clinical errors in session (Yourman &
Farber, 1996). In a more recent study that spanned early practicum
thru internship level trainees, 84% of trainees acknowledged at
least one instance of nondisclosure of salient clinical, personal, or
supervision-related issues in their most recent supervision session,
with an average of 2.68 nondisclosures per supervision session
247
(Mehr, Ladany, & Caskie, 2015). The most common reason for
nondisclosure pertained to a negative supervision experience.
Stronger supervisory working alliance has been found to lead to
greater willingness to disclose, while trainee anxiety and self-
efficacy were reported to be unrelated (Mehr et al., 2015). Simi-
larly, another study found that strong supervisory attachment was
facilitative of trainee disclosures (Gunn & Pistole, 2012). One
possible pathway for increasing the likelihood of a positive alli-
ance may be to allow trainees input into their supervisor assign-
ment. Satisfaction with supervision has been found to be improved
when trainees are able to choose their supervisor (Huber, Sauer,
Mrdjenovich, & Gugiu, 2010).
In a somewhat dated study, a slight majority of trainees reported
encountering a perceived ethical violation by their supervisor
(Ladany, Lehrman-Waterman, Molinaro, & Wolgast, 1999). In a
more recent study, nearly a third of students aware of faculty
ethical misconduct with the most frequent event involving multiple
relationships/boundary issues, followed by research integrity/aca-
demic dishonesty, disrespectful treatment, and/or lack of mentor-
ship/supervision competency (January, Meyerson, Reddy, Do-
cherty, & Klonoff, 2014). Perhaps not surprisingly, supervisor
unethical behaviors have been found to adversely impact the
supervisor working alliance as well as satisfaction with supervi-
sion.
In the absence of a negative supervision event, some trainees
may still be resistant to supervision and supervisory feedback.
Although having trainees create and self-critique transcripts of
their sessions is time consuming, trainees find the process to be
associated with additive feedback beyond that of the supervisor,
easier acceptance of supervisor feedback, and improved clinical
skills (Sobell, Manor, Sobell, & Dum, 2008). Future research to
explore whether self-critiques might be an effective training inter-
vention (e.g., as part of a remediation plan) for students who are
resistant to supervision is encouraged.
Individual differences and diversity. Individual differences
and diversity issues pose yet another reason for supervisor–
supervisee discord or dissatisfaction. Gatmon and colleagues
(2001) reported that few discussions of cultural (ethnicity, gender,
sexual orientation) variables within the context of supervision
were occurring. Within their sample, they found that when discus-
sions did occur they exerted positive effects on both trainee satis-
faction and the supervisory working alliance. The alliance-
affecting, trainee-satisfying impact of such discussions was more
recently supported by Soheilian, Inman, Klinger, Isenberg, and
Kulp (2014) as well as Phillips, Parent, Dozier, and Jackson
(2017). Further, Jernigan, Green, Helms, Perez-Gualdron, and
Henze (2010) found that racial identity status was associated with
perception of diversity discussions.3
Based on the findings from a large, nationwide study of trainee
perceptions of multicultural training and program climate, forma-
tion of a program-specific diversity committee may facilitate
greater comfort with such discussions and improve the overall
climate throughout the program (Tucker, Gregus, Seivert, Stevens,
3
For more information on how multicultural coursework may impact
racial identity status among trainees and result in competency gains that
may be preparatory, see our review of empirical doctoral curriculum
training (Callahan & Watkins, 2018b).
 248 CALLAHAN AND WATKINS
& Callahan, 2018). Notably, Black or African American and
Hispanic/Latinx trainees evidenced significantly lower scores on a
standardized measure assessing the multicultural environment of
their doctoral program in health services psychology. Further,
trainees from underrepresented groups were found to be three
times more likely than other trainees to report that faculty multi-
cultural competency was in need of improvement and twice as
likely to report that faculty needed to demonstrate increased value
of diverse students and ideas. These findings, when considered in
tandem with the findings in our review of individual differences as
they pertain to doctoral program admissions and attrition (Callahan
& Watkins, 2018b) strongly indicate that programs need to provide
faculty with competency fostering professional development op-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
portunities and clearly articulate a broad commitment to diverse
This document is copyrighted by the American Psychological Association or one of its allied publishers.
peoples and ideas.
With regard to differences based on gender identification, male
and female supervisors have been found to differ in what they
consider appropriate boundaries with trainees; male supervisors
are more likely than female supervisors to consider some behav-
iors appropriate (i.e., interactions outside of supervision, personal
disclosures, sharing of supervision content; Heru, Strong, Price, &
Recupero, 2006). Unfortunately, Walker, Ladany, and Pate-
Carolan (2007) found that about half of gender-related events in
psychotherapy supervision were perceived negatively by female
trainees. Notably, supervision events perceived as gender discrim-
inating significantly adversely impact the supervisory alliance as
well as perceptions of the supervisor’s gender-related multicultural
competency (Bertsch et al., 2014). A weakness in this body of
research, to date, is the largely gender binary construction in the
presentation of results. Future research is encouraged to more
carefully consider a more fluid range of gender identification.
Group supervision. The helpfulness of group supervision in
practicum, from the perspective of trainees, clusters into five areas:
supervisor impact, specific instruction, self-understanding, support
and safety, and peer impact. These areas appear to differ from one
another along two dimensions: perceiving supervisor versus peer
impact and acquiring objective versus self-knowledge (Carter,
Enyedy, Goodyear, Arcinue, & Puri, 2009). Taken together the
findings underscore the importance of (a) supervisors facilitating
helpful group processes (e.g., feelings of safety and cohesion)
while attending to individual trainee needs (e.g., feelings of anx-
iety), and (b) providing education within a context that promotes
enhanced emotional awareness and interpersonal learning (Flem-
ing, Glass, Fujisaki, & Toner, 2010).
Conclusions Concerning Supervision
Because the supervision studies are wide ranging in results and
potential implications, we will focus on the safest, most robust
conclusions that can be drawn from these studies. First, although
the database remains limited, significant advances over the past
two decades suggest that supervision may well have the capacity to
exert a modest effect on client outcome. Creative thinking about
how best to approach and address the supervision/outcome conun-
drum has been more recently on ready display than at any other
time in supervision’s over-century-strong history (e.g., Callahan et
al., 2009; Reiser & Milne, 2014). Such efforts accordingly appear
poised to more meaningfully and definitively advance our under-
standing of the likely variable impact of supervision on client
outcome; that long-overdue advance is most sorely needed now.
Calls to prove supervision’s accountability have understandably
grown louder in this age of accountability (Watkins, 2011). More
such pressure-to-prove calls can only be expected and they deserve
a vigorous, compelling empirical counter response.
Second, strong arguments have been made elsewhere that rou-
tinely gathering client outcome data results in improved quality of
care (e.g., Lambert, 2017). Such data may also be eminently
valuable in supervision for therapist training purposes (cf. Lappan,
Shamoon, & Blow, 2017; Swift et al., 2015). Although specific
research about using client outcome data in supervision is admit-
tedly limited, supportive studies have begun to accumulate. Future
research ideally will provide further, more nuanced, understanding
about the potential impact of using client data in supervision as
both a treatment and supervision benefit, but we believe that there
is sufficient justification to recommend that all supervisors make
usage of client outcome data a routine part of their supervision
practice at this time— or at least strongly consider doing so. If data
overwhelmingly supports the benefits of client data on treatment
(Lambert, 2017), then educating trainees about how to use that
client data to derive those benefits would seem best practice.
Third, over 30 years ago, Worthington (1987) lamented that we
lack for a theory of poor or inadequate supervision. Perhaps a most
significant advance since then has been supervision research that
helps us move toward more clearly articulating and elaborating
upon such a theory or model. We now have convincing, compel-
ling, persuasive data that shows supervision can indeed be for
better or worse, and some of what consistently makes supervision
“for worse” involves supervisor behaviors that violate relation-
ships in some capacity. Examples can run the full gamut, from
being inattentive, uninvested, and uninvolved to being sexually
coercive, physically assaultive, and relationally intimidating (e.g.,
Ellis, 2017).
Fourth, “The field of multicultural supervision is one of the
newest kids on the multicultural block . . .” (Inman & Ladany,
2014, p. 655); although that quote captures the current reality,
multicultural supervision is not without progress. Perhaps one such
advance is our now recognizing the preeminent importance of
making multicultural discussions an integral part of the supervi-
sory encounter. That bit of practice wisdom is reflected not only by
means of practice-directing documents (e.g., American Psycholog-
ical Association, 2015), but also in some of the empirical research
summarized here (e.g., Phillips et al., 2017). Where multicultural
discussions are made a core part of supervision, the potential
benefits seem considerable (e.g., more favorable alliance percep-
tions, positive client impacts; Soheilian et al., 2014). But where
those discussions do not happen, the potential for the converse
appears equally considerable (cf. Phillips et al., 2017).
Fifth, group supervision, formerly described as widely practiced
and poorly understood (Holloway & Johnston, 1985), might best
be thought of now as widely practiced and a bit more understood;
“. . . the essential components of group supervision are still being
uncovered . . .” (Riva & Erickson Cornish, 2008, p. 24). From what
we know (e.g., Fleming et al., 2010), it seems best for group
supervision to indeed be approached with a group-dominant mind-
set—that the group situation is not a replication of separate indi-
vidual supervisions but rather invokes a functioning collective that
increasingly develops capacity to function collectively. Group
 SCIENCE OF TRAINING III
cohesion appears critical to constructive process and effective
outcomes.
Sixth, although a moderate effect of supervision on client out-
comes has been observed in some studies of practicum training
(Callahan et al., 2009; Wrape et al., 2015), an open question is
whether the beneficial effects of supervision (in terms of compe-
tency development and/or client outcome) diminish over time.
While increasing competency has been found across preinternship
training (Price, Callahan, & Cox, 2017; Wolfe-Christensen &
Callahan, 2008), experiential supervision (key to deliberate prac-
tice) is used less frequently as trainees advance through the train-
ing pipeline (Hatcher, Wise, Grus, Mangione, & Emmons, 2012;
Scott, Ingram, Vitanza, & Smith, 2000) and competency develop-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
ment may level off by the time one emerges into the profession
This document is copyrighted by the American Psychological Association or one of its allied publishers.
(for a review of literature consistent with the idea of psychotherapy
competency stagnation among professional psychologists, see
Tracey, Wampold, Lichtenberg, & Goodyear, 2014). It is possible
that, if a leveling off of competency exists, it might begin prior to
internship. Although one study (Barabasz, Baer, Sheehan, &
Barabasz, 1986) found that psychologists demonstrated signifi-
cantly better client treatment outcomes than interns, that study
involved a narrowly focused treatment that is rarely used today and
the generalizability is questionable. The other located studies re-
ported no significant differences in attrition or client satisfaction
between interns and practicing psychologists (Grigg, 1961; Hep-
pner & Heesacker, 1983; Krauskopf, Baumgardner, & Mandrac-
chia, 1981), supporting the leveling off hypothesis.
Seventh, while our focus in this review is largely centered on
supervisors and the training they provide, we would be remiss if
we did not point out that supervision is inherently at least a dyadic
process. Beyond supervisors, trainees also contribute to supervi-
sion. A review of that literature is a bit beyond the scope of this
review, but a few relatively recent highlights are worth noting as
illustrative of this literature. Trainee extraversion and openness to
experience have each predicted significant variance in both leader–
member exchange and supervisory working alliance perceptions
(Rieck et al., 2015). Perhaps incidentally, these personality vari-
ables have also been found to be salient to internship match
(Callahan, Hogan, et al., 2014; Hogan et al., 2014). Similarly, low
trainee attachment anxiety and avoidance have been found to be
associated with more positive supervisory alliances (Huber et al.,
2010; Wrape, Callahan, Rieck, & Watkins, 2017). Despite these
findings, no significant difference in outcomes or rate of attrition
was found between groups of clients treated by supervisor–trainee
duos (i.e., cotherapy supervision) and groups of clients treated by
trainees (i.e., under supervision but not as cotherapy; Tanner,
Gray, & Haaga, 2012).
Competency
The bulk of the extant literature on the attainment of compe-
tency is theoretical and/or consensus based, rather than empirical.
The few empirical investigations that have examined the structure
of competency strongly suggest the presence of an overarching
general competency factor (e.g., Callahan, Neumann, Cox, &
Ruggero, 2017; Price et al., 2017; Strom et al., 2016), a feature that
is notably lacking from any of the competency conceptualizations
put forth in prominent contemporary theories. Although the dis-
249
crepancy between theory and empiricism may merit consideration,
our focus herein is on empirical contributions.
There have been several empirical studies examining trainees’
perspectives about their competency development. For example,
trainees describe themselves as acquiring assessment related com-
petencies early in doctoral training (Years 1–3; Kamen, Veilleux,
Bangen, VanderVeen, & Klonoff, 2010), without significant com-
petency growth thereafter (Wolfe-Christensen & Callahan, 2008).
They describe competencies pertaining to intervention, research,
and empathy as taking longer to develop, but fully present by the
time they apply for internship. They again noted a lack of subse-
quent growth between applying for internship and the early months
of internship (the timing of the study precluded sampling later
months of internship; Kamen et al., 2010). Competencies in su-
pervision appear to emerge late in training (i.e., internship or
postdoctoral training; Genuchi, Rings, Germek, & Cornish, 2015),
but are widely perceived by internship directors as not being
central to their training goals and objectives (Stedman, Schoenfeld,
& O’Donnell, 2013), despite their acknowledgment of the impor-
tance of developing competency in supervision (Rings, Genuchi,
Hall, Angelo, & Cornish, 2009). Whether trainees are correct or
not in their perceptions of how competency unfolds is unresolved
and worth exploring. In fact, beyond trainees’ perceptions, and a
single report recently that conscientiousness may be predictive of
competency development (Humphreys, Crino, & Wilson, 2017),
the empirical literature is lacking with respect to our understanding
of how competency attainment unfolds. Addressing that need
necessitates the development of reliable and valid measures for use
in assessing competency longitudinally.
Assessment of Competence
Nearly all (98%) programs evaluate practicum training, but
there is high variability in when and by whom those evaluations
are conducted (Hatcher, Grus, & Wise, 2011). Most programs
appear to rely on locally adapted measures that (a) are intended to
map onto theoretical frameworks, but have unknown psychometric
properties (Grus, Falender, Fouad, & Lavelle, 2016), or (b) suffer
psychometrically and are experienced as unsatisfying to trainees
(Goldberg, DeLamatre, & Young, 2011). Studies of such locally
developed measures have additionally evidenced problems of rater
leniency biases (Gonsalvez et al., 2013; Gonsalvez & Freestone,
2007), gender biases females rated less favorably; (Chung, Mar-
shall, & Gordon, 2001), and poor predictive validity (Goldberg &
Young, 2015; Gonsalvez & Freestone, 2007; Strom et al., 2016).
Additionally, a recent study found that dichotomization of com-
petency (e.g., pass/fail) resulted in significantly poorer psychomet-
ric properties than when gradations of competency evaluation were
provided (Roberts & Dorstyn, 2017). Unfortunately, the overall
impression of a trainee may influence supervisor ratings more than
actual competency (Strom et al., 2016).
There is a small, emerging body of literature focusing on the use
of standardized vignettes in an effort to improve supervisor accu-
racy and/or improve psychometric properties in the assessment of
competency (e.g., Gonsalvez et al., 2013). This effort, however,
confronted a new problem: the results could not differentiate
novice from advanced beginner trainees. A subsequent study using
vignettes provided preliminary evidence that vignettes could be
developed to differentiate varying levels of competency on spe-
 250 CALLAHAN AND WATKINS
cific clinical skills, but that study highlighted the vulnerability of
vignettes to poor item-level psychometrics (Carpenter et al., 2016).
Another program of research has focused on the use of computer
simulations with standardized vignettes, but those involved sub-
stantial administrative burden and relied on avatars that assess
knowledge and/or technical skill execution without the rich con-
text of the salient affective and interpersonal competencies that are
often at issue in trainees with competency problems (e.g., Newell,
2010a, 2010b, 2012; Newell, Newell, & Looser, 2013).
Finally, although the American Psychological Association Task
Force on the Assessment of Competence in Professional Psychol-
ogy called for multisource assessment of competency (a.k.a., 360°
evaluations), including self-appraisals, that recommendation was
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
based on literature that is not within the scope of training health
This document is copyrighted by the American Psychological Association or one of its allied publishers.
service psychologists (Kaslow et al., 2007). The question of
whether health services psychology trainees typically have suffi-
cient competency to engage in accurate self-appraisals had not yet
been empirically considered. Quite distally, there was one report
that trainees’ self-ratings of personal competency may be strongly
influenced by the number of supervisors and the total time spent in
supervision (accounting for 41% of the variance in self-ratings;
Bradley & Olson, 1980). But it was not until more recently that
clear concern about the use of self-appraisals emerged in the
empirical literature. In a recent study on the assessment of com-
petency, trainee self-ratings were found to be problematic from a
validity standpoint (Price et al., 2017). While trainees’ perceptions
may be useful in some regards (e.g., capturing trainee perceptions
of progress, Larkin & Morris, 2015; Tsong & Goodyear, 2014), we
caution programs from treating self-appraisals as reliable or valid
indicators of competency, per se. The work of Larkin and Morris
(2015) on self-assessment of competency is useful.
Use of an item response theory approach to measure compe-
tency development appears to hold promise. Recently, a 50-item
measure of foundational and functional competencies emerged that
uses a web-based visual analog measurement scale to capture
supervisor ratings of competency (Price et al., 2017). In that
prospective longitudinal study, many-facet Rasch measurement
was used to allow for specification of rater severity (e.g., biases) in
the model. Findings demonstrated psychometric support for use of
the measure in the assessment of competency across prepracticum,
practicum, and external practicum settings as well as across mul-
tiple subdisciplines, including clinical, clinical health, and coun-
seling psychology. Significant gains in competency corresponded
to progressive stages of practicum training (i.e., prepracticum,
practicum, and external practicum) and specific competencies
were found to vary in difficulty in that study. Taken together, that
information could shed light on the trajectory of competency
development and be useful in sequencing training toward compe-
tency attainment by scaffolding trainee development in an empir-
ically supported sequence. Future research investigating that pos-
sibility is encouraged.
Competency Problems
A clear finding from the empirical literature is that trainees are
aware of their peers having competency deficits (Mearns & Allen,
1991; Oliver, Bernstein, Anderson, Blashfield, & Roberts, 2004;
Rosenberg, Getzelman, Arcinue, & Oren, 2005; Shen-Miller et al.,
2011). In particular, they are strongly aware when those deficits
are interpersonal in nature (e.g., aversiveness, narcissism, sexism,
Mearns & Allen, 1991; Oliver et al., 2004; Rosenberg et al., 2005)
or unethical (i.e., breach of confidentiality, cheating, dual relation-
ships, Mearns & Allen, 1991; or searching online for client infor-
mation, Asay & Lal, 2014; DiLillo & Gale, 2011). Further, trainees
often describe this awareness of their peers’ competency problems
as provoking their own internal turmoil (e.g., anger, dismay, frus-
tration) and conflict (e.g., ethical dilemmas; Mearns & Allen,
1991; Oliver et al., 2004).
Trainees report several, not mutually exclusive, responses when
faced with a problematic peer. Trainees may engage in gossiping
or consulting with others (Rosenberg et al., 2005), withdrawing
themselves or otherwise ignoring the concerns (Mearns & Allen,
1991; Rosenberg et al., 2005), or directly confronting their peer
(Mearns & Allen, 1991; Rosenberg et al., 2005). Notably, faculty
significantly underestimated how frequently trainees directly con-
fronted their peers about competency related concerns (Mearns &
Allen, 1991).
Trainees generally perceive faculty as needing to take a stronger
stance (e.g., repeat practicum or coursework, program suspension
or termination) when a peer is viewed as consistently lacking
competency (Mearns & Allen, 1991; Oliver et al., 2004; Rosen-
berg et al., 2005), but hope that faculty will exercise leniency (e.g.,
requiring psychotherapy or leave of absence) in cases where tran-
sient competency lapses emerge in response to a stressor (Veilleux,
January, VanderVeen, Reddy, & Klonoff, 2012a). The literature
points to two common sources of transient stress that may be
salient in this regard. In an exploratory study of vicarious trauma
among therapist trainees, Adams and Riggs (2008) found that
trainees with prior personal trauma exposure, who were early in
their practicum training and viewed themselves as self-sacrificing
under stress, were more likely to evidence vicarious traumatization
resulting from working with traumatized clients. In a more recent
study, depression or other clinically significant distress was found
to be negatively associated with trainee competency (Humphreys
et al., 2017).
Trainees viewed faculty as significantly less active in their
gatekeeping role than faculty viewed themselves (Mearns & Allen,
1991). One possible explanation for the discrepancy between
trainee and faculty perceptions may be that they are focused on
different sources of information. While trainees describe primary
concerns in the interpersonal realm, program directors cite inade-
quate clinical skills as the most common reason for dismissal from
training (Vacha-Haase, Davenport, & Kerewsky, 2004). Another
possibility though is that what trainees view as insufficient activity
may instead be indicative of a lack of faculty consensus on how to
proceed with a student. There is some evidence that differing
individual faculty conceptualizations regarding diversity can exert
a disruptive impact in identifying and/or handling trainee compe-
tency deficits (Shen-Miller, Forrest, & Burt, 2012). In contrast,
examination of biases, use of consultation, and a strong program
commitment to conversations regarding diversity have been iden-
tified as important sources of positive influence (Shen-Miller,
Forrest, & Burt, 2012).
Of course, somewhat obviously, needed work examining the
possibility of widespread competency stagnation at some point
(e.g., potentially within the training pipeline) requires the avail-
ability of reliable and valid measures of competency. Unfortu-
nately, empirical work on competency assessment has lagged
 SCIENCE OF TRAINING III
considerably behind theory developments and conceptual shifts
associated with the training field’s “culture of competence” (Rob-
erts, Borden, Christiansen, & Lopez, 2005, p. 356). Based on the
current status of the literature, supervisors appear to play a critical
role in the assessment of trainees’ competencies, performing best
when their ratings are based on experiential observation of train-
ees’ work, but they are vulnerable to bias (Chung et al., 2001;
Gonsalvez et al., 2013; Gonsalvez & Freestone, 2007; Strom et al.,
2016). New measures for assessment of competency that account
for supervisor rating severity and evidence strong psychometric
properties have only recently emerged (e.g., Price et al., 2017).
As measures become more readily available and utilized, addi-
tional possibilities for beneficial intersecting implications may
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
come in to focus. For example, Price and colleagues (2017) sug-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
gested that mean competency scores across all supervisors (to
thwart self-serving biases from trainees selecting a subset of their
supervisors) for a given trainee on a standardized competency
measure could be included in the standard application for intern-
ship. Such competency means, if contextualized by local norms
(M and SD), could be an important and clinically meaningful
enhancement (as called for by Klonoff, 2015 and also Williams &
Cornish, 2015). At present, the application for internship empha-
sizes academic achievements with letters of recommendation
(where competencies may be described) evidencing little utility
(Grote, Robiner, & Haut, 2001; Miller & Van Rybroek, 1988;
Stedman, Hatch, & Schoenfeld, 2009). Implementation of such a
change would necessitate careful thought and nuanced coordina-
tion to be successful.
Interestingly, neither peers nor faculty may be aware or focused
on the competency that was most frequently self-identified as
functionally problematic in a nationwide sample: self-care (El-
Ghoroury, Galper, Sawaqdeh, & Bufka, 2012). Program emphasis
of self-care is associated with trainees’ quality of life (Goncher,
Sherman, Barnett, & Haskins, 2013). Further, in a nationwide
study of self-care behaviors, trainees’ sleep hygiene, social sup-
port, emotion regulation, and acceptance were found to be predic-
tive of subjective stress (Myers et al., 2012). However, Bamonti
and colleagues (2014) reported that only a minority of doctoral
programs reference self-care at all in their handbooks for trainees.
The most common mention of self-care was found to be in the
context of psychotherapy or other mental health services as avail-
able to trainees in need of assistance. This oversight is perhaps a
bit concerning given recent evidence that competency in self-care
is not an easy competency to develop.
The moderate difficulty associated with development of self-
care competency (Price et al., 2017) suggests that training in this
competency during prepracticum might be appropriate. Pakenham
(2017) provided illustration of such an approach by including
self-care skills training in a formal course on acceptance and
commitment therapy (ACT). In that study, most trainees (⬃74%)
reported making at least one behavior change in their self-care
skills as a function of the course. In reviewing the findings,
Pakenham felt that the ACT framework was especially well-suited
to an integration with self-care skills training. However, this may
not be unique to that framework; research investigating the effects
of augmenting other courses with a self-care skills training com-
ponent could be helpful and may be needed. Although generally
positive effects were reported, Pakenham noted that ACT defusion
techniques (ways of disconnecting or otherwise distancing from
251
distressing thoughts and/or feelings) were associated with adverse
impact on stress, distress, life satisfaction, and training program
satisfaction. The positive effects on self-care seemed to be carried
by the mindfulness aspect of acceptance techniques taught as part
of the ACT course. These differential effects underscore evidence
pointing to trainee self-care as a multidimensional construct
(Ayala, Ellis, Grudev, & Cole, 2017). A dismantling design that
more carefully parses mindfulness and acceptance effects on self-
care skills training could be useful in designing training that is
intended to foster self-care competency. A promising multidimen-
sional (i.e., cognitive– emotional–relational, physical, spiritual)
measure of self-care competency has recently emerged that may be
useful in testing the effects of such a training (Santana & Fouad,
2017).
Although we suggest adding training to foster self-care compe-
tency, we realize this suggestion may not be met with strong
endorsement. As a competency, it is referred to as self-care so, to
us, that seems to inherently place appropriate responsibility for
development of that competency on the individual student rather
than on a program. We also appreciate that the literature is replete
with suggestions for adding to training, without regard for the
increased costs (including time) and uncertain benefits. Yet, we
still recommend inclusion of self-care training.
In our opinion, the empirical findings point to training in self-
care to be necessary because (a) there is some indication that
burnout at some point in training may be remarkably common
among doctoral students (see our review in this issue, Callahan &
Watkins, 2018b), (b) there is some indication that such experiences
might be compromising the quality of care in the clinic (Callahan
& Watkins, 2018c), and (c) competency in self-care, as it turns out,
is not all that easy. Item difficulty levels for self-care put it firmly
in the middle of competencies for which years of training are
devoted (Price et al., 2017). In other words, self-care competency
is more difficult to attain than many of the competencies we spend
a good amount of formal instruction time on. In reflecting on the
first two points above, Santana and Fouad (2017) went so far as to
deem attainment of competency in self-care to be an ethical
imperative. When considered with that weight, the observation that
attainment of self-care competency falls at the point of moderate
difficulty seems to advocate for programs including it in training.
Now, having said that, it seems impractical and likely wholly
unnecessary to spend significant time on the topic of self-care.
Based on what we know at this time, brief training in mindfulness
may be facilitative of clinically salient self-care behavior (e.g.,
centering oneself, meditative breaks, etc.) and has been shown to
improve in session effectiveness with clients (see our review in
this issue, Callahan & Watkins, 2018c). Such training can be
accomplished with minimal time or cost allocations. Beyond such
an approach, we think it may be reasonable to expect programs to
encourage students and faculty supervisors to discuss self-care on
an as needed, and individually titrated, basis so that any necessary
additional training can be identified and implemented. For some
trainees, that might mean referral to a therapist for extended
consideration of self-care issues, focal recommendation of con-
sumer debt counseling, requirement of additional coursework in an
area of weakness that is fostering more broad stress and compro-
mising self-care, brief discussion of work-life balance goals, and
so forth The options for such additional competency promoting
activities, and titration of those options, may be as variable as the
 252 CALLAHAN AND WATKINS
students we train. Regardless of what a program does, perception
of the program as emphasizing self-care is known to foster actual
engagement of more self-care among trainees (Zahniser, Rupert, &
Dorociak, 2017).
Moving away from self-care competency, regardless of source,
it can be concluded that competency problems among trainees are
associated with a host of adverse consequences. The adverse
impact to personal functioning, relationships with peers, and mu-
tual underappreciation of trainee and faculty responsiveness is
disruptive to the training environment (Rosenberg et al., 2005).
Within that milieu, trainees with competency deficits are likely to
perceive the program climate more negatively (Veilleux, January,
VanderVeen, Reddy, & Klonoff, 2012b).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Finally, it is worth acknowledging the potential for trainee
This document is copyrighted by the American Psychological Association or one of its allied publishers.
assessments of competency to be used in aggregate form for other
purposes. Reliable and valid assessments of trainee competency
hold potential for use in improving methods associated with as-
sessment of doctoral and internship training programs’ compe-
tency in training (e.g., accreditation reviews by either/both bodies
that currently accredit doctoral psychology programs). The use of
trainee outcomes as indicative of programs’ competency in train-
ing is not entirely new. Match rates at the program level clearly
differentiate programs’ (Parent & Williamson, 2010), as do aggre-
gated pass rates on the Examination for Professional Practice of
Psychology (EPPP; Schaffer et al., 2012). Our final recommenda-
tion then is to encourage exploration into the utility of using
de-identified aggregated assessments of trainee competency in
program evaluation.
Internship
Since the beginning of the modern4 internship match era,
through the 2016 internship match, aggregated data (posted to
appic.org) clearly evidenced a larger supply of prospective interns
than available internship positions. Perhaps for that reason, the
majority of the extant empirical literature located in our search
concerned internship match.
Internship Match: Myths Versus Evidence
Drawing from a large nationwide sample, applicants for intern-
ship were found to errantly believe that the most important vari-
able in their application was the total number of their accumulated
practicum hours (Bangen, VanderVeen, Veilleux, Kamen, &
Klonoff, 2010). However, in reality, the single best predictor of
matching for internship was found to be the number of interview
offers attained (Callahan, Collins, & Klonoff, 2010; Callahan,
Hogan, Klonoff, & Collins, 2014). Drawing from a nationwide
prospective sample followed longitudinally, six or fewer offers to
interview forecasted increased risk of going unmatched, as did
entering the match two or more times (see Figure 1; Callahan,
Hogan, et al., 2014). Further, as shown in Figure 2, intervention
and assessment hours were found to significantly interact with
supervision hours to predict the number of internship interview
invitations. In short, the data indicated that accrual of closely
supervised training experiences was beneficial to applicants. An-
other important interaction effect, illustrated in Figure 3, revealed
that research achievements and accrual of proportionate clinical
hours interacted to predict the number of internship interview
offers. Subsequently, in an unrelated study, the total number of
publications was found to provide a match advantage to internship
applicants (Lund, Bouchard, & Thomas, 2016).
The trainee/faculty lore with which we are familiar also pro-
motes another errant belief: geographic restriction is thought to put
a trainee at risk of not matching. However, when analyses con-
trolled for number of applications submitted, geographic restric-
tion did not account for significant variance in match outcomes
(Callahan, Hogan, et al., 2014). In other words, if a trainee submits
an average number of applications, the fact that they are geograph-
ically restricted does not necessarily need to be a significant source
of match anxiety. This interpretation though needs to be a bit
tempered because some cities and geographic regions are more
competitive than others (Seawell, Krohn, Gorgens, & Erickson
Cornish, 2009). The northeast is the hardest region to secure a
match, followed by the West, Midwest, and South. At the city
level, the Chicago, Illinois, metropolitan area is in a tier by itself.
With a match rate under 4%, it is much more competitive than any
other city. The second tier (Philadelphia, PA; San Francisco, CA;
Denver, CO; and New York, NY) all had match rates of 4%. The
third tier (Boston, MA; Los Angeles, CA; San Diego, CA; Dallas,
TX; Seattle, WA) evidenced match rates of 5%, while the fourth
tier (Miami, FL; Houston, TX) demonstrated match rates of 6 and
7%, respectively.
Although trainees generally feel prepared to apply for intern-
ship, they would like to learn critical information (e.g., match
predictors, geographic competiveness) earlier in training (Bangen
et al., 2010). As trainees get closer to applying for internship, they
have been found to desire individual assistance in preparing their
essays and vitae (Madson, Aten, & Leach, 2007). The Bangen and
colleagues (2010) study reported that, when preparation begins in
earnest, trainees would like to have more information regarding
essay writing strategies and existing resources. At that point,
trainees are also faced with soliciting recommendations to support
their application. Even though letters of reference are widely
reported to be central to internship program selection (e.g., Lopez,
Oehlert, & Moberly, 1996) and letter writers believe that they are
willing to disclose negative information (Grote et al., 2001), letter
readers report that they rarely encounter negative information
(Grote et al., 2001; Miller & Van Rybroek, 1988). Further, using
linguistic analysis, there is strong evidence that unstructured letters
do not meaningfully differentiate internship applicants (Stedman et
al., 2009).
Site Differences
There is emerging evidence of a shifting supply and demand ratio
(anticipated by Hatcher, 2015, but first evident in raw data associated
with the 2017 match that is posted on appic.org), whereby applicants
may now hold more of an advantage (than internship sites) in securing
an internship match. With a shift in the internship imbalance from a
site-favorable market to one that may be more trainee favorable, we
may see a corresponding shift in empirical research associated with
internship match. At present, however, there is scant empirical infor-
mation on internship site differences that may be salient to trainees in
evaluating sites. Nevertheless, three studies were located that provide
4
Computerized algorithm-driven matching of rank order lists with uni-
form electronic notification.
 SCIENCE OF TRAINING III 253

1 25

0.9

Frequency of Aained Interviews

0.8 20

0.7

Probability of Matching
0.6 15

0.5

0.4 10

0.3

0.2 5

0.1
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

0 0
This document is copyrighted by the American Psychological Association or one of its allied publishers.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of Offers to Interview

Figure 1. Straight, dotted line demarks the match rate within the sample. Remaining lines plot the probability
of matching as a function of number of interviews; dashed lines indicate 95% confidence intervals. Histogram
displays the frequency at which the number of offers to interview was attained. From Predicting Match
Outcomes: Science, Practice, and Personality, by J. L. Callahan, L. R. Hogan, E. A. Klonoff, & F. L. Collins Jr,
2014, Training and Education in Professional Psychology, 8, p. 78. Copyright 2014 by American Psychological
Association. Reproduced with permission.

some initial empirical observations that inform our understanding of tations) and practicum training experiences (client contact hours,
(a) how interns who match at different types of sites differ, (b) interns’ supervision hours, ratio of client contact hours to supervision, number
subjective perspectives on types of sites, and (c) barriers to internship of integrated reports). Additionally, findings indicated that having
accreditation. practicum experience in the same type of setting to which an applicant
As shown in Figure 4, interns who match systematically differ in hopes to match may be necessary for many types of sites (e.g.,
their characteristics according to the type of site that they matched to correctional facilities, university counseling centers).
(Callahan, Swift, et al., 2014). A few of these differences precede Trainees widely consider medical schools to be the most pres-
admission to doctoral training (quantitative Graduate Record Exam- tigious type of internship site (Shivy, Mazzeo, & Sullivan, 2007).
ination [GRE] scores) and/or reflect personality factors (openness, Trainees indicated that site prestige, severity of client illness, and
agreeableness, extraversion), but most pertain to research achieve- comfort working with the site’s clientele were all variables that
ments (number of peer-reviewed publications and number of presen- were influential to their rating of sites. Clinical trainees addition-

9.5
Mean Total Number of Offers to Interview

8.5
Total Number of Supervision Hours
8
high (+1 SD = 699.20)
7.5 medium (m = 472.96
low (-1 SD = 246.24)
7

6.5

6
low medium high
(-1 SD = 516.78) (m = 997.63) (+1 SD = 1478.48)

Total Number of Intervenon and Assessment Hours

Figure 2. Moderation of the effect of intervention and assessment hours on number of internship interview
offers by number of supervision hours. From Predicting Match Outcomes: Science, Practice, and Personality, by
J. L. Callahan, L. R. Hogan, E. A. Klonoff, & F. L. Collins Jr, 2014, Training and Education in Professional
Psychology, 8, p. 75. Copyright 2014 by American Psychological Association. Reproduced with permission.
 254
8.4
Mean Total Number of Offers to Interview
8.2
8
7.8
7.6
7.4
7.2
7
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
low
This document is copyrighted by the American Psychological Association or one of its allied publishers.
(-1 SD = 0.88)
Proporon of Intervenon and Assessment Hours to Supervision Hours
Figure 3. Interaction of proportionate clinical hours and number of publications on number of internship
interview offers. From Predicting Match Outcomes: Science, Practice, and Personality, by J. L. Callahan, L. R.
Hogan, E. A. Klonoff, & F. L. Collins Jr, 2014, Training and Education in Professional Psychology, 8, p. 76.
Copyright 2014 by American Psychological Association. Reproduced with permission.
ally considered job prospects and opportunities for research while
counseling trainees considered quality of supervision. Exploration
of the role of these variables in future matches may be of assis-
tance to internship sites needing to attract more applicants to their
internship program.
A reasonable hypothesis worth exploring is that unaccredited in-
ternship sites may be more strongly impacted than accredited sites if
the match imbalance makes a sustained shift from site-favorable to
trainee-favorable. It behooves unaccredited sites to seek accreditation
as a means of increasing their competitiveness for matching to the
interns that are most well-suited to them. Such sites should be aware
that funding (i.e., accreditation application/site visit and ongoing fees;
intern stipends and benefits), administrative burden, and institutional
support all appear to be associated with internship accreditation suc-
cess (DeHay, Ross, Williams, & Ponce, 2016).
Internship Training Experiences
After attainment of an internship match, trainees must quickly
adjust to their new training environment and the experiences
afforded. However, there is very little empirical work informing
our understanding of internship training experiences. What is
known is that the majority of time in training during internship is
centered on learning objectives associated with intervention and
assessment competencies (Stedman et al., 2013). Beyond that
focus, there appear to be measurable differences in the research
training environment at the internship level (Phillips, Szymanski,
Ozegovic, & Briggs-Phillips, 2004) and these differences appear to
influence the scholarly productivity of interns (Szymanski, Oze-
govic, Phillips, & Briggs-Phillips, 2007).
Internship Supervision
There are known differences in the supervision training envi-
ronment at the internship level, with some offering significantly
greater enrichment (i.e., counseling centers; Scott et al., 2000).
There also appear to be some differences across settings in terms
CALLAHAN AND WATKINS
Total Number of Publicaons
high (+1 SD = 7.15)
medium (m = 3.52)
low (-1 SD = 0.0)
medium high
(m = 2.41) (+1 SD = 3.94)
of skill development emphasis. For example, while forensic set-
tings emphasize competencies pertaining to professional behavior
with clients, counseling centers differentially emphasize personal-
ization skills (e.g., attitudes, feelings, and mannerisms as they
relate to the client; Eisenhard & Muse-Burke, 2015). Further
empirical study of setting differences such as these could be very
useful to both sites and internship applicants as they seek to match
trainee strengths and needs with internship site opportunities.
Although some findings may generalize (e.g., no differences in
supervisory style have been identified as associated with internship
setting; Eisenhard & Muse-Burke, 2015), the findings of the stud-
ies herein cannot be assumed to generalize across all internship
settings.
A study from nearly 20 years ago noted that the majority of
interns recorded no experiential observation of their clinical work
by supervisors (e.g., one-way-mirror observation, video recording,
audio recording; Scott et al., 2000). A much more recent study
suggests insufficient experiential observation during internship
may be a persisting problem. Across internship settings, trainees
were found to receive significantly less experiential supervision of
their work (as compared to practicum), though it is unclear how
much empirical observation is necessary at the internship level
(Rodriguez-Menendez, Dempsey, Albizu, Power, & Campbell
Wilkerson, 2017). There is some data to suggest that a decline in
experiential supervision may start prior to internship; external
practicum sites have reported only limited use of direct observa-
tion (Hatcher et al., 2012).
Although experiential supervision may be underutilized, group
supervision is very common at the internship level (Riva, &
Erickson Cornish, 2008) and there appear to be some unique
challenges with that practice. Interns typically have no previous
training experience with the other trainees in group supervision
and, as such, it is perhaps not surprising that absent or ineffective
communication (e.g., regarding roles, expectations, confidential-
ity, privacy, norms, multiple relationships) is key to many ethical
 SCIENCE OF TRAINING III 255
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Figure 4. Applicant characteristics as a function of type of internship setting. Means for each type of internship
setting are presented via spark lines (left to right: 1 ⫽ Armed Forces Medical Center, 2 ⫽ child/adolescent
psychiatry/pediatrics, 3 ⫽ Community Mental Health Center [CMHC], 4 ⫽ consortium, 5 ⫽ medical school,
6 ⫽ prison or other correctional center, 7 ⫽ private general hospital, 8 ⫽ private outpatient clinic, 9 ⫽
psychiatric hospital, 10 ⫽ psychology department, 11 ⫽ school district, 12 ⫽ public hospital, 13 ⫽ University
Counseling Center [UCC], and 14 ⫽ Veterans Affairs Medical Center [VAMC]). Note that the visual inspection
of spark lines may occasionally suggest a difference that is not confirmed by t test due to unequal variances
among groups being compared. Due to insufficient sample size, the Armed Forces Medical Center setting was
not included in pairwise comparisons. GRE ⫽ Graduate Record Examination. From Subjective Well-Being
Among Internship Applicants, by L. R. Hogan, J. L. Callahan, K. A. Tompkins, J. K. Swift, D. R. Connor &
E. A. Klonoff, 2014, Psychotherapy Bulletin, 49, p. 15. Copyright 2014 by American Psychological Association.
Reproduced with permission.

issues emerging during group supervision (Smith, Riva, & Erick-
son Cornish, 2012).
Finally, relational characteristics of clinical trainees and super-
visors appear to influence the supervisory relationship at the in-
ternship level (Riggs & Bretz, 2006). Perceived supervisor attach-
ment style was significantly associated with specific aspects of the
supervisory working alliance (specifically, task agreement and
bond). Secure supervisors evidenced stronger trainee alliances.
Although attachment style among trainees was not found to be
directly related to the supervisory alliance, a path analysis revealed
that parental indifference was associated with compulsive self-
reliance, which in turn was associated with interns’ perceptions of
supervisors’ attachment style. That perception then was associated
with trainees’ appraisals of the alliance.
The findings pertaining to internship match indicate that prep-
aration for a strong application needs to (a) begin far in advance
(i.e., years) to allow time to structure accrual of proportionately
supervised clinical hours with simultaneous pursuit of competitive
research achievements, (b) continue all the way up to the point of
applying for internship with increasingly personalized assistance,
(c) be submitted to a sufficient number of sites, and (d) be
submitted with awareness of geographic considerations.
To elaborate a bit on that first point (and illustrate one of the
intersecting implications that span internship, supervision and
competency),5 the widely held belief that quantity of hours is most
important (Bangen et al., 2010), without appreciation of the pro-
portion of time spent in supervision, may foster unproductive
decisions. For example, contact hours have been found to be
frequently misrepresented throughout training (Rodolfa, Owen, &
Clark, 2007; Tracy, Bucchianeri, & Rodolfa, 2011), presumably to
foster an appearance of greater time spent developing competency.
Underscoring the point that even trainers may be misguided and/or
engaging in impression management concerning competency de-
velopment, internship sites have been found to include leave and
paid time off in their endorsement of trainees’ cumulative prepa-
ratory supervised hours toward licensure (Tracy et al., 2011).
5
The finding that close supervision is important also appears to have
intersecting implications with findings reported in our pre-practicum re-
view of empirical studies (Callahan & Watkins, 2018c). In short, mounting
evidence indicates that training methods that provoke deliberate practice
may foster better, observable outcomes than simple repetitious practice as
early as prepracticum.
 256 CALLAHAN AND WATKINS
In light of the observed interaction effects, at least two conclu-
sions salient to optimal preinternship preparation may be drawn.
First, client contact hours that are so high that they preclude
genuine involvement with research leading to demonstrable re-
search achievements (e.g., peer reviewed publications) is ill ad-
vised. Second, accumulation of client contact hours at a rate that
significantly outpaces supervision can also be problematic. Data
drawn from a large nationwide sample of matched interns revealed
that most internship settings selected closely supervised interns
(the mean intervention/assessment to supervision hour ratio fell
between 2.11 and 2.58 to 1 for all but correctional settings which,
perhaps surprising, evidenced significantly less close supervision
with a mean ratio of 3.12 to 1; Callahan, Hogan, et al., 2014).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Those interested in the mean number of clinical hours, supervision
This document is copyrighted by the American Psychological Association or one of its allied publishers.
hours, research achievements, and other detailed information as
they are associated with a given type of internship setting may
access that information directly (http://psychology.unt.edu/sites/
default/files/Internship-Site-Data.pdf; enlarge file after opening for
optimal viewing).
In acknowledging broad anxiety associated with applying for
internship, it is important to note that trainees have identified
specific faculty behaviors that they found helpful with match
anxiety reduction (Bangen et al., 2010). These include starting the
process early, providing knowledge about the process, empathy/
emotional support, and prompt completion of recommendation
letters. We encourage faculty to engage all of these behaviors to
assist trainees as there is clear quantitative evidence that notifica-
tion of being unmatched is associated with a rapid and signifi-
cantly decline (d ⫽ 1.01) in subjective well-being (Hogan et al.,
2014).
Conclusions
We believe an evidence-based approach to training is in the
sustained best interests of the general public, as well as our field.
As we see it, doctoral training is a systems-level mental health
intervention whereby thousands of lives may be impacted (via
direct clinical care and/or via clinical research) by each individual
who graduates from an accredited doctoral training program in
health services psychology. Although beyond the scope of our
review, there is a very real possibility that little expertise devel-
opment accumulates after emerging into the profession (cf. Tracey,
Wampold, Lichtenberg, & Goodyear, 2014). Quality training is
therefore of critical importance. We encourage our field to resist
divisiveness fueled by tribalistic discourse and instead commit to
an evidence-based approach to training.
Protecting the public from profession-wide competency defi-
cit(s) falls within the responsibility of accrediting bodies. In con-
temporary training, there are two accrediting bodies that may be
considered as bearing responsibility in this regard: Commission on
Accreditation and Psychological Clinical Science Accreditation
System. In the opinion of the authors, regulations developed by
these accrediting bodies should be grounded in empirical research.
The total body of accumulated work reviewed in this series on the
science of training provides compelling evidence that such a
literature is available. If accrediting bodies ignore or otherwise fail
to appreciate the science of training, public welfare may be un-
dermined (if the purpose of accreditation is to protect against too
low a floor being enforced) and it becomes difficult to push the
field forward collectively (if the purpose of accreditation is to raise
the ceiling via enforcing higher standards).
Although we provide that bit of discussion as it pertains to the
two contemporary accreditation systems, we are aware that asking
either of them to implement greater specificity and greater rigor
carries risk for excessive prescription in training methods and
processes that could result in stifling innovation or other advance-
ments to doctoral training. Thus, an alternative path to consider is
one in which directors of clinical training and training directors
assume the onus of responsibility for identifying practices and
pushing the field forward via greater participation in developing
the science of training. In particular, we encourage programs
engaging in in-depth training program evaluations (e.g., self-
studies) to share their findings and grow the body of empirical
training literature.
Finally, we encourage the field to draw from advances in teacher
education and maximize the process and impact of program eval-
uations. LaBoskey (2006) convincingly argued that validation of
self-studies over time is an important contribution of scholarship
that can build a body of knowledge in specific domains and foster
understanding of teaching/training impact. The volume of self-
studies generated across accredited doctoral, internship, and post-
doctoral programs holds potential for “situating individual studies
within coherent research programs on particular substantive is-
sues” (p. 36) as suggested by Zeichner (2007). Zeichner further
argued that action research of this nature “would begin to infuse
the insights . . . into the broader knowledge-base of the field and
to affect the policy-making process” (p. 40). Unfortunately, as
evident to subscribers and regular readers of Training and Educa-
tion in Professional Psychology, validation processes and/or liter-
ature contributions stemming from program self-studies are ex-
ceedingly rare in our field.
To encourage a proliferation of contributions, we propose a
recurrent conference (in the style of the Castle Conferences that
are hosted by a special interest group, Self-Study of Teacher
Education Practices, of the American Educational Research Asso-
ciation). A biennial conference could provide a gathering in which
scholarship derived from self-studies could be shared via presen-
tations while an optional preconference workshop could actively
assist authors with transforming self-studies into submission ready
articles for possible publication. A recurrent conference of that
nature could also provide an important mechanism that enables
attendees to keep abreast of advancements in methodology and
quality expectations associated with self-study research. Further,
such a conference series could encourage validation of self-studies,
as encouraged by LaBoskey (2006), and create opportunities for
development of collaborative research programs, encouragement
of self-studies to build upon the work of others, and foster the
dissemination and publication of empirical work across the doc-
toral, internship, and postdoctoral training spectrum. As noted by
Zeichner (2007), building an empirical literature of that nature can
infuse important insights into the broader knowledge-base of the
field that then shape decisions regarding accreditation standards
and implementing regulations.
References
Adams, S. A., & Riggs, S. A. (2008). An exploratory study of vicarious
trauma among therapist trainees. Training and Education in Profes-
 SCIENCE OF TRAINING III
sional Psychology, 2, 26 –34. http://dx.doi.org/10.1037/1931-3918.2
.1.26
American Psychological Association. (2015). Guidelines for clinical su-
pervision in health service psychology. American Psychologist, 70,
33– 46. http://dx.doi.org/10.1037/a0038112
Asay, P. A., & Lal, A. (2014). Who’s Googled whom? Trainees’ Internet
and online social networking experiences, behaviors, and attitudes with
clients and supervisors. Training and Education in Professional Psy-
chology, 8, 105–111. http://dx.doi.org/10.1037/tep0000035
Ayala, E. E., Ellis, M. V., Grudev, N., & Cole, J. (2017). Women in health
service psychology programs: Stress, self-care, and quality of life. Train-
ing and Education in Professional Psychology, 11, 18 –25. http://dx.doi
.org/10.1037/tep0000141
Bambling, M., King, R., Raue, P., Schweitzer, R., & Lambert, W. (2006).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Clinical supervision: Its influence on client-rated working alliance and
This document is copyrighted by the American Psychological Association or one of its allied publishers.
client symptom reduction in the brief treatment of major depression.
Psychotherapy Research, 16, 317–331. http://dx.doi.org/10.1080/
10503300500268524
Bamonti, P. M., Keelan, C. M., Larson, N., Mentrikoski, J. M., Randall,
C. L., Sly, S. K., . . . McNeil, D. W. (2014). Promoting ethical behavior
by cultivating a culture of self-care during graduate training: A call to
action. Training and Education in Professional Psychology, 8, 253–260.
http://dx.doi.org/10.1037/tep0000056
Bangen, K. J., VanderVeen, J. W., Veilleux, J. C., Kamen, C., & Klonoff,
E. A. (2010). The graduate student viewpoint on internship prepared-
ness: A 2008 council of university directors of clinical psychology
student survey. Training and Education in Professional Psychology, 4,
219 –226. http://dx.doi.org/10.1037/a0020111
Barabasz, A. F., Baer, L., Sheehan, D. V., & Barabasz, M. (1986). A
three-year follow-up of hypnosis and restricted environmental stimula-
tion therapy for smoking. International Journal of Clinical and Exper-
imental Hypnosis, 34, 169 –181. http://dx.doi.org/10.1080/00207148
608406983
Bernard, J. M., & Goodyear, R. K. (2014). Fundamentals of clinical
supervision (5th ed.). Upper Saddle River, NJ: Merrill.
Bertsch, K. N., Bremer-Landau, J. D., Inman, A. G., DeBoer Kreider,
E. R., Price, T. A., & DeCarlo, A. L. (2014). Evaluation of the critical
events in supervision model using gender related events. Training and
Education in Professional Psychology, 8, 174 –181. http://dx.doi.org/10
.1037/tep0000039
Beutler, L. E., & Kendall, P. C. (1995). Introduction to the special section:
The case for training in the provision of psychological therapy. Journal
of Consulting and Clinical Psychology, 63, 179 –181. http://dx.doi.org/
10.1037/0022-006X.63.2.179
Bradley, J. R., & Olson, J. K. (1980). Training factors influencing felt
psychotherapeutic competence of psychology trainees. Professional
Psychology, 11, 930 –934. http://dx.doi.org/10.1037/0735-7028.11.6
.930
Callahan, J. L., Almstrom, C. M., Swift, J. K., Borja, S. E., & Heath, C. J.
(2009). Exploring the contribution of supervisors to intervention out-
comes. Training and Education in Professional Psychology, 3, 72–77.
http://dx.doi.org/10.1037/a0014294
Callahan, J. L., Collins, F. L., & Klonoff, E. A. (2010). An examination of
applicant characteristics of successfully matched interns: Is the glass half
full or half empty and leaking miserably? Journal of Clinical Psychol-
ogy, 66, 1–16. http://dx.doi.org/10.1002/jclp.20664
Callahan, J. L., Gustafson, S., Misner, J., Paprocki, C. M., Sauer, E.,
Saules, K. K., . . . Wise, E. H. (2014). Introducing the Association of
Psychology Training Clinics’ Collaborative Research Network: A study
on client expectancies. Training and Education in Professional Psychol-
ogy, 8, 95–104. http://dx.doi.org/10.1037/tep0000047
Callahan, J. L., Hogan, L. R., Klonoff, E. A., & Collins, F. L., Jr. (2014).
Predicting match outcomes: Science, practice, and personality. Training
257
and Education in Professional Psychology, 8, 68 – 82. http://dx.doi.org/
10.1037/tep0000030
Callahan, J. L., & Hynan, M. T. (2005). Models of psychotherapy outcome:
Are they applicable in training clinics? Psychological Services, 2, 65–
69. http://dx.doi.org/10.1037/1541-1559.2.1.65
Callahan, J. L., Neumann, C. S., Cox, R. J., & Ruggero, C. J. (2017). The
Supervisor Trainee Quarterly Evaluation (STQE): Psychometric support
for use as a measure of competency. Training and Education in Profes-
sional Psychology, 11, 117–127. http://dx.doi.org/10.1037/tep0000140
Callahan, J. L., Swift, J. K., Hogan, L. R., Tompkins, K. A., Connor, D. R.,
& Klonoff, E. A. (2014). Seeking the “perfect” match: An empirical
examination of applicant differences according to internship setting.
Psychotherapy Bulletin, 49, 13–18.
Callahan, J. L., & Watkins, C. E., Jr. (2018a). Evidence-based training: The
time has come. Training and Education in Professional Psychology, 12,
211–218. http://dx.doi.org/10.1037/tep0000204
Callahan, J. L., & Watkins, C. E., Jr. (2018b). The science of training I:
Admissions, curriculum, and research training. Training and Education
in Professional Psychology, 12, 219 –230. http://dx.doi.org/10.1037/
tep0000205
Callahan, J. L., & Watkins, C. E., Jr. (2018c). The science of training II:
Prepracticum and practicum training. Training and Education in Pro-
fessional Psychology, 12, 231–244. http://dx.doi.org/10.1037/
tep0000209
Carpenter, A. L., Pincus, D. B., Conklin, P. H., Wyszynski, C. M., Chu,
B. C., & Comer, J. S. (2016). Assessing cognitive-behavioral clinical
decision-making among trainees in the treatment of childhood anxiety.
Training and Education in Professional Psychology, 10, 109 –116.
http://dx.doi.org/10.1037/tep0000111
Carter, J. W., Enyedy, K. C., Goodyear, R. K., Arcinue, F., & Puri, N. N.
(2009). Concept mapping of the events supervisees find helpful in group
supervision. Training and Education in Professional Psychology, 3, 1–9.
http://dx.doi.org/10.1037/a0013656
Chung, Y. B., Marshall, J. A., & Gordon, L. L. (2001). Racial and gender
biases in supervisory evaluation and feedback. The Clinical Supervisor,
20, 99 –111. http://dx.doi.org/10.1300/J001v20n01_08
DeHay, T., Ross, S., Williams, J., & Ponce, A. N. (2016). Factors associ-
ated with accreditation: A comparison of accredited and nonaccredited
psychology doctoral internship programs. Training and Education in
Professional Psychology, 10, 125–132. http://dx.doi.org/10.1037/
tep0000124
DiLillo, D., & Gale, E. B. (2011). To Google or not to Google: Graduate
students’ use of the Internet to access personal information about clients.
Training and Education in Professional Psychology, 5, 160 –166. http://
dx.doi.org/10.1037/a0024441
Eisenhard, M. L., & Muse-Burke, J. L. (2015). A comparison of individual
supervision at forensic, inpatient, and college counseling internship sites.
Training and Education in Professional Psychology, 9, 61– 67. http://dx
.doi.org/10.1037/tep0000051
El-Ghoroury, N. H., Galper, D. I., Sawaqdeh, A., & Bufka, L. F. (2012).
Stress, coping, and barriers to wellness among psychology graduate
students. Training and Education in Professional Psychology, 6, 122–
134. http://dx.doi.org/10.1037/a0028768
Ellis, M. V. (2006). Critical incidents in clinical supervision and in super-
visor supervision: Assessing supervisory issues. Training and Education
in Professional Psychology, S, 122–132. http://dx.doi.org/10.1037/1931-
3918.S.2.122
Ellis, M. V. (2017). Narratives of harmful clinical supervision. The Clinical
Supervisor, 36, 20 – 87. http://dx.doi.org/10.1080/07325223.2017
.1297752
Enyedy, K. C., Arcinue, F., Puri, N. N., Carter, J. W., Goodyear, R. K., &
Getzelman, M. A. (2003). Hindering phenomena in group supervision:
Implications for practice. Professional Psychology, Research and Prac-
tice, 34, 312–317. http://dx.doi.org/10.1037/0735-7028.34.3.312
 258 CALLAHAN AND WATKINS
Fleming, L. M., Glass, J. A., Fujisaki, S., & Toner, S. L. (2010). Group
process and learning: A grounded theory model of group supervision.
Training and Education in Professional Psychology, 4, 194 –203. http://
dx.doi.org/10.1037/a0018970
Freitas, G. J. (2002). The impact of psychotherapy supervision on client
outcome: A critical examination of 2 decades of research. Psychother-
apy: Theory, Research, & Practice, 39, 354 –367. http://dx.doi.org/10
.1037/0033-3204.39.4.354
Gatmon, D., Jackson, D., Koshkarian, L., Martos-Perry, N., Molina, A.,
Patel, N., & Rodolfa, E. (2001). Exploring ethnic, gender, and sexual
orientation variables in supervision: Do they really matter? Journal of
Multicultural Counseling and Development, 29, 102–113. http://dx.doi
.org/10.1002/j.2161-1912.2001.tb00508.x
Genuchi, M. C., Rings, J. A., Germek, M. D., & Cornish, J. E. (2015).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Clinical supervisors’ perceptions of the clarity and comprehensiveness
This document is copyrighted by the American Psychological Association or one of its allied publishers.
of the supervision competencies framework. Training and Education in
Professional Psychology, 9, 68 –76. http://dx.doi.org/10.1037/
tep0000064
Goldberg, R. W., DeLamatre, J. E., & Young, K. (2011). Intern final oral
examinations: An exploration of alternative models of competency.
Training and Education in Professional Psychology, 5, 185–191. http://
dx.doi.org/10.1037/a0024151
Goldberg, R. W., & Young, K. R. (2015). Oral case examinations for
assessing intern competence. Training and Education in Professional
Psychology, 9, 242–247. http://dx.doi.org/10.1037/tep0000081
Goncher, I. D., Sherman, M. F., Barnett, J. E., & Haskins, D. (2013).
Programmatic perceptions of self-care emphasis and quality of life
among graduate trainees in clinical psychology: The mediational role of
self-care utilization. Training and Education in Professional Psychol-
ogy, 7, 53– 60. http://dx.doi.org/10.1037/a0031501
Gonsalvez, C. J., Bushnell, J., Blackman, R., Deane, F., Bliokas, V.,
Nicholson-Perry, K., . . . Knight, R. (2013). Assessment of psychology
competencies in field placements: Standardized vignettes reduce rater
bias. Training and Education in Professional Psychology, 7, 99 –111.
http://dx.doi.org/10.1037/a0031617
Gonsalvez, C. J., & Freestone, J. (2007). Field supervisors’ assessments of
trainee performance: Are they reliable and valid? Australian Psycholo-
gist, 42, 23–32. http://dx.doi.org/10.1080/00050060600827615
Goodyear, R. K., & Guzzardo, C. R. (2000). Psychotherapy supervision
and training. In S. D. Brown & R. W. Lent (Eds.), Handbook of
counseling psychology (3rd ed., pp. 83–108). New York, NY: Wiley.
Gray, L. A., Ladany, N., Walker, J. A., & Ancis, J. R. (2001). Psycho-
therapy trainees’ experience of counterproductive events in supervision.
Journal of Counseling Psychology, 48, 371–383. http://dx.doi.org/10
.1037/0022-0167.48.4.371
Grigg, A. E. (1961). Client response to counselors at different levels of
experience. Journal of Counseling Psychology, 8, 217–223. http://dx.doi
.org/10.1037/h0048978
Grossl, A. B., Reese, R. J., Norsworthy, L. A., & Hopkins, N. B. (2014).
Client feedback data in supervision: Effects on supervision and outcome.
Training and Education in Professional Psychology, 8, 182–188. http://
dx.doi.org/10.1037/tep0000025
Grote, C. L., Robiner, W. N., & Haut, A. (2001). Disclosure of negative
information in letters of recommendation: Writers’ intentions and read-
ers’ experiences. Professional Psychology: Research and Practice, 32,
655– 661. http://dx.doi.org/10.1037/0735-7028.32.6.655
Grus, C. L., Falender, C., Fouad, N. A., & Lavelle, A. K. (2016). A culture
of competence: A survey of implementation of competency-based edu-
cation and assessment. Training and Education in Professional Psychol-
ogy, 10, 198 –205. http://dx.doi.org/10.1037/tep0000126
Gunn, J. E., & Pistole, M. C. (2012). Trainee supervisor attachment:
Explaining the alliance and disclosure in supervision. Training and
Education in Professional Psychology, 6, 229 –237. http://dx.doi.org/10
.1037/a0030805
Hatcher, R. L. (2015). The internship match: New perspectives from
longitudinal data. Training and Education in Professional Psychology,
9, 292–299. http://dx.doi.org/10.1037/tep0000097
Hatcher, R. L., Grus, C. L., & Wise, E. H. (2011). Administering practicum
training: A survey of graduate programs’ policies and procedures. Train-
ing and Education in Professional Psychology, 5, 244 –252. http://dx
.doi.org/10.1037/a0025088
Hatcher, R. L., Wise, E. H., Grus, C. L., Mangione, L., & Emmons, L.
(2012). Inside the practicum in professional psychology: A survey of
practicum site coordinators. Training and Education in Professional
Psychology, 6, 220 –228. http://dx.doi.org/10.1037/a0029542
Henry, W. P., Schacht, T. E., Strupp, H. H., Butler, S. F., & Binder, J. L.
(1993). Effects of training in time-limited dynamic psychotherapy: Me-
diators of therapists’ responses to training. Journal of Consulting and
Clinical Psychology, 61, 441– 447. http://dx.doi.org/10.1037/0022-006X
.61.3.441
Heppner, P. P., & Heesacker, M. (1983). Perceived counselor characteris-
tics, client expectations, and client satisfaction with counseling. Journal
of Counseling Psychology, 30, 31–39. http://dx.doi.org/10.1037/0022-
0167.30.1.31
Heru, A. M., Strong, D., Price, M., & Recupero, P. R. (2006). Self-
disclosure in psychotherapy supervisors: Gender differences. American
Journal of Psychotherapy, 60, 323–334. http://dx.doi.org/10.1176/appi
.psychotherapy.2006.60.4.323
Hogan, L. R., Callahan, J. L., Tompkins, K. A., Swift, J. K., Connor, D. R.,
& Klonoff, E. A. (2014). Subjective well-being among internship appli-
cants. Psychotherapy Bulletin, 49, 13–16.
Holloway, E. L. (2012). Professional competence in supervision. In J. N.
Fuertes, A. Spokane, & E. L. Holloway (Eds.), The professional com-
petencies in counseling psychology (pp. 165–181). New York, NY:
Oxford University Press.
Holloway, E. L., & Johnston, R. (1985). Group supervision: Widely
practiced but poorly understood. Counselor Education and Supervision,
24, 332–340. http://dx.doi.org/10.1002/j.1556-6978.1985.tb00494.x
Holloway, E. L., & Neufeldt, S. A. (1995). Supervision: Its contributions
to treatment efficacy. Journal of Consulting and Clinical Psychology,
63, 207–213. http://dx.doi.org/10.1037/0022-006X.63.2.207
Huber, D. M., Sauer, E. M., Mrdjenovich, A. J., & Gugiu, P. C. (2010).
Contributions to advisory working alliance: Advisee attachment orien-
tation and pairing methods. Training and Education in Professional
Psychology, 4, 244 –253. http://dx.doi.org/10.1037/a0019213
Humphreys, L., Crino, R., & Wilson, I. (2017). Psychological functioning
predicts competence development for postgraduate students of profes-
sional psychology. Training and Education in Professional Psychology,
11, 49 –56. http://dx.doi.org/10.1037/tep0000139
Inman, A. G., & Ladany, N. (2008). Research: The state of the field. In
A. K. Hess, K. D. Hess, & T. H. Hess (Eds.), Psychotherapy supervi-
sion: Theory, research, and practice (2nd ed., pp. 500 –517). Hoboken,
NJ: Wiley.
Inman, A. G., & Ladany, N. (2014). Multicultural competencies in psy-
chotherapy supervision. In F. T. L. Leong, L. Comas-Díaz, G. C.
Nagayama Hall, V. C. McLoyd, & J. E. Trimble (Eds.), APA handbook
of multicultural psychology, Vol. 2. Applications and training (pp.
643– 658). Washington, DC: American Psychological Association.
http://dx.doi.org/10.1037/14187-036
January, A. M., Meyerson, D. A., Reddy, L. F., Docherty, A. R., &
Klonoff, E. A. (2014). Impressions of misconduct: Graduate students’
perception of faculty ethical violations in scientist-practitioner clinical
psychology programs. Training and Education in Professional Psychol-
ogy, 8, 261–268. http://dx.doi.org/10.1037/tep0000059
Jernigan, M. M., Green, C. E., Helms, J. E., Perez-Gualdron, L., & Henze,
K. (2010). An examination of people of color supervision dyads: Racial
identity matters as much as race. Training and Education in Professional
Psychology, 4, 62–73. http://dx.doi.org/10.1037/a0018110
 SCIENCE OF TRAINING III
Kamen, C., Veilleux, J. C., Bangen, K. J., VanderVeen, J. W., & Klonoff,
E. A. (2010). Climbing the stairway to competency: Trainee perspectives
on competency development. Training and Education in Professional
Psychology, 4, 227–234. http://dx.doi.org/10.1037/a0021092
Kaslow, N. J., Rubin, N. J., Bebeau, M. J., Leigh, I. W., Lichtenberg, J. W.,
Nelson, P. D., . . . Smith, I. L. (2007). Guiding principles and recom-
mendations for the assessment of competence. Professional Psychology,
38, 441– 451. http://dx.doi.org/10.1037/0735-7028.38.5.441
Keenan-Miller, D., & Corbett, H. I. (2015). Metasupervision: Can students
be safe and effective supervisors? Training and Education in Profes-
sional Psychology, 9, 315–321. http://dx.doi.org/10.1037/tep0000090
Klonoff, E. A. (2015). Assessing competencies: “What have you done for
me lately?” Clinical Psychology: Science and Practice, 22, 404 – 409.
http://dx.doi.org/10.1111/cpsp.12126
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Krauskopf, C. J., Baumgardner, A., & Mandracchia, S. (1981). Return rate
This document is copyrighted by the American Psychological Association or one of its allied publishers.
following intake revisited. Journal of Counseling Psychology, 28, 519 –
521. http://dx.doi.org/10.1037/0022-0167.28.6.519
LaBoskey, V. K. (2006). The fragile strengths of self-study: Making bold
claims and clear connections. In P. Aubusson & S. Schuck (Eds.),
Teaching learning and development: The mirror maze (pp. 251–262).
Dordrecht, the Netherlands: Springer. http://dx.doi.org/10.1007/978-1-
4020-4642-1_16
Ladany, N., Ellis, M. V., & Friedlander, M. L. (1999). The supervisory
working alliance, trainee self-efficacy, and satisfaction. Journal of
Counseling and Development, 77, 447– 455. http://dx.doi.org/10.1002/j
.1556-6676.1999.tb02472.x
Ladany, N., & Inman, A. G. (2011). Training and supervision. In E.
Altmaier & J. C. Hansen (Eds.), Oxford handbook of counseling psy-
chology (pp. 179 –207). New York, NY: Oxford University Press.
Ladany, N., Lehrman-Waterman, D., Molinaro, M., & Wolgast, B. (1999).
Psychotherapy supervisor ethical practices: Adherence to guidelines, the
supervisory working alliance, and supervisee satisfaction. The Counsel-
ing Psychologist, 27, 443– 475. http://dx.doi.org/10.1177/0011000
099273008
Lambert, M. J. (2017). Maximizing psychotherapy outcome beyond
evidence-based medicine. Psychotherapy and Psychosomatics, 86, 80 –
89. http://dx.doi.org/10.1159/000455170
Lappan, S., Shamoon, Z., & Blow, A. (2017). The importance of adoption
of formal client feedback in therapy: A narrative review. Journal of
Family Therapy. Advance online publication. http://dx.doi.org/10.1111/
1467-6427.12183
Larkin, K. T., & Morris, T. L. (2015). The process of competency acqui-
sition during doctoral training. Training and Education in Professional
Psychology, 9, 300 –308. http://dx.doi.org/10.1037/tep0000091
Lopez, S. J., Oehlert, M. E., & Moberly, R. L. (1996). Selection criteria for
American Psychological Association-accredited internship programs: A
survey of training directors. Professional Psychology: Research and
Practice, 27, 518 –520. http://dx.doi.org/10.1037/0735-7028.27.5.518
Lund, E. M., Bouchard, L. M., & Thomas, K. B. (2016). Publication
productivity of professional psychology internship applicants: An in-
depth analysis of APPIC survey data. Training and Education in Pro-
fessional Psychology, 10, 54 – 60. http://dx.doi.org/10.1037/tep0000105
Madson, M. B., Aten, J. D., & Leach, M. M. (2007). Applying for the
predoctoral internship: Training program strategies to help students
prepare. Training and Education in Professional Psychology, 1, 116 –
124. http://dx.doi.org/10.1037/1931-3918.1.2.116
Mearns, J., & Allen, G. J. (1991). Graduate students’ experiences in
dealing with impaired peers, compared with faculty predictions: An
exploratory study. Ethics & Behavior, 1, 191–202. http://dx.doi.org/10
.1207/s15327019eb0103_3
Mehr, K. E., Ladany, N., & Caskie, G. L. (2015). Factors influencing
trainee willingness to disclose in supervision. Training and Education in
Professional Psychology, 9, 44 –51. http://dx.doi.org/10.1037/tep
0000028
259
Miller, R. K., & Van Rybroek, G. J. (1988). Internship letters of recom-
mendation: Where are the other 90%? Professional Psychology: Re-
search and Practice, 19, 115–117. http://dx.doi.org/10.1037/0735-7028
.19.1.115
Milne, D. L., & James, I. A. (2002). The observed impact of training on
competence in clinical supervision. British Journal of Clinical Psychol-
ogy, 41, 55–72. http://dx.doi.org/10.1348/014466502163796
Mours, J. M., Campbell, C. D., Gathercoal, K. A., & Peterson, M. (2009).
Training in the use of psychotherapy outcome assessment measures at
psychology internship sites. Training and Education in Professional
Psychology, 3, 169 –176. http://dx.doi.org/10.1037/a0016135
Myers, S. B., Sweeney, A. C., Popick, V., Wesley, K., Bordfeld, A., &
Fingerhut, R. (2012). Self-care practices and perceived stress levels
among psychology graduate students. Training and Education in Pro-
fessional Psychology, 6, 55– 66. http://dx.doi.org/10.1037/a0026534
Newell, M. L. (2010a). Exploring the use of computer simulation to
evaluate the implementation of problem-solving consultation. Journal of
Educational and Psychological Consultation, 20, 228 –255. http://dx.doi
.org/10.1080/10474412.2010.500511
Newell, M. L. (2010b). The implementation of problem-solving consulta-
tion: An analysis of problem conceptualization in a multiracial context.
Journal of Educational and Psychological Consultation, 20, 83–105.
http://dx.doi.org/10.1080/10474411003785529
Newell, M. L. (2012). Transforming knowledge to skill: Evaluating the
consultation competence of novice school-based consultants. Consulting
Psychology Journal: Practice and Research, 64, 8 –28. http://dx.doi.org/
10.1037/a0027741
Newell, M. L., Newell, T. S., & Looser, J. (2013). A competency-based
assessment of school-based consultants’ implementation of consultation.
Training and Education in Professional Psychology, 7, 235–245. http://
dx.doi.org/10.1037/a0033067
Nyman, S. J., Nafziger, M. A., & Smith, T. B. (2010). Client outcomes
across counselor training level within a multitiered supervision model.
Journal of Counseling and Development, 88, 204 –209. http://dx.doi.org/
10.1002/j.1556-6678.2010.tb00010.x
Okiishi, J. C., Lambert, M. J., Eggett, D., Nielsen, L., Dayton, D. D., &
Vermeersch, D. A. (2006). An analysis of therapist treatment effects:
Toward providing feedback to individual therapists on their clients’
psychotherapy outcome. Journal of Clinical Psychology, 62, 1157–
1172. http://dx.doi.org/10.1002/jclp.20272
Oliver, M. N. I., Bernstein, J. H., Anderson, K. G., Blashfield, R. K., &
Roberts, M. C. (2004). An exploratory examination of student attitudes
toward ‘impaired’ peers in Clinical Psychology Training Programs.
Professional Psychology: Research and Practice, 35, 141–147. http://
dx.doi.org/10.1037/0735-7028.35.2.141
Pakenham, K. I. (2017). Training in acceptance and commitment therapy
fosters self-care in clinical psychology trainees. Clinical Psychologist,
21, 186 –194. http://dx.doi.org/10.1111/cp.12062
Parent, M. C., & Williamson, J. B. (2010). Program disparities in un-
matched internship applicants. Training and Education in Professional
Psychology, 4, 116 –120. http://dx.doi.org/10.1037/a0018216
Peterson, A. P., & Fagan, C. (2017). Training the next generation in routine
outcome monitoring: Current practices in psychology training clinics.
Training and Education in Professional Psychology, 11, 182–189. http://
dx.doi.org/10.1037/tep0000148
Phillips, J. C., Parent, M. C., Dozier, V. C., & Jackson, P. L. (2017). Depth
of discussion of multicultural identities in supervision and supervisory
outcomes. Counselling Psychology Quarterly, 30, 188 –210. http://dx
.doi.org/10.1080/09515070.2016.1169995
Phillips, J. C., Szymanski, D. M., Ozegovic, J. J., & Briggs-Phillips, M.
(2004). Preliminary examination and measurement of the internship
research training environment. Journal of Counseling Psychology, 51,
240 –248. http://dx.doi.org/10.1037/0022-0167.51.2.240
 260 CALLAHAN AND WATKINS
Price, S. D., Callahan, J. L., & Cox, R. J. (2017). Psychometric investiga-
tion of competency benchmarks. Training and Education in Profes-
sional Psychology, 11, 128 –139. http://dx.doi.org/10.1037/tep0000133
Reese, R. J., Usher, E. L., Bowman, D. C., Norsworthy, L. A., Halstead,
J. L., Rowlands, S. R., & Chisholm, R. R. (2009). Using client feedback
in psychotherapy training: An analysis of its influence on supervision
and counselor self-efficacy. Training and Education in Professional
Psychology, 3, 157–168. http://dx.doi.org/10.1037/a0015673
Reiser, R. P., & Milne, D. L. (2014). A systematic review and reformula-
tion of outcome evaluation in clinical supervision: Applying the fidelity
framework. Training and Education in Professional Psychology, 8,
149 –157. http://dx.doi.org/10.1037/tep0000031
Rieck, T., & Callahan, J. L. (2013). Emotional intelligence and psycho-
therapy outcomes in the training clinic. Training and Education in
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Professional Psychology, 7, 42–52. http://dx.doi.org/10.1037/a0031659
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Rieck, T., Callahan, J. L., & Watkins, C. E., Jr. (2015). Clinical supervi-
sion: An exploration of possible mechanisms of action. Training and
Education in Professional Psychology, 9, 187–194. http://dx.doi.org/10
.1037/tep0000080
Riggs, S. A., & Bretz, K. M. (2006). Attachment processes in the super-
visory relationship: An exploratory investigation. Professional Psychol-
ogy: Research and Practice, 37, 558 –566. http://dx.doi.org/10.1037/
0735-7028.37.5.558
Rings, J. A., Genuchi, M. C., Hall, M. D., Angelo, M., & Cornish, J. E.
(2009). Is there consensus among predoctoral internship training direc-
tors regarding clinical supervision competencies? A descriptive analysis.
Training and Education in Professional Psychology, 3, 140 –147. http://
dx.doi.org/10.1037/a0015054
Riva, M. T., & Erickson Cornish, J. A. (2008). Group supervision practices
at psychology predoctoral internship programs: 15 years later. Training
and Education in Professional Psychology, 2, 18 –25. http://dx.doi.org/
10.1037/1931-3918.2.1.18
Roberts, M. C., Borden, K. A., Christiansen, M., & Lopez, S. J. (2005).
Fostering a culture shift: assessment of competence in the education and
careers of professional psychologists. Professional Psychology, Re-
search and Practice, 36, 355–361. http://dx.doi.org/10.1037/0735-7028
.36.4.355
Roberts, R. M., & Dorstyn, D. S. (2017). Does changing grade-based
marking to pass-fail marking change student achievement in a postgrad-
uate psychology course?. Training and Education in Professional Psy-
chology, 11, 57– 60. http://dx.doi.org/10.1037/tep0000129
Rodolfa, E. R., Owen, J. J., & Clark, S. (2007). Practicum training hours:
Fact and fantasy. Training and Education in Professional Psychology, 1,
64 –73. http://dx.doi.org/10.1037/1931-3918.1.1.64
Rodriguez-Menendez, G., Dempsey, J. P., Albizu, T., Power, S., & Camp-
bell Wilkerson, M. (2017). Faculty and student perceptions of clinical
training experiences in professional psychology. Training and Educa-
tion in Professional Psychology, 11, 1–9. http://dx.doi.org/10.1037/
tep0000137
Rosenberg, J. I., Getzelman, M. A., Arcinue, F., & Oren, C. Z. (2005). An
exploratory look at students’ experiences of problematic peers in Aca-
demic Professional Psychology Programs. Professional Psychology: Re-
search and Practice, 36, 665– 673. http://dx.doi.org/10.1037/0735-7028
.36.6.665
Rousmaniere, T. G., Swift, J. K., Babins-Wagner, R., Whipple, J. L., &
Berzins, S. (2016). Supervisor variance in psychotherapy outcome in
routine practice. Psychotherapy Research, 26, 196 –205. http://dx.doi
.org/10.1080/10503307.2014.963730
Santana, M. C., & Fouad, N. A. (2017). Development and validation of a
self-care behavior inventory. Training and Education in Professional
Psychology, 11, 140 –145. http://dx.doi.org/10.1037/tep0000142
Schaffer, J. B., Rodolfa, E., Owen, J., Lipkins, R., Webb, C., & Horn, J.
(2012). The Examination for Professional Practice in Psychology: New
data–practical implications. Training and Education in Professional
Psychology, 6, 1–7. http://dx.doi.org/10.1037/a0026823
Scott, K. J., Ingram, K. M., Vitanza, S. A., & Smith, N. G. (2000). Training
in supervision: A survey of current practices. The Counseling Psychol-
ogist, 28, 403– 422. http://dx.doi.org/10.1177/0011000000283007
Seawell, D. B., Krohn, N., Gorgens, K. A., & Erickson Cornish, J. A.
(2009). Geography and internship match rates: Quantifying competitive-
ness and discussing implications for the internship imbalance. Training
and Education in Professional Psychology, 3, 127–134. http://dx.doi
.org/10.1037/a0015298
Shen-Miller, D. S., Forrest, L., & Burt, M. (2012). Contextual influences
on faculty diversity conceptualizations when working with trainee com-
petence problems. The Counseling Psychologist, 40, 1181–1219. http://
dx.doi.org/10.1177/0011000011431832
Shen-Miller, D. S., Grus, C. L., Van Sickle, K. S., Schwartz-Mette, R.,
Cage, E. A., Elman, N. S., . . . Kaslow, N. J. (2011). Trainees’ experi-
ences with peers having competence problems: A national survey. Train-
ing and Education in Professional Psychology, 5, 112–121. http://dx.doi
.org/10.1037/a0023824
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing
treatment outcome of patients at risk of treatment failure: Meta-analytic
and mega-analytic review of a psychotherapy quality assurance system.
Journal of Consulting and Clinical Psychology, 78, 298 –311. http://dx
.doi.org/10.1037/a0019247
Shivy, V. A., Mazzeo, S. E., & Sullivan, T. N. (2007). Clinical and
counseling psychology doctoral trainnees: How students perceive intern-
ships. Training and Education in Professional Psychology, 1, 163–173.
http://dx.doi.org/10.1037/1931-3918.1.3.163
Smith, R. D., Riva, M. T., & Erickson Cornish, J. A. (2012). The ethical
practice of group supervision: A national survey. Training and Educa-
tion in Professional Psychology, 6, 238 –248. http://dx.doi.org/10.1037/
a0030806
Sobell, L. C., Manor, H. L., Sobell, M. B., & Dum, M. (2008). Self-
critiques of audiotaped therapy sessions: A motivational procedure for
facilitating feedback during supervision. Training and Education in
Professional Psychology, 2, 151–155. http://dx.doi.org/10.1037/1931-
3918.2.3.151
Soheilian, S. S., Inman, A. G., Klinger, R. S., Isenberg, D. S., & Kulp, L. E.
(2014). Multicultural supervision: Supervisees’ reflections on culturally
competent supervision. Counselling Psychology Quarterly, 27, 379 –
392. http://dx.doi.org/10.1080/09515070.2014.961408
Stedman, J. M., Hatch, J. P., & Schoenfeld, L. S. (2009). Letters of
recommendation for the predoctoral internship in medical schools and
other settings: Do they enhance decision making in the selection pro-
cess? Journal of Clinical Psychology in Medical Settings, 16, 339 –345.
http://dx.doi.org/10.1007/s10880-009-9170-y
Stedman, J. M., Schoenfeld, L. S., & O’Donnell, L. (2013). An investiga-
tion of internship directors’ perspectives on the learning objectives
required by the commission on accreditation. Training and Education in
Professional Psychology, 7, 134 –138. http://dx.doi.org/10.1037/
a0031660
Strom, T. Q., Arbisi, P., Thuras, P. D., Leskela, J., Finn, J., Anderson, C.,
& Siegel, W. (2016). Summative intern performance evaluations: A
wealth of information or a gathering of “two cents”? Training and
Education in Professional Psychology, 10, 78 – 83. http://dx.doi.org/10
.1037/tep0000114
Swift, J. K., Callahan, J. L., Rousmaniere, T. G., Whipple, J. L., Dexter, K.,
& Wrape, E. R. (2015). Using client outcome monitoring as a tool for
supervision. Psychotherapy: Theory, Research, & Practice, 52, 180 –
184. http://dx.doi.org/10.1037/a0037659
Szymanski, D. M., Ozegovic, J. J., Phillips, J. C., & Briggs-Phillips, M.
(2007). Fostering scholarly productivity through academic and intern-
ship research training environments. Training and Education in Profes-
 SCIENCE OF TRAINING III
sional Psychology, 1, 135–146. http://dx.doi.org/10.1037/1931-3918.1.2
.135
Tanner, M. A., Gray, J. J., & Haaga, D. A. F. (2012). Association of
cotherapy supervision with client outcomes, attrition, and trainee effec-
tiveness in a psychotherapy training clinic. Journal of Clinical Psychol-
ogy, 68, 1241–1252. http://dx.doi.org/10.1002/jclp.21902
Tracey, T. J. G., Wampold, B. E., Lichtenberg, J. W., & Goodyear, R. K.
(2014). Expertise in psychotherapy: An elusive goal? American Psychol-
ogist, 69, 218 –229. http://dx.doi.org/10.1037/a0035099
Tracy, E. N., Bucchianeri, M. M., & Rodolfa, E. R. (2011). Internship
hours revisited: Further evidence for a national standard. Training and
Education in Professional Psychology, 5, 97–101. http://dx.doi.org/10
.1037/a0023294
Tsong, Y., & Goodyear, R. K. (2014). Assessing supervision’s clinical and
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
multicultural impacts: The Supervision Outcome Scale’s psychometric
This document is copyrighted by the American Psychological Association or one of its allied publishers.
properties. Training and Education in Professional Psychology, 8, 189 –
195. http://dx.doi.org/10.1037/tep0000049
Tucker, R. P., Gregus, S. J., Seivert, N. P., Stevens, K. T., & Callahan, J. L.
(2018). Student perceptions of multicultural training and program cli-
mate in clinical psychology doctoral programs. Manuscript submitted
for publication.
Vacha-Haase, T., Davenport, D. S., & Kerewsky, S. D. (2004). Problematic
students: Gatekeeping practices of Academic Professional Psychology
Programs. Professional Psychology: Research and Practice, 35, 115–
122. http://dx.doi.org/10.1037/0735-7028.35.2.115
Veilleux, J. C., January, A. M., VanderVeen, J. W., Reddy, L. F., &
Klonoff, E. A. (2012a). Differentiating amongst characteristics associ-
ated with problems of professional competence: Perceptions of graduate
student peers. Training and Education in Professional Psychology, 6,
113–121. http://dx.doi.org/10.1037/a0028337
Veilleux, J. C., January, A. M., VanderVeen, J. W., Reddy, L. F., &
Klonoff, E. A. (2012b). Perceptions of climate in clinical psychology
doctoral programs: Development and initial validation of the Graduate
Program Climate Scale. Training and Education in Professional Psy-
chology, 6, 211–219. http://dx.doi.org/10.1037/a0030303
Walker, J. A., Ladany, N., & Pate-Carolan, L. M. (2007). Gender-related
events in psychotherapy supervision: Female trainee perspectives. Coun-
selling & Psychotherapy Research, 7, 12–18. http://dx.doi.org/10.1080/
14733140601140881
Watkins, C. E., Jr. (2011). Does psychotherapy supervision contribute to
client outcomes? Considering thirty years of research. The Clinical
Supervisor, 30, 235–256. http://dx.doi.org/10.1080/07325223.2011
.619417
261
Wheeler, S., & Richards, K. (2007). The impact of clinical supervision on
counselors and therapists, their practice and their clients: A systematic
review of the literature. Counselling & Psychotherapy Research, 7,
54 – 65. http://dx.doi.org/10.1080/14733140601185274
White, E., & Winstanley, J. (2010). A randomised controlled trial of
clinical supervision: Selected findings from a novel Australian attempt
to establish the evidence base for causal relationships with quality of
care and patient outcomes, as an informed contribution to mental health
nursing practice development. Journal of Research in Nursing, 15,
151–167. http://dx.doi.org/10.1177/1744987109357816
Williams, J. D., & Cornish, J. A. E. (2015). Responding to clinical
psychology practicum competencies: An APPIC perspective. Clinical
Psychology: Science and Practice, 22, 410 – 413. http://dx.doi.org/10
.1111/cpsp.12127
Wolfe-Christensen, C., & Callahan, J. L. (2008). Current state of standard-
ization adherence: A reflection of competency in psychological assess-
ment. Training and Education in Professional Psychology, 2, 111–116.
http://dx.doi.org/10.1037/1931-3918.2.2.111
Worthington, E. L. (1987). Changes in supervision as counselors and
supervisors gain experience: A review. Professional Psychology, Re-
search and Practice, 18, 189 –208. http://dx.doi.org/10.1037/0735-7028
.18.3.189
Wrape, E. R., Callahan, J. L., Rieck, T., & Watkins, C. E., Jr. (2017).
Attachment theory within clinical supervision: Application of the con-
ceptual to the empirical. Psychoanalytic Psychotherapy, 31, 37–54.
http://dx.doi.org/10.1080/02668734.2016.1261927
Wrape, E. R., Callahan, J. L., Ruggero, C. J., & Watkins, C. E., Jr. (2015).
An exploration of faculty supervisor variables and their impact on client
outcome. Training and Education in Professional Psychology, 9, 35– 43.
http://dx.doi.org/10.1037/tep0000014
Yourman, D. B., & Farber, B. A. (1996). Nondisclosure and distortion in
psychotherapy supervison. Psychotherapy: Theory, Research, & Prac-
tice, 33, 567–575. http://dx.doi.org/10.1037/0033-3204.33.4.567
Zahniser, E., Rupert, P. A., & Dorociak, K. E. (2017). Self-care in clinical
psychology graduate training. Training and Education in Professional
Psychology, 11, 283–289. http://dx.doi.org/10.1037/tep0000172
Zeichner, K. (2007). Accumulating knowledge across self-studies in
teacher education. Journal of Teacher Education, 58, 36 – 46. http://dx
.doi.org/10.1177/0022487106296219
Received November 2, 2017
Revision received April 23, 2018
Accepted April 28, 2018 䡲