From the Laboratory to the Therapy Room
National Dissemination and Implementation of
Evidence-Based Psychotherapies in the
U.S. Department of Veterans Affairs Health Care System
Bradley E. Karlin
Gerald Cross
Despite their established efficacy and recommendation—
oflen as first-line treatments—in cliniccd practice guide-
lines, evidence-based psychotherapies (EBPs) have largely
failed to make their way into mainstream clinical settings.
Numerous attempts over the years to promote the transla-
tion of EBPs from science to practice, typically relying on
one-dimensional dissemination approaches, have yielded
limited success. As part of the transformation of its mental
health care system, the Veterans Health Administration
(VHA) of the U,S. Department of Veterans Affairs (VA) is
working to disseminate and implement a number of EBPs
for various mental and behavioral health conditions
throughout the VA health care system. This article exam-
ines VHA's multidimensional model and specific strategies,
involving policy, provider, local systems, patient, and ac-
countability levels, for promoting the national dissemina-
tion and implementation of EBPs in VHA. In addition, the
article identifies key lessons learned and next steps for
further promoting EBP delivery and sustainability in the
VA health care system. Beyond promoting the availabil-
ity of effective treatments for veterans returning from
Iraq and Afghanistan and for veterans of previous com-
bat eras, VHA's EBP dissemination and implementation
model and key lessons learned may help to inform other
private and public health care systems interested in
disseminating and implementing EBPs,
Keywords: evidence-based psychotherapy, dissemination,
implementation, sustainability, veterans
O
ver the past 30 years, psychotherapeutic inter-
ventions have been developed and empirically
validated for a wide range of psychological con-
ditions, including conditions that not too long ago were
considered untreatable, such as posttraumatic stress disor-
der (PTSD). For many mental and behavioral health disor-
ders, specific evidence-based psychotherapies (EBPs; i.e.,
psychotherapies shown to be efficacious in randomized
controlled trials) have been shown to be at least as effective
as psychopharmacotherapies (Butler, Chapman, Forman, &
Beck, 2006; DeRubeis, Gelfand, Tang, & Simons, 1999;
Jacobs, Pace-Schott, Stickgold, & Otto, 2004). And, for
some conditions, such as PTSD and other anxiety disor-
ders, EBPs have been found to have greater empirical
January 2014 • American Psychologist
In the public domain
Vol. 69. No. 1. 19-33 DOI: lü.l037/a0O33888
U. S. Department of Veterans Affairs Central Office and
Johns Hopkins University
U. S. Department of Veterans Affairs Central Office
support and to have more enduring effects than medications
(Butler et al, 2006; Institute of Medicine, 2007). Conse-
quently, specific EBPs are highly recommended in clinical
practice guidelines and are recognized in some contexts as
first-line treatments (e.g.. National Institute for Clinical
Excellence, 2005; National Institutes of Health, 2005; U.S.
Department of Veterans Affairs & U.S. Department of
Defense, 2010).
Notwithstanding their established efficacy and recom-
mendation in clinical practice guidelines, EBPs have
largely remained within the confines of clinical and aca-
demic research settings and have not crossed the bridge
into community practice settings in public and private
mental health care systems (Freiheit, Vye, Swan, & Cady,
2004; McAlpine, Schroder, Pankratz, & Maurer, 2004;
Rosen et al., 2004). For example, a survey conducted by
Rosen and colleagues (2004) of mental health providers at
six sites found that fewer than 10% of PTSD specialists and
generaUst mental health providers routinely provided evi-
dence-based psychological treatments for PTSD. The sci-
ence-to-practice lag time for EBPs is, in fact, longer (and in
many cases, far longer) than the typical lengthy lag time of
This article was published Online First September 2, 2013.
Bradley E. Karlin, U.S. Department of Veterans Affairs Central Office,
Washington, DC, and Bloomberg School of Public Health, Johns Hopkins
University; Gerald Cross, U.S. Department of Veterans Affairs Central
Office, Washington, DC.
Portions of this article were presented in August 2011 at the annual
convention of the American Psychological Association in Washington,
DC, and in November 2011 at the annual convention of the Association
for Behavioral and Cognitive Therapies in Toronto, Ontario, Canada.
We wish to note that the efforts described in this article are the result
of the dedication of individuals and of specific actions taken at multiple
levels of the Veterans Health Administration (VHA). It is the collective
commitment of these individuals and the organization that has made the
dissemination and implementation of evidence-based psychotherapies in
VHA possible. We specifically thank Ira Katz, Antonette Zeiss, and
Madhulika Agarwal for their support for the dissemination and imple-
mentation of evidence-based psychotherapies in VHA. The opinions ex-
pressed in this manuscript do not necessarily represent the official policy
position of the Department of Veterans Affairs (VA).
Correspondence concerning this article should be addressed to Brad-
ley E. Karlin, National Mental Health Director for Psychotherapy and
Psychogeriatrics, Mental Health Services (10P4M), U.S. Department of
Veterans Affairs Central Office, 810 Vermont Avenue, NW, Washington,
DC, 20420. E-mail: bradley.karlin2@va.gov
19

Bradley E.
Karlin
15 to 20 years identified for medical interventions by the
Institute of Medicine (2001) in its seminal report Crossing
the Quality Chasm: A New Health System for the 21st
Century. Moreover, research has shown that when EBPs
are delivered, they are often delivered with low fidelity to
the established psychotherapeutic model (Madson &
Campbell, 2006). As a result, many patients who do receive
EBPs may not receive the true treatment that has been
shown to be effective.
The foregoing delivery and fidelity gaps were perhaps
described most poignantly by the President's New Freedom
Commission on Mental Health (2003), which was charged
by then-President George W. Bush with examining the
state of mental health care delivery (public and private) in
the nation. In its final report, the President's New Freedom
Commission (2003) concluded that fundamental transfor-
mation of mental health care delivery in the United States
is needed and expressed particular concerns with the lim-
ited delivery and low fidelity of evidence-based psycho-
logical treatments:
Over the years, research has yielded important advances in our
knowledge of the brain, behavior, and effective treatments and
service delivery strategies for many mental disorders. An array of
evidence-based medications and psychosocial interventions—typ-
ically used together—now allows successful treatment of most
mental disorders. Despite these advances in science, many Amer-
icans are not benefiting from these investments.
Far too often, treatments and services based on rigorous clinical
research languish for years rather than being used effectively at
the earliest opportunity.. . . Even when these discoveries become
routinely available at the community level, too often the clinical
practice is highly uneven and inconsistent with the original treat-
ment model that was shown to be effective, (p. 67)
From a reimbursement perspective, the Patient Pro-
tection and Affordable Care Act (2010) has placed addi-
20
tional emphasis on the delivery of evidence-based health
and mental health care, in that it requires that covered
treatments meet standards for safety and efficacy. In addi-
tion. Medicare is increasingly moving toward a pay for
performance model.
From Recommendation to Reality
In an effort to bridge the science-to-practice gap and realize
the full potential of EBPs, the Veterans Health Adminis-
tration (VHA), the health care arm of the U.S. Department
of Veterans Affairs (VA), has developed and has been
actively implementing a national initiative to disseminate
and implement EBPs. The purpose of the present article is
to examine VHA's multidimensional model and specific
strategies for promoting the national dissemination and
implementation of EBPs, which were informed by dissem-
ination and implementation science. In addition, the article
identifies key lessons learned and next steps for further
promoting and sustaining EBP delivery.
Significantly, the dissemination and implementation
of EBPs is a key component of a broader focus to transform
the VA mental health care system to an evidence-based and
recovery-oriented system of care (Zeiss & Karlin, 2008).
This transformation initiative began in 2005, guided by a
VHA Comprehensive Mental Health Strategic Plan devel-
oped in response to the President's New Freedom Com-
mission's recommendations for fundamental transforma-
tion of public and private mental health care. The overall
initiative has resulted in major enhancements to the struc-
ture of the VA mental health care system and to numerous
care delivery processes (Edwards, 2008). This focus on
promoting mental health care in VHA has been particularly
important as the nation has been at war, although the
intended reach is indeed much broader than to those re-
turning from the conflicts in Iraq and Afghanistan and is
designed to improve the lives of veterans of all combat
eras. As VHA has been implementing the transformation,
the system has expanded its mental health workforce by
over 8,000 staff since 2006 to more than 20,600 as of the
end of Fiscal Year (FY) 2012 (September 30, 2012).
VHA operates the largest integrated health care sys-
tem in the nation, providing care to veterans at 152 hospi-
tals, 817 community-based outpatient clinics (CBOCs),
133 nursing homes (now called "Community Living Cen-
ters"), and 300 Readjustment Counseling Centers (also
known as "Vet Centers"; National Center for Veterans
Analysis and Statistics, 2012). Over the past two decades,
VHA has promoted noninstitutional care and access
through its network of CBOCs, which is reflected in the
large number of clinics in operation. In FY 2011 (October
1, 2010-September 30, 2011), approximately 5.4 million
veterans received care at VHA medical centers and clinics;
of these veterans, approximately 1.26 million had a con-
firmed mental health condition (Greenberg, Pilver, & De-
sai, 2012). Significantly, as an integrated system, VHA
establishes its own policies and services and funds the care
delivered by the system. Further, VHA is a major supporter
of clinical research, including mental health care research.
January 2014 • American Psychologist
 The model for nationally disseminating and imple-
menting EBPs in VHA has been developed with careful
consideration of the findings in the literature related to the
dissemination and implementation of EBPs specifically and
of health care services more generally. In recent years,
there has been a growing body of research in this area (see,
e.g., Booz Allen Hamilton, 2010; Brownson, Colditz, &
Proctor, 2012; Fixsen, Naoom, Blase, Friedman, & Wal-
lace, 2005). This research has identified key factors at the
provider, systems, patient, and policy levels that impact
dissemination and implementation success. Provider-level
barriers to dissemination and implementation of general
health and mental health care services include limited pro-
vider knowledge of and/or skills in the intervention. A
primary factor that has contributed to the low provision of
EBPs in particular is the limited amount of intensive,
competency-based training in EBPs—beyond mere educa-
tion about or exposure to—available at both the graduate
training and postgraduate levels (Becker & Stirman, 2011;
Crits-Christoph, Frank, Chambless, Brody, & Karp, 1995;
Rosen et el., 2004; Schonbrun et al., 2012; Woody, Weisz,
Gerald Cross & McLean, 2005). Further, research has shown that thera-
pists often overestimate their ability to deliver EBPs and
that clinician self-reports of their implementation of the
As a component of the transformation of the VA
mental health care system, the national dissemination and
implementation of EBPs is designed to increase EBP avail-
ability and fidelity to ensure that veterans have full access Table 1
to EBPs for a wide range of conditions when clinically Evidence-Based Psychotherapies Being Disseminated
indicated. Beyond making EBPs widely available, the ini- in the Veterans Health Administration
tiative is intended to promote psychotherapy as a regularly Evidence-based psychotherapy Condition
available treatment option in VHA in addition to other
treatment services (e.g., case management, psychopharma- Cognitive processing therapy Posttraumatic stress
cotherapy). This includes enhancing the value of psycho- Prolonged exposure therapy disorder
therapy and implementing a systems infrastructure that Cognitive behavioral therapy for Depression
supports psychotherapy delivery. depression
Acceptance and commitment
VHA's National Multilevel EBP therapy for depression
Dissemination and Implementation Interpersonal psychotherapy
Model Behavioral family therapy Serious mental illness
In breadth and scope, the dissemination and implementa- Multiple family group therapy
tion of EBPs in VHA is the largest EBP dissemination Social skills training
initiative in the United States, though other major efforts Integrated behavioral couples Relationship distress
are now under way in this nation and abroad (most notably therapy
in the United Kingdom; McHugh & Barlow, 2010). VHA
Cognitive behavioral therapy for Insomnia
is nationally disseminating and implementing more than 15
insomnia
EBPs (see Table 1) for a wide variety of mental and
behavioral health conditions in adults. While the issue of Cognitive behavioral therapy for Chronic pain
what may be constituted an EBP is beyond the scope of the chronic pain
present article—which focuses on the strategy, process, and Motivational interviewing Motivation and
impact of national EBP dissemination and implementa- adherence
tion—the selection of specific therapies for dissemination
was based on several factors. These considerations in- Motivational enhancement Substance use
cluded the efficacy and effectiveness of the therapy, its therapy disorders
recommendation in clinical practice guidelines (including Contingency management
Behavioral couples therapy
the VA/Department of Defense Clinical Practice Guide- Cognitive behavioral therapy for
lines), its clinical utility with a veteran patient population, substance use disorders
and the feasibility of its implementation in VHA.

January 2014 • American Psychologist 21

therapy are poorly correlated with behavioral observations
of the therapy sessions (Brosan, Reynolds, & Moore,
2008). Additional provider-level barriers to the dissemina-
tion of EBPs include lack of knowledge, negative attitudes
(e.g., EBPs are too mechanistic), and views that EBPs are
not appropriate for the patient population seen by the
provider (Aarons, 2005; Cabana et al., 1999; Martino,
2010; Nakamura, Higa-McMillan, Okamura, & Shimabu-
kuro, 2011; Walrath, Sheehan, Holden, Hernandez, & Blau,
2006; Willenbring et al., 2004).
While provider-level factors are frequently identified
as the reason for dissemination success or failure and are
often where most energy is expended, they are only one
component of the complex dissemination puzzle. System-
level factors, which include organizational resources and
staff capacity (Katon, Zatzick, Bond, & Williams, 2006;
Proctor et al., 2007; Stirman, Crits-Christoph, & DeRubeis,
2004), are often especially significant with respect to dis-
semination of EBPs relative to other health care services.
EBP sessions are typically longer than other health care
visits, requiring 50-120 minutes depending on the EBP
and the modality of delivery (i.e., individual, group, cou-
ple). In addition to resources and staffing, the organiza-
tional culture (i.e., shared values and norms within the
organization) is a critical system-level factor and may
interact with provider-level factors (Aarons, 2005; Aarons
& Sawitzky, 2006). Thus, as important as competency-
based training is, even though a mental health clinician
may be trained in an EBP by the most skilled therapist and
trainer available (and have very favorable attitudes and
opinions toward the EBP), without the support of the local
clinic or organizational system, the therapy will not be
implemented or implemented with fidelity. Thus, while
necessary, competency-based training (and other provider-
level requirements) is not sufficient for EBP implementa-
tion (Fixsen et al., 2005).
Furthermore, health care dissemination and imple-
mentation are impacted by important patient-level barriers,
including existing treatment preferences, lack of aware-
ness, low access, limited affordability, and stigma (Guriter
& Whittal, 2010; Shafran et al., 2009; A. S. Young, Klap,
Sherboume, & Wells, 2001). Patient knowledge and atti-
tudes are especially important within the context of EBPs,
compared with other health care services, given that EBPs
typically necessitate active participation of patients in the
therapy session and between sessions.
Beyond provider, systems, and patient levels, there
has been growing recognition of the potential significance
of the role of policy in dissemination and implementation
(Goldman et al., 2001; Tabak, Khoong, Chambers, &
Brownson, 2012). Policies can promote or impede dissem-
ination and implementation by, for example, directly im-
pacting the operation of programs, influencing the level of
program funding and resources, and/or raising or lowering
the priority of a program or service (Fixsen et al., 2005).
For these reasons, Fixsen et al. referred to policies as
important "influence factors."
Given the multiple factors described above, it is per-
haps not surprising that research has shown that successful
22
dissemination and implementation require a comprehen-
sive, multilevel approach that is adapted to the specific
organization or system and that accounts for barriers at
different levels and stages (Booz Allen Hamilton, 2010;
Fixsen et al., 2005; Mittman, 2012; Walrath et al., 2006).
Indeed, a spate of multilevel frameworks have recently
been introduced in the literature incorporating individual,
system, organization, community, and policy levels (Tabak
et al., 2012). The national EBP dissemination and imple-
mentation model in VHA, which is presented in Table 2,
was developed utilizing a multilevel, system-fit approach
incorporating key dissemination and implementation fac-
tors identified in the literature (Fixsen et al., 2005; Tabak et
al., 2012). This model includes five key domains: (a) policy
level, (b) provider level, (c) local systems level, (d) patient
level, and (e) accountability mechanisms (i.e., mechanisms
for monitoring and evaluating the implementation and im-
pact of EBPs). Each component of this multidimensional
model is described below. In addition to promoting dis-
semination and implementation, the model is designed to
facilitate ongoing sustainability.
Policy Level: National Requirements for EBP
Availability
At the policy level, VHA has established national policy
requiring that all veterans with specific mental health con-
ditions have access to identified EBPs. These requirements
are contained in VHA Handbook 1160.01, Uniform Mental
Health Services in VA Medical Centers and Clinics (U.S.
Department of Veterans Affairs, 2008), development of
which was guided by the VHA Comprehensive Mental
Health Strategic Plan. VHA Handbook 1160.01 identifies
the full range of specific evidence-based and recovery-
oriented mental health services that must be available to
veterans throughout VHA. With respect to EBPs, the hand-
book requires that veterans with PTSD receiving care in
VHA have access to cognitive processing therapy (CPT) or
prolonged exposure therapy (PE), that veterans with de-
pression have access to cognitive behavioral therapy
(CBT), acceptance and commitment therapy (ACT), or
interpersonal psychotherapy (IPT), and that veterans with
serious mental illness have access to social skills training
(SST). The handbook also requires sufficient staff capacity
to deliver these therapies at medical centers and large
clinics and that these EBPs be delivered "as designed and
shown to be effective" (p. 31) to address the fidelity issue
described above.
In 2011, VHA established additional national po-
licy—the VHA Mental Health Initiative Operating Plan—
expanding the EBP requirements in VHA Handbook
1160.01. The VHA Mental Health Initiative Operating Plan
includes requirements for a number of additional EBPs to
be widely available to veterans throughout VHA, including
the EBPs for behavioral health and substance use disorders
listed in Table 1.
Provider Level: Staff Training and Support
At the provider level. Mental Health Services in the VA
Central Office has developed competency-based staff train-
January 2014 • American Psychologist
Table 2
National Evidence-Based Psychotherapy Dissemination and Implementation Model in the Veterans
Health Administration
Primary
Implementation strategy
level Focus Strategies type

Policy National requirements for EBP VHA Comprehensive Mental Health Strategic Plan Push
availability VHA Handbook 1160.01 : Ur^ifortn Mental Health
Services in VA Medical Centers and Clinics
VHA Mental Health Initiative Operating Plan
Provider Staff training and support Competency-based staff training programs Push
•Structured and collaborative consultation
•Organized recruitment and selection processes
Longer-term consultation support
•"Virtual office hours"
•Local peer consultation and communities of
practice
Local systems Local clinical infrastructures Local EBP coordinators and PTSD mentors Pull
and buy-in Adaptations to organization and culture of care,
scheduling grid
Demonstrate direct value and impact of EBPs
•Quantitative data
•Effectiveness
•Satisfaction/acceptability
• Service/cost offset
•Success stories
External facilitation
VHA Handbook 1160.05: Local Implementation of
EBPs for Mental and Behavioral Health Conditions
Patient Clinical implementation Patient informed choice Pull
strategies • Pretreatment processes
Motivational enhancement
Socialization to treatment
Assessing and enhancing the therapeutic relationship
Case conceptualization and goals-based approach
Accountability Monitoring and evaluating Computerized EBP documentation templates Push/pull
implementation and impact Surveys of local EBP delivery
Performance measure
Online psychotherapy metrics dashboard
EBP training program evaluation
•Therapist-level outcomes
• Patient-level outcomes
Noie. EBP = evidence-based psychotherapy; VHA = Veterans Health Administration; PTSD = posttraumatic stress disarder.

ing programs for each EBP being nationally disseminated.
These training programs are designed to address the fun-
damental need for competency-based training in EBPs
among mental health providers. VHA's competency-based
EBP training model includes two key components designed
to build skill mastery and promote successful EBP imple-
mentation and sustainability: (a) participation in an in-
person, experientially based workshop and (b) ongoing
telephone-based clinical consultation on actual therapy
cases with a training program consultant who is an expert
in the psychotherapy. The consultation phase typically lasts
approximately six months and includes review and rating
of sessions using structured therapist rating scales, such as
the Cognitive Therapy Rating Scale (J. Young & Beck,
1980). Of note, for a number of EBPs, therapist rating
scales surprisingly do not exist and have, therefore, been
carefully developed by the training programs. This training
model is significantly more intensive than typical training
approaches (e.g., didactics, continuing education work-
shops) that may promote knowledge but not new compe-
tencies. Competency development generally requires ongo-
ing practice and feedback (Kaslow, 2004).
The VHA EBP training programs include protocols
and supplemental training materials (e.g., therapist manu-

January 2014 • American Psychologist 23

als, therapist training videos, EBP pocket cards) adapted
specifically for veteran patients (e.g.. Chard, Resick, Mon-
son, & Kattar, 2008; Resick, Monson, & Chard, 2007;
Walser, Sears, Chartier, & Karlin, in press; U.S. Depart-
ment of Veterans Affairs, 2010; Wenzel, Brown, & Karlin,
2011). A number of interactive. Web-based EBP training
programs are being developed to provide pretraining to
staff prior to participation in the EBP workshop training
and to solidify and extend core skills acquired through the
EBP training programs.
The primary focus of the VHA EBP training programs
is on staff who spend a significant amount of time treating
the condition that is the target of the therapy in the training
program. Focusing on these practitioners (as opposed to
clinicians who spend a more limited time per week treating
the target condition) thereby allows for maximizing train-
ing yield and throughput. For most training programs, this
includes licensed independent providers (i.e., psycholo-
gists, psychiatrists, social workers, and mental health
nurses, as well as licensed professional mental health coun-
selors and marriage and family therapists, which are new
occupations in VHA). The training programs use a struc-
tured application process for recruiting training partici-
pants. Regional mental health directors help coordinate the
nomination and selection process with local medical cen-
ters and clinics within the region and with national training
program staff. Each Veterans Integrated Service Network
(VISN), or region, is provided with a specific number of
training slots for which the VISN may nominate appropri-
ate staff who meet training program eligibility criteria. It is
encouraged that a minimum of two staff members from a
site participate in a training cohort, when possible, for
mutual support as the staff members learn and implement
the therapy.
In addition to formal consultation provided in the EBP
training programs, VHA has also been working to establish
informal consultation opportunities over the longer term to
provide support for challenging cases and for sustaining
implementation and refining therapy skills. One such
mechanism is the offering of regular "virtual office hours"
in which clinicians call in during established times to speak
with experts in the particular EBP about the therapy or to
ask questions about specific cases. In addition to informal
telephone consultation, VHA has also been working to
promote the implementation of peer consultation groups at
local facilities. Such local networks may not only provide
support and solidify provider skills but may also help to
facilitate communities of practice (CoPs) that can promote
ongoing local implementation. CoPs are groups of individ-
uals in an organization with common interests and goals
that can help drive innovation or practice improvement
(Wenger, McDermott, & Snyder, 2002). CoPs have in-
creasingly been identified in the implementation literature
as useful for fostering organizational change, and they are
now receiving attention in the health care field (Ranmuth-
ugala et al., 2011). CoPs are beginning to be used with
success to promote evidence-based mental health practices
(Barwick, Peters, & Boydell, 2009). In addition to in-
person peer consultation groups in VHA, virtual telephone
24
or televideo consultation groups have been used as CoPs to
support EBP delivery. In fact, a number of consultation
groups in the VHA EBP training programs have on their
own elected to continue to meet by telephone following
completion of the training programs to provide one another
with ongoing clinical and implementation support.
Local Systems Level: Local Clinical
Infrastructures and Buy-In
As significant as the national policy and provider levels are
to national dissemination strategically, addressing local
leadership and organizational factors has been shown to
yield greater implementation success (Fixsen et al., 2005;
Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou,
2004). The local organizational level is especially signifi-
cant in VHA, which, despite being a fully integrated sys-
tem, is decentralized in many respects. Each VHA medical
center has its own director with local priorities, needs, and
stakeholders. Moreover, medical centers are clustered into
21 VISNs, each with its own VISN director with specific
priorities.
To facilitate local implementation of EBPs, a local
EBP coordinator has been placed at each VHA medical
center to serve as a champion for EBPs at the local level.
The local EBP coordinator serves largely in the role of
what is referred to in the implementation literature as an
internal/ací/ííaíor, which has been shown to help promote
the implementation of EBPs into local clinical practice
(Kauth et al., 2010), as well as what has been identified in
the literature as a dissemination field agent (Kreuter, Casey,
& Bemhardt, 2012). Specifically, the local EBP coordina-
tors provide education on EBPs and their potential benefits
to elicit support from leadership and staff (including pri-
mary care and other potential referring staff) at numerous
levels. They also work with facility and mental health
leadership, appointment schedulers, informatics staff, and
others to implement clinical infrastructures (e.g., availabil-
ity of 60-120-minute sessions on the scheduling grid) and
reinforce local cultures that support the delivery of evi-
dence-based care (versus, for example, primarily long-
term, supportive therapy). In addition, local EBP coordi-
nators provide or arrange for longer term EBP consultation
support for clinicians. The local EBP coordinators also
track EBP delivery at the local level and provide this
information to leadership. In addition to local EBP coordi-
nators, PTSD mentors guided by the VA National Center
for PTSD have been appointed within each VISN to facil-
itate local implementation of EBPs for PTSD and to change
the PTSD treatment culture to an evidence-based, recov-
ery-oriented model (Bernardy, Hamblen, Friedman, Ruzek,
&McFall, 2011).
To elicit the support of local leadership, local EBP
coordinators and other implementation champions must
account for and increase the "signal-to-noise ratio." The
signal-to-noise ratio represents the likelihood that an advo-
cate's request or message will be recognized by leadership
(Karlin & Duffy, 2004). The extent to which a request or
concem (signal) is perceived by the local leader (e.g.,
hospital director) depends on the ability of that message to
January 2014 • American Psychologist
get through the many messages (noise) from other cham-
pions, clinical service lines, or constituents seeking atten-
tion. Thus, leaders' support for a request or initiative de-
pends on the degree to which the champion's signal can
permeate the noise of competing interests. In times of fiscal
constraint, the signal must be even more powerful to over-
come the noise competing for fewer resources.
To successfully increase the signal-to-noise ratio, EBP
implementation champions must demonstrate the direct
value and positive impact of EBPs and, thereby, increase
the stimulus of the signal not only with senior organiza-
tional leadership but also with multiple other local leaders
and key stakeholders. In VHA, local EBP coordinators
demonstrate the direct value and impact of EBPs with
veterans to facility, mental health, and clinic leadership, to
patients and family members, and to veteran and commu-
nity stakeholder organizations. One strategy for doing so is
through the communication of quantitative, empirical data
on effectiveness, satisfaction/acceptability, and service off-
set associated witb EBPs, tailored to the audience. These
data include national program evaluation data from VHA's
EBP training programs, as described below. Beyond dem-
onstrating the value of EBPs to local leadership and exter-
nal stakeholders, the communication of empirical data has
also been shown to promote buy-in for EBPs among mental
health clinicians (Stewart & Chambless, 2010).
Another communication strategy used by local EBP
coordinators that can be a very powerful method of per-
suasion and a facilitator of implementation but that is not
widely utilized in EBP dissemination efforts is the use of
storytelling. In the context of EBP dissemination, stories
may consist of formal case reports or informal depictions or
anecdotes provided by patients, providers, and/or programs
of their experiences with EBPs. The use of case examples,
for instance, has been shown to increase provider "training
willingness" related to EBPs (Stewart & Chambless, 2010,
p. 16).
There is irony in the fact that psychologists, who are
experts in the science of persuasion and human behavior,
have scantly used or written about storytelling to elicit
buy-in for psychotherapy. Because stories connect at a
deeper level than other communication methods, they often
more readily influence attitudes and perspectives. Further,
by connecting more deeply and evoking emotions, stories
can increase what Malcolm Gladwell (2002) termed the
"stickiness factor"—that is, how memorable or lasting a
message is. According to Gladwell, the stickiness factor is
essential for driving change and reaching the tipping point,
or threshold, for broad uptake. It is this "stickiness" that
can raise the strength of and personal connection to the
"signal."
In addition to the work of local EBP coordinators as
internal facilitators, external facilitation is provided to sites
to help address logistical, cultural, or other barriers to EBP
implementation. Included in the external facilitation pro-
cess is targeted training related to (a) scheduling and struc-
turing of EBP services, (b) maximizing the use of EBP
trained staff, and (c) the effectiveness of and cost and
service offsets associated with EBPs. Furthermore, VHA
January 2014 • American Psychologist
has implemented a national handbook (VHA Handbook
1160.05) operationalizing the broad requirements for mak-
ing EBPs available. This handbook specifies for local lead-
ers the expectations and specific procedures for fully im-
plementing EBPs locally, including access and capacity
requirements, clinic and scheduling needs, treatment plan-
ning and clinical implementation issues, and training needs
(U.S. Department of Veterans Affairs, 2012).
Patient Level: Clinical Implementation
Strategies
Often the most neglected components in the dissemination
of health care services—and EBPs, specifically—are pa-
tient-level factors. The patient level is a significant com-
ponent of VHA's dissemination and implementation
model. There has been an important focus on not moving
ahead of veterans and on including them as part of the
dissemination process. This is especially important consid-
ering that veterans are often unaccustomed to EBPs, given
VHA's historical focus on psycbopharmacotherapy, case
management, and supportive therapies.
EBPs, no matter how effective in the laboratory, will
have little clinical impact unless patients are informed
about and engaged in the treatment. Accordingly, "pretreat-
ment" processes, which may consist of orientation groups
and other processes to provide information about EBPs and
allow for discussion of their potential utility for a particular
veteran's problems, have been implemented at many VHA
facilities. These introductory processes are designed to
promote veterans' informed choice in the treatment plan-
ning process and can promote engagement in the therapy.
Ultimately, the selection of treatment for a particular vet-
eran is the decision of the veteran in consultation with his
or her therapist. Veterans may also participate in other
treatments and supportive services, which facilities are
encouraged to have available in addition to the require-
ments for EBP availability. In some situations, other treat-
ments may be pursued as an adjunct to or in preparation for
a course of evidence-based psychotherapy.
A particularly valuable EBP clinical implementation
and patient engagement strategy is assessing and enhancing
motivation for therapy. VHA has incorporated motivational
enhancement components into many of its EBP protocols
to promote initial and ongoing engagement in the treat-
ment. During the initial motivational enhancement process,
therapists help patients examine (through their own eyes)
the consequences of their symptoms, recognize how the
therapy can help them achieve identified (behavioral)
goals, and identify potential obstacles to participating in
treatment. Research is beginning to suggest that integrating
motivational enhancement techniques (or full motivational
interviewing) into CBT can enhance patient outcomes and
positively impact the therapy course, including accelerating
therapy gains (Merio et al., 2010; Westra & Dozois, 2006).
In the years ahead, motivational interviewing is likely to be
increasingly part of EBPs, as the two psychological ap-
proaches are highly complementary.
Following the initial motivational enhancement, many
of VHA's EBP protocols include a treatment socialization
25
component designed to provide information about the ther-
apy, the therapy process, and the roles of the therapist and
patient. This component is designed to ensure that the
therapist and veteran remain in parallel as active therapy
components begin. Socializing patients to active-oriented
EBPs is especially important with older patients, who are
often quite unfamiliar with what participation in EBPs
involves, which is often different from the typically more
passive role of the patient in health care treatment (Karlin,
2011).
Furthermore, VHA's EBP protocols place an impor-
tant emphasis on the therapeutic relationship and therapy
process with the goal of emphasizing the "psychotherapy"
in "evidence-based psychotherapy." Unfortunately, EBPs
are often criticized for being mechanistic or overly skills-
based. A rigid focus on techniques may, in part, be the
result of attenuation in fidelity resulting from efforts to
broadly disseminate and adapt CBT and other EBPs. It may
also be due to overly liberal application of the label "CBT,"
for example, to purely psychoeducational activities, which
has resulted in contamination of the label. However, high-
fidelity CBT and other EBPs were originally meant to place
a significant emphasis on relationship factors. In fact, a
number of domains on the Cognitive Therapy Rating Scale
focus on relationship or common factors, such as Under-
standing, Interpersonal Effectiveness, and Collaboration.
A focus on the therapeutic relationship in EBPs may
help to promote patient acceptance and engagement, which
are critical to implementation, but may also help overcome
provider-level attitudinal barriers to EBP implementation.
In their examination of practitioner attitudes toward evi-
dence-based practice. Nelson, Steele, and Mize (2006)
found a preference among providers of evidence-based
practices for more of a focus on the therapeutic relation-
ship.
As part of the focus on the therapeutic relationship and
therapy process, several of VHA's EBP protocols incorpo-
rate a therapeutic alliance measure, such as the Working
Alliance Inventory-Short Revised (WAI-SR; Hatcher &
Gillaspy, 2006). Including such measures for routine use in
EBP delivery promotes therapists' attention to the patients'
beliefs about therapy goals and approaches and allows for
tracking the psychological proximity between the patient
and therapist. This process itself communicates value and
respect toward the patient and allows for mid-course cor-
rections that may promote retention and enhance outcomes.
In addition to the focus on the therapeutic relationship,
VHA's EBP protocols are goals-based and guided by case
conceptualization to allow for individualizing the therapy
to the patient (Persons, 1989). Case conceptualization
forms are commonly used to facilitate conceptualization-
driven therapy. This individualized approach is consistent
with Kazdin's (2008) call to "tailor treatment to meet the
needs of individuals" (p. 149) as part of promoting the
delivery of evidence-based psychological treatments and
with Jensen, Weersing, Hoagwood, and Goldman's (2005)
emphasis on greater attention to specific psychological
procedures and "nonspecific" (p. 70) factors, such as en-
gagement, empathy, and the therapeutic alliance. Beyond
26
promoting patient engagement and outcomes, applying
EBPs in an individualized fashion, with fidelity to the
established therapy model, may increase EBP adoption by
providers (Nelson et al., 2006).
Promoting the "Pull" and the "Push" in EBP
Dissemination
Effective dissemination of EBPs must incorporate "pull,"
as well as "push," factors (Orleans, 2007). Push strategies
are top-down approaches to push a product or service out to
the field. In VHA's EBP dissemination and implementation
model, push strategies include national policies requiring
veteran access to EBPs and competency-based staff train-
ing programs. These mechanisms are developed and de-
ployed from the top to promote the achievement of the
goals of the organization. Pull strategies, on the other hand,
are designed to promote interest in and demand for a
product or service among potential users or other stake-
holders and advocates (e.g., family members). Whereas
push strategies typically focus on the organization or "sup-
ply chain" and are designed to get products or services to
the frontlines, pull strategies focus on consumers and ad-
vocates and are designed to promote knowledge about the
service or product, elicit interest, and draw demand. Most
approaches to disseminating evidence-based psychological
treatments have relied almost exclusively on push strate-
gies (e.g., toolkits, guidelines, training), though pull strat-
egies may be even more effective in large-scale dissemi-
nation efforts.
The power of pull in health care dissemination has
been well recognized and utilized by the pharmaceutical
industry through its significant focus on and use of direct-
to-consumer education and its deemphasis of other meth-
ods of communication (Parker & Pettijohn, 2005). Through
this quintessential pull strategy, the pharmaceutical indus-
try has utilized appeals directly to consumers to create
interest and demand for psychopharmacotherapy. While
likely due to multiple factors, use of psychotropic medica-
tion has increased fourfold as direct-to-consumer ap-
proaches have been utilized (Pratt, Brody, & Gu, 2011).
VHA's EBP dissemination and implementation model
includes an important focus on pull strategies at multiple
levels, including the provider, local systems, and patient
levels. For example, the sharing of success stories and
positive patient outcome data by the local EBP coordina-
tors, described above, is a key method of promoting the
pull of EBPs among veterans, family members, local lead-
ers and staff, advocates, and communities. Moreover, the
sharing of therapist training outcomes (e.g., increased com-
petency and self-efficacy, positive attitudes toward the
training) may help to promote EBP interest in and uptake
among providers.
VHA has developed an EBP staff and public aware-
ness campaign to promote patient and provider awareness
of and interest in EBPs. Included in this campaign are
nationally developed EBP brochures, provider fact sheets,
and posters designed to provide education on and promote
awareness of EBPs among staff and veterans at VHA
facilities and community agencies. In addition to increasing
January 2014 • American Psychologist
awareness, this campaign is designed to promote requests
for EBP and questions about EBP from patients to their
providers, including primary care and other non-mental-
health providers who may promote referral to and engage-
ment in EBPs.
Accountability: Monitoring and Evaluating
Implementation and Impact of EBPs
In addition to the strategies at the national policy, provider,
local systems, and patient levels, VHA has developed ac-
countability mechanisms to promote EBP implementation.
These include a multimethod monitoring and evaluation
process, which has been identified as essential to initial and
ongoing implementation success (Booz Allen Hamilton,
2010).
Significantly, Current Procedural Terminology codes
typically used by health care systems to track services do
not distinguish between specific types of psychotherapy
and, therefore, do not allow for precise tracking of EBPs.
As part of a unique approach to allow for specific tracking
of EBP services, VHA has developed computerized docu-
mentation templates for each of the EBPs being dissemi-
nated that will be inputted into its electronic health record
system, beginning in the summer of F Y 2013. These ses-
sion-by-session documentation templates have been built
utiHzing health factors that will allow for tracking of in-
trasession characteristics (e.g., characteristics of the pa-
tient, setting, therapy process) in addition to tracking of
sessions and treatment completion. Furthermore, by includ-
ing, and allowing for tracking of, therapy components, the
templates will enable monitoring of fidelity that can inform
the development of additional provider resources.
Due to the limitations associated with Current Proce-
dural Terminology codes for tracking EBP services, VHA
has to date implemented a variety of creative and comple-
mentary methods for evaluating local EBP implementation.
One such method has involved administration of national
surveys of VHA facilities assessing EBP delivery. In ad-
dition, VHA has developed an online "dashboard" of met-
rics related to the delivery of courses of psychotherapy
approximating EBPs. The dashboard is designed primarily
as a feedback mechanism for local managers and leadership
to track the degree of local activity in this nationally
prioritized service area and allows for comparison with
other facilities and regional and national activity.
Furthermore, VHA has developed a performance mea-
sure that requires that veterans with PTSD returning from
Iraq and Afghanistan receive a minimum of eight sessions
of psychotherapy within a 14-week period. The introduc-
tion of performance measures was a major factor associated
with the successful transformation of the VA health care
system in the early 21st century and its ability to outper-
form private sector health care on numerous quality indi-
cators (Asch et al., 2004; Watkins et al., 2011). Perfor-
mance measures, which are designed to incentivize the
dehvery of evidence-based and innovative services, include
established targets for a particular health care service or
activity and the frequency with which it must be provided.
Performance measures are incorporated into the perfor-
January 2014 • American Psychologist
mance plans of senior VHA managers at local, regional,
and national levels, and performance on these measures
often impacts performance ratings and bonuses managers
receive. Performance measurement in health care deHvery
has been shown to have a significant impact on how care is
provided (Asch et al., 2004).
Finally, a major component of EBP monitoring and
evaluation in VHA is the evaluation of EBP training and
implementation outcomes. To assess these domains, VHA
has incorporated summative and formative program eval-
uation components into each of its national EBP training
programs. Specifically, these program evaluation proce-
dures examine therapist training outcomes (e.g., changes in
therapist competencies, self-efficacy in the EBP, attitudes
and beliefs toward the therapy, follow-up adoption) and
patient clinical outcomes (e.g., changes on measures of
symptoms, quality of life, and the therapeutic alliance and
treatment completion rates). Of note, the ongoing monitor-
ing of treatment effects has been identified by Kazdin
(2008) as essential to promoting the implementation of
evidence-based psychological practices. Among other pur-
poses, program evaluation results are used for ongoing
quality assurance and program improvement in the EBP
training programs, for informing the development of sup-
plemental training resources, and for communicating the
impact of EBPs to support local implementation.
Impact and Lessons Learned: Training
and Clinical Outcomes
VHA has greatly expanded its capacity to deliver EBPs as
a result of its dissemination and implementation process.
As of the end of FY 2012, VHA had provided training in
one or more EBPs to more than 6,400 VHA staff. Most of
these staff (close to 90%) were licensed independent pro-
viders (LIPs), though some were non-LIPs—for treatments
(e.g., social skills training for serious mental illness) com-
monly delivered by non-LIPs. At the end of FY 2012, there
were approximately 9,600 mental health staff in VHA who
were LIPs. Moreover, all facilities now provide CPT or PE
for PTSD, the most common mental health diagnosis in
VHA, and virtually all sites provide both of these therapies
even though only one of these therapies is required to be
made available. This represents significant diffusion that is
in marked contrast to what existed and was reported in the
literature prior to VHA's national EBP dissemination ini-
tiatives (Rosen et al., 2004).
National program evaluation results reveal that the
training in EBPs in VHA has yielded significant, positive
therapist training outcomes, including increased clinical
competencies, enhanced self-efficacy, and improved
knowledge and attitudes (Chard, Ricksecker, Healy, Karlin,
& Resick, 2012; Eftekhari, Ruzek, Crowley, Rosen, &
Karlin, 2013; Karlin et al., 2012; Karlin, Trockel, Taylor,
Gimeno, & Manber, 2013; Walser, Karlin, Trockel,
Mazina, & Taylor, 2013). Further, program evaluation re-
sults indicate that busy clinicians can, overwhelmingly,
complete intensive EBP training. The rate of successful
27
completion (including attainment of required competency)
has generally been 80%-90%-l- across training programs.
Among the most important questions in large-scale
dissemination and implementation of EBPs is whether im-
plementation of EBPs in real-world settings yields patient
outcomes comparable to those observed in randomized
controlled trials (RCTs). As Rakovshik and McManus
(2010) noted in their recent review of CBT training and
dissemination, "Many of the CBT protocols shown to be
efficacious in RCT have not been evaluated in RCP [rou-
tine clinical practice]" (p. 497). This may be due to pro-
vider factors (i.e., therapists in RCTs are usually well
trained and competent to deliver the therapy), patient fac-
tors (e.g., exclusionary criteria and lack of comorbidities
among participants in RCTs), and system factors (e.g.,
reduction in the number of sessions due to capacity or
economic reasons).
VHA EBP program evaluation data indicate that the
magnitude of the effect for primary patient outcomes across
various EBPs has been in the medium-to-large or large
range in intention to treat and compléter analyses (e.g.,
Eftekhari et al, 2013; Karlin et al, 2012; Karlin, Trockel,
et al, 2013; Karlin, Walser, et al, 2013). These findings are
especially encouraging in light of the fact that they come
from the implementation of the therapies with veteran
patients, who often have complex or chronic mental health
problems, by therapists undergoing EBP training and con-
sultation. It is also worth noting that, on the whole, EBPs
for PTSD and other conditions have generally been shown
to be about equally effective across veteran age groups and
cohorts, including younger veterans returning from Iraq
and Afghanistan and older veterans, who often present with
greater chronicity of symptoms (e.g.. Chard et al, 2012;
Eftekhari et al, 2013; Karlin, Trockel, et al, 2013; Karlin
et al, in press).
Beyond improvements on primary outcome measures,
program evaluation results have demonstrated significant
veteran improvements in quality of life and high WAI-SR
(therapeutic alliance) scores, with increases on all WAI-SR
subdomains (Goal, Task, and Bond) over the therapy
course (Karlin et al, 2012; Karlin, Trockel, et al, 2013;
Karlin, Walser, et al, 2013). The latter findings indicate a
high degree of patient acceptability toward the therapies
and suggest that EBPs can enhance and not be contradic-
tory to relationship and process factors.
In addition to training and clinical outcomes, an ad-
ditional key question on which program evaluation data
shed light is the attrition rate for veterans receiving EBPs,
for which there are very limited data in the literature.
Program evaluation results (Eftekhari et al, 2013; Karlin et
al, 2012; Karlin, Trockel, et al, 2013; Karlin, Walser, et
al, 2013) indicate an almost identical completion rate of
approximately 70% across therapies, which compares fa-
vorably to the mean dropout rate of 44%-47% reported in
psychotherapy research studies (Bados, Balaguer, & Sal-
dana, 2007; Wierzbicki & Pekarik, 1993). The favorable
findings related to patient attrition may, in part, be due to
the important focus EBPs in VHA have placed on the
therapeutic relationship, the case-conceptualization-driven
28
approach to treatment, and the incorporation of motiva-
tional enhancement and treatment socialization, as de-
scribed above. Moreover, completion of EBPs for PTSD,
for example, among veterans has been associated with a
30% reduction in mental health service utilization and in an
approximate 40% reduction in health care costs in the year
following treatment (Meyers et al, 2013; Tuerk et al, in
press).
It is also significant that the program evaluation data
across EBP training programs clearly demonstrate that the
intensive consultation component of the training programs
is essential to promoting clinician competencies and to
facilitating ongoing implementation (Karlin et al, 2012;
Karlin, Trockel, et al, 2013; Walser et al, 2013). More-
over, the consultation process has led to significant in-
creases in therapist self-efficacy to deliver EBP—gains not
yielded by workshop training alone (e.g., Karlin et al,
2012; Manber et al, 2013; Walser et al, 2013). This
experience is consistent with past research that has shown
structured supervision or consultation to be critical to the
development of new skills and clinical competencies
(Miller, Yahne, Moyers, Martinez, & Pirritano, 2004;
Sholomskas, 2005). For these reasons, expert consultation
capacity, which takes time to establish and build, has been
shown to be the key rate-limiting factor in broad EBP
dissemination.
Follow-through with consultation has significantly in-
creased when greater structure has been added to the con-
sultation process, including fixed consultation groups and
assigned consultants, and when consultation requirements
and expectations are clearly set and communicated to ap-
plicants prior to selection. In addition, the framing of the
consultation as a collaborative learning process for clini-
cians with recognized skills has helped to promote cohe-
sion among and engagement by consultées, including reg-
ular submission of session tapes for rating by training
consultants.
The Road Ahead
Promoting Implementation Depth and
Expansion
While VHA has made significant progress in its dissemi-
nation and implementation efforts, with all sites now pro-
viding EBPs, there is variability across sites in the degree
to which EBPs are delivered. At many sites, the "lights" are
quite bright, with many veterans being offered and receiv-
ing EBPs. At other sites, the magnitude of EBP delivery is
lower due to a variety of factors, including scheduling
challenges for weekly protocol-based treatments, local
structural issues, and limited dedicated time provided to the
local EBP coordinator. Accordingly, there is a significant
current focus on standardizing the degree of EBP delivery
throughout the system. Several mechanisms are being im-
plemented to achieve this, including the development and
implementation of the VHA handbook on local require-
ments for fully implementing EBPs (U.S. Department of
Veterans Affairs, 2012), site visits and follow-up technical
assistance, sharing and incorporation of best practices re-
January 2014 • American Psychologist
lated to specific local implementation issues, and the pro-
vision of extemal facilitation. Furthermore, VHA is imple-
menting a new staff expansion initiative, which includes
the hiring of approximately 1,600 mental health providers
and 300 support staff to increase capacity to provide EBPs
and other mental health services.
Furthermore, access challenges to EBPs exist for vet-
erans in some mral areas. As part of a major expansion
effort to promote EBP access and availability for veterans
residing in rural and frontier communities, VHA is nation-
ally implementing a system of EBPs delivered through
telehealth modalities. As part of this initiative, which has
an initial focus on CPT and PE delivered through clinical
video teleconferencing, more than 100 mental health pro-
viders have been hired or assigned to deliver EBP for
PTSD via telemental health at carefully selected sites
throughout VHA. In addition, three regional EBP for PTSD
telemental health clinics have been established to augment
local delivery of EBP for PTSD telemental health services.
Ensuring Sustainability
Another important challenge identified in this large-scale
EBP dissemination process, and one on which there is
limited research, is ensuring sustainability of EBP delivery.
While VHA has trained a large number of providers and
broadly implemented EBPs at this point in the process, this
does not guarantee that EBPs will continue to thrive in the
system years into the future. One key issue related to both
the degree of current implementation and ongoing sustain-
ability borne out of VHA's experience to date concerns
how EBPs are implemented at local facilities. Where EBPs
have been implemented to the greatest degree and contin-
ued to thrive is in facilities that have significantly shifted
the structure and culture of care toward a model that
supports EBP delivery (rather than primarily relying on
long-term therapy, for example). While longer term therapy
services are still available at these facilities as necessary,
many veterans who have completed a course of EBP have
not found the need for ongoing therapy (other than "booster
sessions" sometimes incorporated into EBPs). Moreover,
some facilities have successfully utilized staff other than
specialized mental health providers (e.g., peer support spe-
cialists) to lead follow-up skills provision or support
groups. Notably, research has shown that other implemen-
tation efforts have sometimes failed to be sustained when
such services were added on top of other services without
incorporating changes to the culture or organization of care
(Fixsen et al., 2005). Thus, more fully weaving EBPs into
an organized and supportive local system of evidence-
based and recovery-oriented care appears to be essential to
systemic and enduring EBP implementation.
Also important for sustainability is ongoing adoption
of EBPs by providers. As noted above, VHA has incorpo-
rated longer term provider-level supports (e.g., "virtual
office hours," local EBP coordinators, communities of
practice) to promote ongoing adoption of EBPs. However,
additional research on practical methods for promoting
long-term adoption of EBPs is needed. Also limited, yet
important for ongoing sustainability and quality assurance.
January 2014 • American Psychologist
is research on feasible and reliable methods for assessing
fidelity to EBPs well beyond participation in training, when
fidelity assessment is more practical.
Decentralized Training and Consultation
Capacity
During the initial phase of EBP dissemination and imple-
mentation, VHA has utilized a centralized training model,
with national training programs providing and directly
overseeing EBP training. This centralized model has al-
lowed for training a large number of clinicians and for
maximizing training yield by focusing on core providers.
This was especially important for quickly establishing ca-
pacity in EBPs when the nation was at the height of war.
Moreover, such a centralized training process allows for
the training model, protocols, and quality assurance proce-
dures to be established and recognized in the system. In the
years ahead, VHA will establish a model of decentralized
training and consultation capacity, with national oversight,
to broaden dissemination and promote sustainability. This
has begun to be implemented for the earlier established
VHA EBP training programs.
The decentralized training model will establish re-
gional trainers and consultants who will provide capacity to
train a broader array of local providers than the core staff
trained during the centralized training phase. Decentralized
training capacity will also yield an enduring training infra-
structure for training new staff as they come on board.
Thus, while the centralized process is designed to put the
training process in play, to reach an initial "tipping point,"
and to develop a critical mass of trainers and consultants
(the ramp-up phase), the decentralized process is designed
to promote local sustainability, to meet local training needs
beyond core staff, and to enhance the breadth of local
adoption of EBPs (the expansion and maintenance phase).
Conclusion
There is great urgency and opportunity to bridge the wide
research-to-practice gap that has limited the availability of
EBPs. Perhaps in no other health care context have treat-
ments with the level of effectiveness of many EBPs and, in
some cases, treatments identified as first-line treatments
remained in the laboratory and largely unavailable to pa-
tients for so long. Based on emerging dissemination and
implementation science, VHA's dissemination and imple-
mentation model includes multiple levels designed to ac-
count for a variety of facilitators and barriers within and
outside of different levels of the organization. Such a
theoretically informed, multidimensional approach appears
essential to the dissemination and implementation of EBPs
and other complex treatments that require changes in or-
ganizational systems and cultures, provider competencies
and beliefs, and patient knowledge, perceptions, and treat-
ment readiness. Earlier approaches to the dissemination of
EBPs and other services have generally relied on unidi-
mensional approaches, often focusing exclusively on indi-
vidual clinicians and neglecting larger systems, the overall
organization, and patients. Further, these approaches have
largely incorporated passive methods of information ex-
29
change, including continuing education, dissemination of
clinical practice guidelines, clinical reminder systems, and
practice toolkits designed more for promoting knowledge
than for promoting skill acquisition or behavior change
(Mittman, 2012).
As dissemination science and large-scale dissemina-
tion efforts currently underway point toward employing
more sophisticated, theoretically and empirically driven
models of dissemination, research and dissemination prac-
tice also suggest that before implementing a multidimen-
sional framework—such as the national model described
here—it is important for organizations to tailor the model
to their specific systems. This is particularly important
given that specific barriers within an overall domain tend to
be specific to the organization (Booz Allen Hamilton,
2010; Fixsen et al., 2005). For this reason, it is important to
note that the dissemination and implementation strategies
described above—in particular, the VHA-specific national
policy strategies—may not fully generalize to other sys-
tems. Unlike many other organizations, VHA is a large,
fully integrated system that has the ability to set national
policy specifying the services the system provides. Despite
this, however, local and regional medical center directors
have a wide range of needs and priorities—much like
health care leaders in other systems—requiring additional
strategies for yielding dissemination success at the local
level. The push and pull strategies employed at the pro-
vider, local systems, and patient levels are likely to be very
relevant to other organizations and systems. It is also
important to recognize that there may be unique cost con-
siderations associated with disseminating EBPs in other
systems that are not as integrated as VHA or that directly
fund the services they provide.
Furthermore, it is important to note that timing can be
a salient factor in dissemination and likely contributed
favorably to the feasibility and effectiveness of VHA's
efforts. The initiative was initially developed and imple-
mented in 2005-2006, when the nation was at the peak of
war and there was a recognized need and a national imper-
ative for effective treatments for PTSD and other mental
health conditions. This contextual factor—and the calls for
greater availability of evidence-based psychological treat-
ments by the President's New Freedom Commission on
Mental Health (2003) and, subsequently, by the VHA
Comprehensive Mental Health Strategic Plan—ostensibly
heightened the need and opportunity for EBP dissemination
in VHA.
A critical and universal barrier that rests firmly within
the EBP science-to-practice gap—and on which VHA's
dissemination experience may offer important insight to
virtually all health care systems—is the lack of compe-
tency-based training. Although competency-based training,
by itself, is not sufficient for achieving successful dissem-
ination and implementation of EBPs, it is clear from
VHA's experience and other recent dissemination efforts
that it is one of the most fundamental components for
making EBPs widely available. In addition to promoting
EBP skills and enhancing fidelity, competency-based train-
ing, when implemented successfully, can affect multiple
30
other barriers to dissemination that account for a significant
amount of the adoption variance (e.g., provider beliefs and
attitudes, process for patient-level engagement). Unfortu-
nately, there are relatively few avenues for obtaining com-
petency-based training in EBPs, both at the graduate
level and especially at the postgraduate level, in the
private sector. The lessons learned regarding effective,
competency-based training in EBPs in VHA are likely
generalizable to other systems. This is an area worthy of
greater attention by psychology and other professional or-
ganizations, educational institutions, policymakers, and
mental health provider organizations.
Beyond implementing multicomponent, front-end ap-
proaches to promoting dissemination and implementation,
organizations are likely to yield greater success and sus-
tainability in these efforts by also incorporating monitoring
and accountability mechanisms that provide organizational,
provider, and patient-level feedback. The incorporation of
such a closed-loop dissemination and implementation sys-
tem allows for ongoing assessment of impact at different
levels that can be used to facilitate specific components of
the dissemination model (e.g., empirical evidence of pa-
tient improvement and service offset can help to promote
consumer and leadership interest) or for making changes to
strategies to maximize success. In particular, we believe
mechanisms for precisely tracking and evaluating EBP
delivery, such as those utilized by VHA, are relevant and
essential for all health care systems. At the same time, we
recognize that precise tracking of EBPs would be more
challenging and burdensome in organizations and systems
without electronic health records.
The mental health profession and provider organiza-
tions are encouraged to better communicate the value of
EBPs to (potential) patients, professionals, family mem-
bers, payers, and the broader public for EBPs to reach their
full potential. Promoting both the pull as well as the push
in EBP dissemination is essential for these treatments to
reach the clinical tipping point and become the mainstream,
first-line treatments that science often dictates they should
be and that individuals suffering from mental illness de-
serve.
There is significant opportunity and need for psychol-
ogists, in particular, to help advance dissemination and
implementation and contribute to each of the levels and
specific dissemination and implementation strategies de-
scribed herein. In particular, educational and organizational
psychologists can inform the development and implemen-
tation of training models for therapists to acquire and
maintain core competencies. Social psychologists can add
to the theory and practice of changing provider, patient, and
leadership attitudes and norms to support acceptance of
EBPs. Organizational psychologists can help to identify
and implement mechanisms at the local organizational
level for securing support for and commitment of resources
to EBPs. Further, cognitive psychologists can inform how
clinicians and patients may more effectively use relevant
scientific information in clinical decision making. And
consumer psychologists may advance the use of pull strat-
egies to increase public demand for EBPs.
January 2014 • American Psychologist
 VHA's EBP dissemination and implementation activ-
ities are unique in breadth and scope and indicate that EBPs
can be effective and promote recovery in routine clinical
settings with real-world patients suffering from a wide
variety of conditions. Beyond the impact within the VA
health care system, however, VHA's efforts suggest to the
larger health care community that the broad translation of
EBPs from science to practice is feasible. We hope that
VHA's EBP dissemination and implementation model and
experiences inform and encourage other private and public
health care systems interested in disseminating and imple-
menting EBPs.
REFERENCES
Aarons, G. A. (2005). Measuring provider attitudes toward adoption of
evidence-based practice: Consideration of organizational context and
individual differences. Child and Adolescent Psychiatric Clinics of
North America, 14, 255-271. doi:10.1016/j.chc.2004.04.008
Aarons, G. A., & Sawitzky, A. C. (2006). Organizational culture and
climate and mental health provider attitudes toward evidence-based
practice. Psychotogical Services, 3(\), 61-72. doi:l0.1037/1541-1559
.3.1.61
Asch, S. M., McGlynn, E. A., Hogan, M. H., Hayward, R. A., Shekelle,
S., Rubenstein, L . , . . . Kerr, E. A. (2004). Comparison of quality of care
for patients in the Veterans Health Administration and patients in a
national sample. Annals of Internal Medicine, 141, 938-945. doi:
10.7326/0003-4819-141-12-200412210-00010
Bados. A., Balaguer, G., & Saldana, C. (2007). The efficacy of cognitive-
behavioral therapy and the problem of drop-out. Journal of Clinical
Psychology, 63, 585-592. doi:10.1002/jclp.20368
Barwick, M. A., Peters, J., & Boydell, K. (2009). Getting to uptake: Do
communities of practice support the implementation of evidence-based
practice? Journal of the Canadian Academy of Child and Adolescent
Psychiatry, 18, 16-29.
Becker, K. D.. & Stirman, S. W. (2011). The science of training in
evidence-based treatments in the context of implementation programs:
Current status and prospects for the future. Administration and Policy
in Mental Health and Mental Health Services Research, 38, 217-222.
doi:10.1007/sl0488-011-0361-0
Bemardy, N. C, Hamblen, J. L., Friedman, M. J., Ruzek, J. I., & McFall,
M. E. (2011). Implementation of a posttraumatic stress disorder men-
toring program to improve treatment services. Psychological Trauma:
Theory, Research, Practice, and Policy, 3, 292-299. doi:10.1037/
aO024847
Booz Allen Hamilton. (2010). Implementing clinical guidelines and sup-
port tools in the mental health and military environments: A review of
the recent literature. Arlington, VA: Defense Centers of Excellence for
Psychological Health & Traumatic Brain Injury.
Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of
cognitive therapy performance: Do therapists know how competent
they are? Behavioural and Cognitive Psychotherapy, 36, 581-587.
doi:10.1017/S1352465808004438
Brownson, R. C , Colditz, G. A., & Proctor, E. K. (Eds.). (2012). Dis-
semination and implementation research in health: Translating science
to practice. Oxford, England: Oxford University Press. doi:10.1093/
acprof:oso/9780199751877.001.0001
Butler, A. C , Chapman, J. E., Forman, E. M., & Beck, A. T. (2006). The
empirical status of cognitive-behavioral therapy: A review of meta-
analyses. Clinical Psychology Review, 26, 17-31. doi:10.1016/j.cpr
.2005.07.003
Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilson, M. H.,
Abbotid. P. A., & Rubin, H. R. (1999). Why don't physicians follow
clinical practice guidelines? A framework for improvement. Journal of
the American Medical Association, 282, 1458-1465. doi:10.1001/jama
.282.15.1458
Chard, K. M., Resick, P. A., Monson, C. M., & Kattar, K. (2008).
Cognitive processing therapy group manual: Veteran/military version.
Washington, DC: U.S. Department of Veterans Affairs.
January 2014 • American Psychologist
Chard, K. M., Ricksecker, E. G., Healy, E., Karlin, B. E., & Resick, P. A.
(2012). Dissemination and experience with cognitive processing ther-
apy. Journal of Rehabilitation Research and Development, 49, 667-
678. doi:10.1682/JRRD.2011.10.0198
Crits-Christoph, P., Frank, E., Chambless, D. L., Brody, C , & Karp, J. F.
(1995). Training in empirically validated treatments: What are clinical
psychology students learning? Professional Psychology: Research and
Practice, 26, 514-522. doi:10.1037/0735-7028.26.5.514
DeRubeis, R. J., Gelfand, L. A., Tang, T. Z., & Simons, A. D. (1999).
Medications versus cognitive behavior therapy for severely depressed
outpatients: Mega-analysis of four randomized comparisons. American
Journal of Psychiatry, 156, 1007-1013.
Edwards, D. J. (2008). Transforming the Veterans Administration; the
New Freedom Commission's report guides changes at the VA. Behav-
ioral Healthcare, 28(1), 14-17.
Eftekhari, A., Ruzek, J. I., Crowley, J. J., Rosen, C. S., & Karlin, B. E.
(2013). Effectiveness of national implementation of Prolonged Expo-
sure Therapy in Veterans Affairs care. JAMA Psychiatry. Advance
online publication. doi:10.1001/jamapsychiatry.2013.36
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace,
F. (2005). Implementation research: A synthesis of the literature.
Tampa, FL: University of South Horida, Louis de la Parte Florida
Mental Health Institute.
Freiheit, S., Vye, C , Swan, R., & Cady, M. (2004). Cognitive-behavioral
therapy for anxiety: Is dissemination working? The Behavior Therapist,
27, 25-32.
Gladwell, M. (2002). The tipping point: How little things can make a big
difference. Boston: Little, Brown.
Goldman, H. H.. Ganju, V., Drake, R. E., Gorman, P., Hogan, M., Hyde,
P. S., & Morgan, O. (2001). Policy implications for implementing
evidence-based practices. Psychiatric Services, 52, 1591-1597. doi:
10.1176/appi.ps.52.12.1591
Greenberg, G., Pilver. L., & Desai, R. (2012). 2011 national mental health
fact sheet. West Haven, CT: Northeast Program Evaluation Center.
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O.
(2004). Diffusion of innovations in service organizations: Systematic
review and recommendation. Milbank Quarterly, 82, 581-629. doi:
10.1111/J.0887-378X.2004.00325.X
Gunter, R. W., & Whittal, M. L. (2010). Dissemination of cognitive-
behavioral treatments for anxiety disorders: Overcoming barriers and
improving patient access. Clinical Psychology Review, 30, 194-202.
doi:10.1016/j.cpr.2009.11.001
Hatcher, R. L., & Gillaspy, J. A. (2006). Development and validation of
a revised short version of the Working Alliance Inventory. Psychother-
apy Research, 16, 12-25. doi:10.1080/10503300500352500
Institute of Medicine. (2001). Crossing the quality chasm: A new health
system for the 21st century. Washington, DC: National Academies
Press.
Institute of Medicine. (2007). Treatment ofPTSD: An assessment of the
evidence. Washington, DC: National Academies Press.
Jacobs, G. D., Pace-Schott, E. F., Stickgold, R., & Otto, M. W. (2004).
Cognitive behavior therapy and pharmacotherapy for insomnia: A ran-
domized controlled trial and direct comparison. Archives of Internal
Medicine, 164, 1888-1896. doi:10.I001/archinte.l64.17.1888
Jensen, P. S., Weersing, R., Hoagwood, K. E., & Goldman, E. (2005).
What is the evidence for evidence-based treatments? A hard look at our
soft underbelly. Mental Health Services Research, 7, 53-74. doi:
10.1007/s 11020-005-1965-3
Karlin, B. E. (2011). Cognitive behavioral therapy with older adults. In
K. H. Sorocco & S. Lauderdale (Eds.), Cognitive behavioral therapy
with older adults: Innovations across care settings (pp. 1-28). New
York, NY: Springer.
Karlin, B. E., Brown, G. B., Trockel, M., Cunning, D., Zeiss, A. M., &
Taylor, C. B. (2012). National dissemination of cognitive behavioral
therapy for depression in the Department of Veterans Affairs health
care system: Therapist and patient-level outcomes. Journal of Consult-
ing and Clinical Psychology, 80, 101-l\i. doi:10.1037/a0029328
Karlin, B. E., & Duffy, M. (2004). Geriatric mental health policy: Impact
on service delivery and directions for effecting change. Professional
Psychology: Research and Practice, 35, 509-519. doi: 10.1037/0735-
7028.35.5.509
31
Karlin, B. E., Trockel, M., Brown, G. K., Gordienko, M., Yesavage, J., &
Taylor, C. B. (in press). Comparison of the effectiveness of cognitive
behavioral therapy for depression among older versus younger veter-
ans: Results of a national evaluation. Journal of Gerontology: Psycho-
logical Sciences.
Kariin, B. E., Trockel, M., Taylor, C. B., Gimeno, J., & Manber, R.
(2013). National dissemination of cognitive behavioral therapy for
insomnia in veterans: Clinician and patient-level outcomes. Journal of
Consulting and Clinical Psychology. Advance online publication, doi:
I0.1037/a0032554
Karlin, B. E., Walser, R. D., Yesavage, J.. Zhang, A., Trockel, M., &
Taylor, C. B. (2013). Effectiveness of acceptance and commitment
therapy for depression: Comparison among older and younger veterans.
Aging and Mental Health, 17, 555-563. doi:10.1080/13607863.2013
.789002
Kaslow, N. J. (2004). Competencies in professional psychology. Ameri-
can Psychologist, 59(8), 774-781. doi:10.1037/0003-066X.59.8.774
Katon, W. J., Zatzick, Z., Bond, G., & Williams, J. (2006). Dissemination
of evidence-based mental health interventions: Importance to the
trauma field. Journal of Traumatic Stress, 19, 611-623. doi: 10.1002/
jts.20147
Kauth, M. R., Sullivan, G., Blevins, D., Cully, J. A., Landes, R. D., Said,
Q., & Teasdal, T. A. (2010). Employing external facilitation to imple-
ment cognitive behavioral therapy in VA clinics: A pilot study. Imple-
mentation Science, 5, Article 75. doi:l0.1186/1748-5908-5-75
Kazdin, A. E. (2008). Evidence-based treatment and practice: New op-
portunities to bridge clinical research and practice, enhance the knowl-
edge base, and improve patient care. American Psychologist, 63, 146-
159. doi:10.1037/0003-066X.63.3.146
Kreuter, M. W., Casey. C. M., & Bernhardt, J. M. (2012). Enhancing
dissemination through marketing and distribution systems: A vision for
public health. In R. Browson, G. Colditz, & E. Proctor (Eds.), Dissem-
ination and implementation research in health: Translating science to
practice (pp. 213-222). New York, NY: Oxford University Press,
doi: 10.1093/acprof:oso/9780199751877.003.0011
Madson, M. B., & Campbell, T. C. (2006). Measures of fidelity in
motivational enhancement: A systematic review. Journal of Substance
Abuse Treatment, 31, 67-73. doi:1016/j.jsat.2006.03.010
Manber, R., Trockel, M., Batdorf, W., Siebem, A. T., Taylor, C. B.,
Gimeno, J., & Karlin, B. E. (2013). Lessons learned from the national
dissemination of cognitive behavioral therapy for insomnia in the
Veterans Health Administration: Impact of training on therapists ' self-
efßcacy and attitudes. Manuscript submitted for publication.
Martino, S. (2010). Strategies for training counselors in evidence-based
treatments. Addiction Science & Clinical Practice, 5(2), 30-39.
McAIpine, D. E., Schroder, K., Pankratz, S., & Maurer. M. (2004). Survey
of regional health care providers on selection of treatment for bulimia
nervosa. International Journal of Eating Disorders, 35, 27-32. doi:
10.1002/eat.l0217
McHugh, R. K., & Barlow, D. H. (2010). The dis.semination and imple-
mentation of evidence-based psychological treatments: A review of
current efforts. American Psychologist, 65, 73-84. doi:10.1037/
a0018121
Merio, L. J., Storch, E. A., Lehmkuhl, H. D., Jacob, M. L., Murphy, T. K.,
Goodman, W. K., & Geffken, G. R. (2010). Cognitive-behavioral
therapy plus motivational interviewing improves outcome for pédiatrie
obsessive-compulsive disorder: A preliminary study. Cognitive Behav-
iour Therapy, 39, 24-27. doi:10.1080/16506070902831773
Meyers, L. L., Strom, T. Q., Leskela, J., Thuras, P., Kehle-Forbes, S. M.,
& Curry, K. T. (2013). Service utilization following participation in
cognitive processing therapy or prolonged exposure therapy for post-
traumatic stress disorder. Military Medicine, 178, 95-99. doi: 10.7205/
MILMED-D-12-00302
Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J. M., & Pirritano,
M. (2004). A randomized trial of methods to help clinicians learn
motivational interviewing. Journal of Consulting and Clinical Psychol-
ogy, 72, 1050-1062. doi:10.1037/0022-006X.72.6.1050
Mittman, B. S. (2012). Implementation science in health care. In R.
Browson, G. Colditz, & E. Proctor (Eds.), Dissemination and imple-
mentation research in health: Translating science to practice (pp.
400-418). New York, NY: Oxford University Press. doi:10.1093/
acprof:oso/9780199751877.003.00!9
32
Nakamura, B. J., Higa-McMillan, C. K., Okamura, K. H., & Shimabu-
kuro, S. (2011). Knowledge of and attitudes towards evidence-based
practices in community child mental health practitioners. Administra-
tion and Policy in Mental Health and Mental Health Services Research,
38, 287-300. doi:10.1007/sl0488-011-0351-2
National Center for Veterans Analysis and Statistics. (2012). Department
of Veterans Affairs statistics at a glance. Retrieved from http://
www.va.gov/vetdata/docs/quickfacts/Homepage-slideshow.pdf
National Institute for Clinical Excellence. (2005). Post-traumatic stress
disorder (PTSD): The management of PTSD in adults and children in
primary and secondary care (Clinical Guideline 26). London, England:
Author.
National Institutes of Health. (2005). National Institutes of Health state-
of-the-science conference statement: Manifestations and Management
of Chronic In.somnia in Adults, June 13-15, 2005. Sleep, 28, 1049-
1057.
Nelson, T. D., Steele, R. G., & Mize, J. A. (2006). Practitioner attitudes
toward evidence-based practice: Themes and challenges. Administra-
tion and Policy in Mental Health and Mental Health Services Research,
33, 398-409. doi:10.1007/sl0488-006-0044-4
Orleans, C. T. (2007). Increasing the demand for and use of effective
smoking-cessation treatments reaping the full health benefits of tobac-
co-control science and policy gains-in our lifetime. American Journal
of Preventive Medicine, 33, S340-S348. doi:10.1016/j.amepre.2007.09
.003
Parker, R. S., & Pettijohn, C. E. (2005). Pharmaceutical drug marketing
strategies and tactics: A comparative analysis of attitudes held by
pharmaceutical representatives and physicians. Health Marketing
Quarterly, 22, 27-43. doi:10.1300/J026v22n04_03
Patient Protection and Affordable Care Act, Pub. L. No. 111-148, 124
Stat. 119 (2010). Retrieved from http://www.gpo.gov/fdsys/pkg/
PLAW-11 lpubll48/pdf/PLAW-l 1 Ipubll48.pdf
Persons, J. B. (1989). Cognitive therapy in practice: A case formulation
approach. New York, NY: W. W. Norton.
Pratt, L. A., Brody, D. J., & Gu, Q. (2011 ). Antidepressant use in persons
aged 12 and over: United States, 2005-2008. NCHS data brief, no 76.
Hyattsville, MD: National Center for Health Statistics.
President's New Freedom Commission on Mental Health. (2003). Achiev-
ing the promise: Transforming mental health care in America. Einal
report (DHHS Pub. No. SMA-03-3832). Rockville, MD: U.S. Depart-
ment of Health and Human Services.
Proctor, E. K., Knudsen, K. J., Fedoravicius, N., Hovmand, P., Rosen, A.,
& Perron, B. (2007). Implementation of evidence-based practice in
community behavioral health: Agency director perspectives. Adminis-
tration and Policy in Mental Health and Mental Health Services Re-
search, 34, 479-488. doi:10.1007/sl0488-007-0129-8
Rakovshik, S. G., & McManus, E. (2010). Establishing evidence-based
training in cognitive behavioral therapy: A review of current empirical
findings and theoretical guidance. Clinical Psychology Review, 30,
496-516. doi:10.1016/j.cpr.2010.03.004
Ranmuthugala, G., Plumb, J. J., Cunningham, F. C , Georgiou, A., West-
brook, J. I., & Braithwaite, J. (2011). How and why are communities of
practice established in the healthcare sector? A systematic review of the
literature. BMC Health Services Research, ¡I, Article 273. doi:10.1186/
1472-6963-11-273
Resick, P. A., Monson, C. M., & Chard, K. M. (2007). Cognitive pro-
cessing therapy: Veteran/military version. Washington, DC: U.S. De-
partment of Veterans Affairs.
Rosen, C. S., Chow, H. C , Finney, J. F., Greenbaum, M. A., Moos, R. H.,
Sheikh, J. I., & Yesavage, J. (2004). VA practice patterns and practice
guidelines for treating posttraumatic stress disorder. Journal of Trau-
matic Stres.i, 17, 213-222. doi:10.1023/B:JOTS.0000029264.23878.53
Schonbrun, Y. C , Weite, T., Stuart, G. L., Glynn, T. R., Titelius, E. N.,
& Strong, D. (2012). Mental health experts' perspectives on barriers to
dissemination of couples treatment for alcohol use disorders. Psycho-
logical Services, 9, 64-73. doi:10.1037/a0026694
Shafran, R., Clark, D., Fairburn, C. G., Arntz, A., Barlow, D. H., Ehlers,
A., . . . Wilson, G. T. (2009). Mind the gap: Improving the dissemina-
tion of CBT. Behaviour Research and Therapy, 47, 902-909. doi:
10.1016/j.brat.2009.07.003
Sholomskas, D. E. (2005). We don't train in vain: A dissemination trial of
three strategies of training clinicians in cognitive-behavioral therapy.
January 2014 • American Psychologist
 Journal of Consulting and Clinical Psychology, 73, 106-115. doi:
10.1037/OO22-OO6X.73.1.106
Stewart, R. E., & Chambless, D. L. (2010). Interesting practitioners in
training in empirically supported treatments: Research reviews versus
case studies. Journal of Clinical Psychology, 66, 73-95. doi: 10.1002/
jclp.20630
Stirman, S. W., Crits-Christoph, P.. & DeRubeis, R. J. (2004). Achieving
successful dissemination of empirically supported psychotherapies: A
synthesis of dissemination theory. Clinical Psychology: Science and
Practice, 11, 343-359. doi:10.1093/clipsy.bph091
Tabak, R. G., Khoong, E. C , Chambers, D. A., & Brownson, R. C. (2012).
Bridging research and practice: Models for dissemination and imple-
mentation research. American Journal of Preventive Medicine, 43,
337-350. doi:10.1016/j.amepre.2012.05.024
Tuerk, P. W., Wangelin, B., Rauch, S. A. M., Dismuke, C. E., Yoder, M.,
Myrick, H., . . . Aciemo, R. (2013). Health service utilization before
and after evidence-based treatment for PTSD. Psychological Services.
Advance online publication. doi:10.1037/a0030549
U.S. Department of Veterans Affairs. (2(X)8). Uniform mental health
services in VA medical centers and clinics (VHA Handbook 1160.01).
Washington, DC: Author.
U.S. Department of Veterans Affairs. (2010). Cognitive behavioral ther-
apy for depression in veterans and military servicemembers [DVD]. St.
Louis, MO: VHA Employee Education System.
U.S. Department of Veterans Affairs. (2012). Local implementation of
evidence-based psychotherapies for mental and behavioral health con-
ditions (VHA Handbook 1160.05). Washington, DC: Author.
U.S. Department of Veterans Affairs, & U.S. Department of Defense.
(2010). VA/DoD clinical practice guideline for management of post-
traumatic stress (Version 2.0). Washington, DC: Authors.
Walrath, C. M., Sheehan, A. K., Holden, E. W., Hernandez, M., & Blau,
G. (2006). Evidence-based treatments in the field: A brief report on
provider knowledge, implementation, and practice. Journal of Behav-
ioral Health Services and Research, 33, 244-253. doi: 10.1007/sl 1414-
005-9008-9
Walser, R. D., Karlin, B. E., Trockel, M., Mazina, B., & Taylor, C. B.
(2013). Training in and implementation of acceptance and commitment
therapy for depression in the Veterans Health Administration: Therapist
and patient outcomes. Behaviour Research and Therapy, 51, 555-563.
doi:10.1016/j.brat.2013.05.009
January 2014 • American Psychologist
Walser, R. D., Sears, K., Chartier, M., & Karlin, B. E. (in press).
Acceptance and commitment therapy for depression in veterans: Ther-
apist manual. Washington, DC: U.S. Department of Veterans Affairs.
Watkins, K. E., Pincus, H. A., Paddock, S., Smith, B., Woodroffe, A.,
Farmer, C , . . . Call, C. (2011). Care for veterans with mental and
substance use disorders: Good performance, but room to improve on
many measures. Health Affairs, 30, 2194-2203. doi:10.1377/hlthaff
.2011.0509
Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating com-
munities of practice. Boston, MA: Harvard Business School Press.
Wenzel, A., Brown, G. K., & Karlin, B. E. (2011)- Cognitive behavioral
therapy for depression in veterans and military servicemembers: Ther-
apist manual. Washington, DC: U.S. Department of Veterans Affairs.
Westra, H. A., & Dozois, D. J. A. (2006). Preparing clients for cognitive
behavioral therapy: A randomized pilot study of motivational inter-
viewing for anxiety. Cognitive Therapy and Research, 30, 481-498.
doi : 10.1007/s 10608-006-9016-y
Wierzbicki, M., & Pekarik, G. (1993). A meta-analysis of psychotherapy
dropout. Professional Psychology: Research and Practice, 24, 190-
195. doi: 10.1037/0735-7028.24.2.190
Willenbring, M. L., Kivlahan, D., Kenny, M., Grillo, M., Hagedorn, H., &
Postier, A. (2004). Beliefs about evidence-based practices in addiction
treatment: A survey of Veterans Administration program leaders. Jour-
nal of Substance Abuse Treatment, 26, 79-85. doi:10.1016/S0740-
5472(03)00161-2
Woody, S. K., Weisz, J., & McLean, C. (2005). Empirically supported
treatments 10 years later. The Clinical Psychologist, 58, 5-11.
Young, A. S., Klap, R., Sherbourne, C. D., & Wells, K. B. (2001). The
quality of care for depressive and anxiety disorders in the United States.
Archives of General Psychiatry, 58, 55-61. doi:I0.1001/archpsyc.58
.1.55
Young, J., & Beck, A. (1980). Cognitive Therapy Scale: Rating manual.
Unpublished manuscript. Center for Cognitive Therapy, Philadelphia,
PA.
Zeiss, A. M., & Karlin, B. E. (2008). Integrating mental health and
primary care services in the Department of Veterans Affairs. Journal of
Clinical Psychology in Medical Settings, 15, 73-78. doi: 10.1007/
s 10880-008-9100-4
33
Copyright of American Psychologist is the property of American Psychological Association
and its content may not be copied or emailed to multiple sites or posted to a listserv without
the copyright holder's express written permission. However, users may print, download, or
email articles for individual use.