BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest.
Protected by copyright.
Open access
To cite: Sayer NA,
Maieritsch KP, Yamokoski CA,
et al. Evaluation of
implementation facilitation
integrated into a national
mentoring programme to
improve access to evidence-­
based psychotherapy for
post-t­ raumatic stress
disorder within the veterans
health administration:
a quality improvement
report. BMJ Open Quality
2024;13:e002449. doi:10.1136/
bmjoq-2023-002449
Received 6 June 2023
Accepted 7 December 2023
© Author(s) (or their
employer(s)) 2024. Re-­use
permitted under CC BY-­NC. No
commercial re-­use. See rights
and permissions. Published by
BMJ.
For numbered affiliations see
end of article.
Correspondence to
Dr Nina A Sayer;
​nina.​sayer@v​ a.​gov
Quality improvement report
Evaluation of implementation
facilitation integrated into a national
mentoring programme to improve access
to evidence-­based psychotherapy for
post-­traumatic stress disorder within the
veterans health administration: a quality
improvement report
Nina A Sayer ‍ ‍,1,2,3 Kelly P Maieritsch,4 Cynthia A Yamokoski,4 Robert J Orazem,1
Barbara A Clothier,1 Siamak Noorbaloochi1,3
ABSTRACT
WHAT IS ALREADY KNOWN ON THIS TOPIC
Despite the resources dedicated to specialised
⇒ Despite the resources dedicated to specialised
mental healthcare for patients with post-­traumatic
mental healthcare for patients with post-­traumatic
stress disorder (PTSD) within the US Veterans Health
Administration, evidence-­based psychotherapies stress disorder (PTSD) within the US Veterans Health
(EBPs) for PTSD have been underutilised, as evidenced Administration, evidence-­ based psychotherapies
by low EBP reach to patients. A research-­operation (EBPs) for PTSD have been underutilised, as ev-
collaboration evaluated whether implementation idenced by low EBP reach to patients. There is a
facilitation delivered by regional PTSD mentors as need for sustainable and scalable strategies to im-
part of a national mentoring programme improved prove EBP access in low-­performing clinics.
EBP reach compared with less-­intensive quality
WHAT THIS STUDY ADDS
improvement interventions. We used a non-­equivalent
⇒ Implementation facilitation delivered by staff with
comparison-­group design that included all PTSD
regional mentoring responsibilities and tailored to
clinics with low EBP reach at baseline (n=51). Clinics
specialty care for PTSD increased uptake of EBPs
were grouped into one of four quality improvement
in low-­performing clinics more than less-­intensive
conditions according to self-­selection by regional
quality improvement interventions.
PTSD mentors: facilitation (n=6), learning collaborative
(n=15), mentoring as usual in the regions that had HOW THIS STUDY MIGHT AFFECT RESEARCH,
facilitation-­target clinics (n=15) and mentoring as PRACTICE OR POLICY
usual in other regions (n=15). The primary outcome
⇒ Implementation facilitation can be integrated into
was EBP reach among therapy patients with PTSD
a national programme for quality improvement for
at preintervention baseline and postintervention
PTSD specialty care and may be particularly useful
sustainment periods. We used the ratio of odds ratios
when less-­intensive approaches are not sufficiently
(ROR) between the two time periods to evaluate the
effective.
effectiveness of facilitation compared with the other
conditions, adjusting for patient-­level and clinic-­level
confounders. 26 126 veterans with PTSD received
psychotherapy in one of 51 low-­reach PTSD clinics when less-­intensive approaches are not sufficiently
during preintervention baseline and postintervention effective.
sustainment periods. The odds of a patient receiving
an EBP increased over time across conditions. The BACKGROUND
adjusted ORs of a patient receiving an EBP from
Post-­
traumatic stress disorder (PTSD) is a
baseline to sustainment were 1.35–1.69 times larger
potentially disabling psychiatric disorder
in clinics that received facilitation compared with
resulting from trauma exposure that is
the three comparison conditions (adjusted RORs of
comparison condition versus facilitation ranged from particularly prevalent among veterans who
receive medical care through the US Veterans
0.59 (95% CI 0.47 to 0.75) to 0.74 (95% CI 0.58 to
0.94)). Implementation facilitation can be integrated Health Administration (VHA).1 2 To ensure
into a national programme for quality improvement that veterans have access to high-­ quality
for PTSD specialty care and may be particularly useful PTSD care, in 2005, VHA began nationwide
Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449 1

Open access
rollout of two of the evidence-­ based psychotherapies
(EBPs) recommended as frontline treatments across
PTSD clinical practice guidelines—cognitive processing
therapy (CPT) and prolonged exposure (PE).3 4 Chart
note templates were developed and performance meas-
ures based on template data are used to monitor and
incentivise delivery of CPT and PE to patients with
PTSD, particularly those seen in outpatient clinics that
specialise in PTSD treatment, known as PTSD Clinical
Teams (PCTs). A national PTSD mentoring programme
provides guidance to PCTs on policy and practice through
a network of regional mentors.5
Despite the resources dedicated to PTSD care
within VHA, CPT and PE have been underutilised.6–8
The Promoting Effective, Routine, and Sustained
Implementation of Stress Treatments (PERSIST)
programme, a multiproject collaboration between
VHA health services researchers and operational part-
ners in VHA’s Office of Mental Health and Suicide
Prevention (OMHSP), was designed to address this
implementation gap by improving PCT clinician
uptake and veteran access to CPT and PE. The first
PERSIST project identified team and organisational
factors associated with high versus low reach of CPT
and PE to patients with PTSD.9 The practices associ-
ated with the high reach PCT model of care included
establishing a team mission that prioritised delivery
of EBPs; engaging the team in the mission; estab-
lishing clinic processes that enable the mission (eg,
monitoring patient outcomes); and enlisting external
clinics (eg, general mental health clinics) and facility
leadership to support EBP specialisation within the
PCT. The next PERSIST project piloted implementa-
tion facilitation (IF) tailored to low EBP reach PCTs.
In IF, a facilitator supports individuals and teams to
identify organisational barriers and enabling factors
and make changes to implement evidence into prac-
tice.10–12 For PERSIST, IF was guided by a specialised
toolkit and informed by audit on EBP reach. The
toolkit bundled strategies associated with high reach
in the first PERSIST study9 along with resources to
develop an EBP-­focused mission, build engagement
in this mission, implement procedures to facilitate
EBP delivery, foster positive perceptions of EBPs, and
strengthen facility leadership support for a team struc-
tured to promote EBP implementation. The facilitator
was part of the PTSD mentoring programme leader-
ship team and external to the implementation sites.
The evaluation showed a much larger increase in EBP
reach in intervention compared with matched control
PCTs,13 with continued improvement over the year
after facilitation (unpublished data).
Results from the above implementation project
solidified the decision among the involved researchers
and VHA operational partners to scale up the PERSIST
implementation strategies through VHA’s PTSD
mentoring programme. To accomplish this, the special-
ised toolkit was integrated into existing resources for
2 Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449
BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
regional PTSD mentors and reports to audit EBP
reach, referred to as ‘reach reports’, were expanded
and modified to better align with the VHA’s regional
structure and performance monitoring systems.
Concurrently, the PTSD mentoring programme estab-
lished a unified set of principles of PTSD specialty
care that contained and reframed the elements of
high-­reach teams identified through the first PERSIST
study so that they were congruent with mental health
policy and meaningful to clinicians. Central to these
principles was prioritisation of delivery of EBPs for
PTSD, which are time-­limited (usually 8 to 15 sessions)
psychotherapies to reduce PTSD symptoms, and use
of measurement-­based care (MBC), which is required
for patients seen in PCTs, regardless of intervention
delivered. MBC involves the use of patient-­reported
outcomes to track progress in mental health treatment
and has been shown to improve patient outcomes.14 15
Last, the PTSD mentoring programme leadership team
piloted an expanded role for the regional PTSD
mentors involving the option to: (1) use IF to improve
EBP reach; (2) participate in a learning collaborative
(LC) organised around the principles of PTSD care;
or (3) continue in mentoring as usual (MAU) without
added responsibilities.
This evaluation was conducted to determine whether
IF integrated into the national mentoring programme
and delivered by regional PTSD mentors without
researchers’ involvement in decision-­making improved
EBP reach to patients more than the other concurrent
but less-­intensive quality improvement interventions.
The secondary objective of the evaluation was to deter-
mine whether IF also increased uptake of MBC. While
MBC is not an EBP-­specific therapeutic technique, the
CPT and PE protocols indicate that PTSD symptom
monitoring should be implemented.16 17 Therefore,
interventions that increase uptake of EBPs may also
increase uptake of MBC, at least for EBP patients. This
evaluation will inform decisions regarding the use of
regional mentors as internal facilitators to address
quality gaps within PCTs and may serve as a model
for integration of an evidence-­based implementation
intervention into a national field-­based programme
within a healthcare system.
METHODS
The evaluation was designed for internal purposes in
support of the VHA mission and findings were to be used
for VHA programme improvement. Researchers were
not involved in intervention delivery; they conducted
the evaluation using data collected through routine
care. OMHSP provided documentation that the project
involved non-­research operations activities pursuant to
Department of Veterans Affairs Office of Research and
Development Program Guide 1200.21. The Minneapolis
VA Healthcare System IRB determined that the activi-
ties did not meet the definition of research. The Revised

Standards for Quality Improvement Reporting Excel-
lence (SQUIRE 2.0) guidelines provided the framework
for this article.18
Context and design overview
VA facilities are organised into regional networks referred
to as Veterans Integrated Services Networks (VISNs). At
baseline, there were 118 PCTs across VHA’s 18 VISNs
(3–11 PCTs per VISN). We excluded the six PCTs for
which data were inconsistently available over the evalu-
ation because of transitioning to a new medical record
system or losing or gaining a PCT. Because IF was custom-
ised for PCTs needing assistance to improve patient access
to EBPs, this evaluation included the 51 PCTs that were
providing CPT or PE to less than 25% of therapy patients
with PTSD in the preimplementation baseline period;
25% was the median and mean of EBP reach across all
PCTs at that time. PCTs were assigned to condition based
on mentors' self-­selection.
This programme evaluation used a non-­ equivalent
comparison group design. The preimplementation
baseline and postintervention sustainment periods each
covered 6 months (1 July 2020–31 December 2020 and 1
October 2021–31 March 2022, respectively). The IF and
LC implementation periods also covered 6 months (1
April 2021–30 September 2021). During the 3 months
before implementation, IF and LC mentors were engaged
in activities to prepare for implementation. Other IF proj-
ects have used comparable evaluation periods.13 19
It was not appropriate or possible to involve patients or
the public in the design, conduct, reporting or dissemina-
tion of this evaluation.
Improvement interventions
Mentoring as usual
All 112 PCTs received MAU. MAU involved the PTSD
mentoring programme national leadership team and 36
regional mentors (2 for each of VHA’s 18 VISNs) who
were clinical staff with expertise in EBP delivery and
PTSD specialty care. As part of MAU, the national lead-
ership team provided education on best practices and
policy and reviewed PCT-­relevant data in monthly group
meetings. The mentors then shared this knowledge with
and provided consultation to PCT leadership in their
VISNs in separate monthly meetings. In the quarter prior
to baseline, the PTSD mentoring programme augmented
MAU to include education and consultation on unified
principles of PTSD care that were consistent with the
characteristics of high EBP reach PCTs,9 access to the
toolkit expanded from prior work,13 and quarterly audit
and feedback on EBP reach in PCTs and other mental
health clinics.
Thirty of the 51 low-­reach PCTs received MAU only,
meaning that they did not also receive IF or LC. However,
15 of these 30 PCTs were in the same VISN as a PCT
targeted for IF. Because these PCTs were subject to the
same regional influences as PCTs in IF, we labelled them
IF_VISN PCTs and considered them as a separate group
Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449
BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
Open access
from the 15 PCTs that received MAU only in the 7 VISNs
that did not have an IF or LC PCT.
Implementation facilitation
Six PCTs within five VISNs received IF. The VISN that had
two IF PCTs had the lowest EBP reach across VHA. The
mentors providing IF had 10% protected time for facili-
tation. IF was preceded by the training of PTSD mentors
in facilitation during a 2.5-­day workshop through the
VHA Implementation Facilitation Learning Hub.20 The
mentors who were to serve as facilitators identified a PCT
within their VISN to receive IF based on the following
criteria: (1) low EBP reach (< 25%) as identified through
PERSIST EBP reach reports; (2) a potential champion
for the project at the PCT; and (3) facility leadership
and network chief mental health officer support for the
project. The PTSD mentoring programme used the IF
model adopted by VHA10 and customised for low-­reach
PCTs,13 with reach reports to monitor progress. In the
3-­month preimplementation, the facilitator worked with
the site champion to identify goals for improving EBP
reach, barriers and facilitators, and key stakeholders to
include during the implementation phase. The 6-­month
implementation phase began with a site-­visit followed by
weekly consultation between the facilitator and project
champion at the PCT to enact a structured implementa-
tion plan. The PTSD mentoring programme leadership
team provided consultation to the facilitators on a weekly
basis.
Learning collaborative
Mentors for 15 low-­reach PCTs across 6 VISNs partic-
ipated in LC. The LC model was based on recommen-
dations for effective LCs developed by the Agency for
Healthcare Research and Quality.21 Mentors in LC partic-
ipated in monthly learning sessions led by the PTSD
mentoring programme leadership team. These calls were
structured to provide training on available tools and data
sources and to facilitate exchange of ideas for strategies
to address organisational barriers to quality improve-
ment, taking into account contextual factors within each
VISN. While the overall goal of LC was to implement the
principles of PTSD specialty care, the agreed on priority
was improvement in MBC because OMHSP had recently
developed benchmarks for MBC within PCTs.
Measures
The data for all measures were available through VHA’s
repository of clinical and administrative data, the Corpo-
rate Data Warehouse (CDW), or dashboards used for
monitoring performance.
Outcomes
The primary outcome was EBP reach at the patient-­level
(yes or no EBP receipt). We used CDW data to retrospec-
tively identify all patients who had psychotherapy for
PTSD as outpatients in the 51 PCTs. Each qualifying visit
during the baseline and sustainment time periods was
classified as an EBP (CPT or PE), or ‘other’ using health
3

Open access
factors generated by structured EBP templates that clini-
cians were required to use for EBP documentation. By the
time of this work, the CPT and PE templates were identi-
fying the vast majority of CPT and PE cases.22
The secondary outcome was VHA’s metric of MBC in
PCTs. VHA operationalises MBC for PCTs as the propor-
tion of PCT patients with any diagnosis who have at least
two patient reported outcome measures using the PTSD
Checklist-­5.23 MBC is calculated each quarter of the fiscal
year and available on an internal dashboard designed
to support PCT management. For this evaluation, oper-
ational partners provided the MBC values for the seven
quarters from baseline through sustainment.
Other measures
Clinic-­level EBP reach, extracted from the quarterly reach
reports disseminated to all PCTs, is operationalised as the
proportion of therapy patients with PTSD who received at
least one session of CPT or PE in the PCT. EBP reach data
were available for the year prior to baseline through the
end of sustainment.
We extracted the following variables for patients with
PTSD who received psychotherapy through the 51 low-­
reach PCTs during the two evaluation periods: age, sex
(gender is inconsistently available in administrative data),
race, ethnicity, marital status, period of military service,
driving distance from home to the nearest VHA medical
centre, psychiatric hospitalisation, disability status for
PTSD, and type and number of psychiatric comorbidi-
ties in addition to PTSD. Additionally, we constructed
a measure of baseline clinic workload by dividing the
number of patients with PTSD seen by PCT clinicians by
the number of clinicians in the clinic, as done in prior
research.24
Analysis
Preliminary analyses identified confounding variables
to include in adjusted models of EBP reach. We defined
patient-­level confounders as patient characteristics
that were imbalanced across conditions and associated
with a patient receiving an EBP for PTSD at baseline
at p<0.05. We used analysis of variance or Pearson’s χ2
tests, depending on the variable type and distribution to
identify these confounders. Age, sex, race, marital status,
PTSD disability status and the presence of the following
diagnoses met inclusion criteria: depression, anxiety
disorder, alcohol use disorder and other substance use
disorders.
A priori identified clinic-­level confounders were base-
line workload, baseline EBP reach and the linear time
trend for EBP reach. To estimate the linear time trend,
we predicted what sustainment (post) period EBP reach
would have been in the 51 PCTs in the absence of the
quality improvement interventions using the EBP reach
data from the 4 quarters prior to baseline. We devel-
oped a prediction model using logistic regression with a
random slope (time), a random intercept (site), time (in
quarters) as the independent variable, and EBP receipt
4 Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449
BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
during each of the four quarters prior to baseline as the
dependent variable. We used this model to predict EBP
reach during the sustainment period.
Primary analyses examined the effect of condition on
the change in the odds of a patient receiving an EBP from
baseline to sustainment. First, we implemented simple
and multiple logistic regression to obtain unadjusted and
adjusted ORs, respectively, of a patient receiving an EBP
during each time period by condition. Adjusted models
included the above identified patient-­level and clinic-­level
confounders. To avoid deletion of patients seen in only
one of the two evaluation periods, we ran the regressions
separately for each evaluation period. Next, we used ratio
of odds ratios (RORs) to compare the change in the odds
of EBP receipt by condition. The ORs for each period
were the odds of receiving an EBP versus not with IF as
the reference condition. The ROR was the sustainment
period ORs divided by baseline period ORs. We used 500
bootstrap replicates to obtain the 95% CIs for the RORs.
We did not include random effects for the clustering of
PCTs within VISNs because including random effects
did not improve model fit (Akaike Information Crite-
rion=10 619.72 versus 10 621.72 without and with random
effect, respectively).
We had limited data (number of cases and eligible
patients) on MBC. We used these data to assess the effect
of conditions on MBC using repeated measure logistic
regression. We also implemented alternative models using
MBC values for the quarter at baseline and sustainment
closest to implementation and the average of MBC values
across the quarters at each evaluation period. The alter-
native models produced inferentially equivalent RORs
and did not make use of all the available data. Therefore,
we report findings from the repeated measures logistic
regression. Due to the lack of patient-­level and prebase-
line MBC data, we could not adjust for confounders or
the time trend in MBC.
Analyses were preformed using SAS 9.4 and R V.4.2.1.
RESULTS
Over the 2 evaluation periods, 26 126 unique patients
with PTSD received psychotherapy in one of the 51 PCTs
with low reach at baseline. This included 2829 in IF, 8073
in LC, 8464 in IF_VISN and 6760 in MAU PCTs. Charac-
teristics of patients by condition are presented in table 1.
The only characteristic that was not imbalanced across
conditions was psychiatric hospitalisation (p=0.45). Out
of the remaining patient characteristics, ethnicity, driving
distance, bipolar disorder and psychotic disorders were
not related to the receiving an EBP for PTSD and thus not
treated as confounders.
Figure 1 presents EBP reach by condition per quarter
beginning the year before baseline through the end of
sustainment. As can be seen, EBP reach was low and largely
stable in the four quarters before baseline, even after the
onset of the COVID-­19 pandemic which resulted in a
reduction in the number of patients with PTSD receiving
 BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
Open access

Table 1 Characteristics of patients with post-­traumatic stress disorder (PTSD) who received psychotherapy during the
evaluation periods
Implementation Learning Implementation Mentoring as
facilitation collaborative facilitation VISN usual
n=2829 n=8073 n=8464 n=6760
Age M (SD) 51.81 (15.83) 50.54 (15.02) 49.83 (15.03) 50.57 (14.91)
Female N (%) 456 (16.12) 1513 (18.74) 1353 (15.99) 1184 (17.51)
Race N (%)
 Black 697 (24.64) 1900 (23.54) 2161 (25.53) 2138 (31.63)
 White 1740 (61.51) 5084 (62.98) 5063 (59.82) 3942 (58.31)
 Other or unknown 392 (13.86) 1089 (13.49) 1240 (14.65) 680 (10.06)
Hispanic ethnicity N (%) 498 (17.60) 452 (5.60) 1465 (17.31) 447 (6.61)
Marital status N (%)
 Married 1464 (51.75) 4141 (51.29) 4313 (50.96) 3705 (54.81)
 Divorced 572 (20.22) 1671 (20.70) 1688 (19.94) 1463 (21.64)
 Widowed 45 (01.59) 125 (01.55) 99 (1.17) 84 (01.24)
 Never married 582 (20.57) 1585 (19.63) 1781 (21.04) 1127 (16.67)
 Unknown 166 (05.87) 551 (06.83) 583 (06.89) 381 (05.64)
Driving distance from home to nearest VHA 20.98 (18.14) 23.39 (24.73) 18.24 (20.10) 26.17 (28.87)
medical centre M (SD)
Psychiatric comorbidities N (%)
 Depressive 1237 (43.73) 3930 (48.68) 3871 (45.73) 3783 (55.96)
 Anxiety 627 (22.16) 2108 (26.11) 2013 (23.78) 1677 (24.81)
 Alcohol use 317 (11.21) 1055 (13.07) 1234 (14.58) 981 (14.51)
 Other substance use 229 (8.09) 708 (8.77) 847 (10.01) 632 (9.35)
 Bipolar 141 (4.98) 589 (7.30) 524 (6.19) 446 (6.60)
 Psychosis 39 (1.38) 127 (1.57) 87 (1.03) 87 (1.29)
Number of psychiatric comorbidities M (SD) 0.92 (0.99) 1.05 (1.04) 1.01 (1.03) 1.13 (1.02)
Past year psychiatric hospitalisation N (%) 11 (0.39) 28 (0.35) 26 (0.31) 15 (0.22)
PTSD service connection N (%) 1922 (67.94) 5189 (64.28) 5624 (66.45) 4635 (68.57)
VISN, Veterans Integrated Services Network; VHA, Veterans Health Administration.

psychotherapy. Importantly, EBP reach increased across
all conditions from baseline to sustainment.
Table 2 presents the ORs for receiving an EBP at each
evaluation period and the RORs for the change in the
odds from baseline to sustainment with IF as the reference
group. At baseline, the odds of a patient receiving an EBP
for PTSD were lower in IF than in the other three condi-
tions. During sustainment, the odds of a patient receiving
an EBP for PTSD were lower in MAU and IF_VISN than
in IF. The RORs demonstrated that the improvement in
the odds of a patient receiving an EBP for PTSD were
greater for IF compared with each other condition. In
unadjusted analyses, the magnitude of improvement in
IF was almost twice that in the other conditions. As shown
in the lower half of table 2, when we adjusted for patient-­
level confounders, baseline clinic workload, baseline EBP
reach, and the time trend in EBP reach from the four
quarters preceding baseline, the odds of improvement in
reach in PCTs that received IF was 1.35–1.69 times greater
than in PCTs in the other conditions.
Figure 1 presents MBC over the seven quarters from
baseline through sustainment. The RORs showed that the
OR for a patient receiving MBC from baseline to sustain-
ment was greater in IF compared with other quality
improvement conditions. Specifically, with IF as the refer-
ence condition, the ROR for LC was 0.71 (95% CI 0.55
to 0.90), the ROR for IF_VISN was 0.53 (95% CI 0.42 to
0.67) and the ROR for MAU was 0.62 (0.48 to 0.80). Thus,
the odds of improvement in MBC in PCTs that received IF
were 1.41–1.89 larger compared with those in the other
conditions.
DISCUSSION
A research-­operation collaboration evaluated whether IF
delivered by regional PTSD mentors as part of a national
mentoring programme improved EBP reach compared

Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449 5

 BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
Open access

Figure 1 Primary and secondary outcomes over time by condition. Bars represent 95% CIs. The X-­axis shows the month/
year per quarter and the number of eligible patients for each outcome. The number of eligible patients for the first quarter of the
baseline was not available (NA) for measurement-­based care. EBP, evidence-­based psychotherapy; MBC, measurement-­based
care; IF, implementation facilitation; IF_VISN, implementation facilitation Veterans Integrated Service Network; LC, learning
collaborative; MAU, mentoring as usual.

with less intensive quality improvement interventions.
Our primary finding was that while on average EBP reach
improved in all low-­reach PCTs, the PCTs that received
IF experienced the largest improvement. The magnitude
of improvement in EBP reach among the IF PCTs was
1.35–1.69 times greater than in the other conditions after
adjustment for time trends and confounders. The effect
of IF was specific to the targeted low-­reach PCTs and did
not spread to other low-­reach PCTs within the same VISN
(ie, the IF_VISN PCTs). We conclude, therefore, that it
is unlikely that the improvement in EBP reach in IF was
due to changes in policies or procedures implemented
regionally. The fact that the effects of IF did not spill over
to other PCTs within the same VISN demonstrates that IF
requires focused activities with a target clinic. Overall, our
evaluation’s results are consistent with research evidence
that facilitation can improve implementation outcomes
in low-­performing clinics.25 26

Table 2 Estimated effect of implementation facilitation on changes in the odds of receiving of an evidence-­based
psychotherapy (EBP) for post-­traumatic stress disorder (PTSD) from baseline to sustainment evaluation periods
Baseline Sustainment
Condition
Ref: Implementation facilitation
OR 95% CI OR 95% CI ROR 95% CI†
Learning collaborative 1.73*** (1.44 to 2.08) 0.97 (0.85 to 1.09) 0.56*** (0.45 to 0.69)
Implementation facilitation VISN 1.41*** (1.17 to 1.68) 0.66*** (0.58 to 0.75) 0.47*** (0.38 to 0.59)
Mentoring as usual 1.46*** (1.20 to 1.76) 0.75*** (0.66 to 0.86) 0.51*** (0.41 to 0.64)
AOR‡ 95% CI AOR‡ 95% CI ROR 95% CI†
Learning collaborative 0.99 (0.81 to1.20) 0.73*** (0.64 to 0.84) 0.74** (0.58 to 0.94)
Implementation facilitation VISN 0.98 (0.81 to 1.19) 0.58*** (0.51 to 0.66) 0.59*** (0.47 to 0.75)
Mentoring as usual 1.02 (0.84 to 1.25) 0.65*** (0.57 to 0.75) 0.64*** (0.50 to 0.81)
**p < 0.01.
***p < 0.001.
† Bootstrapped 500 samples.
‡ Odds ratio adjusted for age, sex, race, marital status, PTSD service connection, depressive disorder, anxiety disorder, alcohol use disorder,
other substance use disorder, baseline clinic workload, baseline EBP reach, and time trend for EBP reach.
AOR, adjusted OR; ROR, ratio of OR; VISN, Veterans Integrated Service Network.

6 Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449


We find it interesting that EBP reach increased from
baseline to sustainment across all conditions. By the time
of this evaluation, MAU had been enhanced to include
unified principles of PTSD care that were consistent with
the characteristics of high EBP reach PCTs,9 access to a
specialised toolkit and regular audit and feedback on EBP
reach. While we cannot derive definitive conclusions as to
cause, we conjecture that these enhancements to MAU
contributed to the average level of improvement in EBP
reach in PCTs over time. This premise is supported by
the observation that before these enhancements to MAU,
EBP reach did not increase in PCTs over a comparable
21-­month time frame.13 It is further supported by the
finding that there was little or no change in the propor-
tion of therapy patients with PTSD who received an EBP
in general mental clinics from baseline to sustainment.
Specifically, data from the quarterly reach reports showed
that the across the 51 involved facilities, the proportion
of therapy patients with PTSD who received an EBP in
general mental health clinics was 4% and 3% during
baseline and sustainment, respectively, regardless of the
quality improvement condition for the PCT in the same
medical centre. The increase in EBP reach over time was
specific to PCTs, all of which received MAU, and not part
of a larger trend within the involved medical centres.
Mirroring the pattern observed with EBP reach, the
magnitude of improvement in MBC was greater in IF than
in the other conditions, including LC which had priori-
tised implementation of MBC. This is not surprising given
that the EBP treatment manuals call for regular admin-
istration and review of PTSD symptom measures.16 17
However, we were not able to determine the degree to
which the improvement in MBC was attributable to use of
MBC as part of EBP delivery.
IF was enacted by regional PTSD mentors, whereas in
prior work, it was enacted by an external facilitator who
was part of the PTSD mentoring programme leadership
team.13 The PTSD mentoring programme made this
change to enhance IF scalability and the capacity for IF
across PTSD specialty care. If successful as facilitators,
PTSD mentors could apply facilitation skills to improve
reach and other outcomes among PCTs in their VISN.
The PTSD mentors may have been uniquely positioned to
assume the facilitator role after brief training because of
their existing relationships with PCTs within their region.
Regardless, this evaluation demonstrated the feasibility of
using clinical staff with regional responsibilities to expand
the cadre of internal facilitators for quality improvement.
The six facilitators had 10% (4 hours per week)
protected time; mentors in LC and MAU did not have
protected time for these quality improvement activi-
ties. While the IF PCTs experienced a greater level of
improvement in EBP reach than PCTs in the other condi-
tions, this evaluation does not answer the question as
to whether this difference in improvement is worth the
resources required. Given that IF is more intensive than
MAU and LC, we propose a stepped-­care approach27 28 to
improving the quality of care offered in PCTs, with the
Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449
BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
Open access
clinic being stepped to increasingly intensive implemen-
tation interventions depending on clinic performance.
For example, it seems reasonable to use IF when a PCT
does not show the desired improvement from MAU or
other less-­intensive interventions. A disadvantage of the
stepped approach is that clinics that need a higher level
of support for change would have to wait until they have
demonstrated continued need after failing to improve.
Alternatively, it may make sense to use IF when a clinic
is known to have a greater need for support to overcome
barriers to making structural changes just as it makes sense
to offer more intensive clinical interventions to patients
who are experiencing high levels of clinical need. At the
same time, effective IF requires a champion for change
within the clinic10 and not all low-­ performing clinics
have such a potential change agent. A different approach
would be needed to improve clinic performance when
there is no clear site-­level change leader.
This evaluation used a quasi-­ experimental observa-
tional design. As such, clinics were grouped into condi-
tion based on their mentors’ self-­ selection and the
mentors and clinics in each condition may have differed
on unmeasured factors. We were, however, able to
restrict the evaluation to PCTs with largely comparable
EBP reach at baseline and to adjust for patient case-­mix,
baseline clinic reach, workload and the time trend for
reach in our primary analyses. Unfortunately, we did not
have patient-­level or prebaseline MBC data and there-
fore cannot rule out the influence of confounders or the
prebaseline time trend for this secondary outcome. We
also acknowledge that this project took place during the
COVID-­19 pandemic and that the number of therapy
patients dropped considerable during the first phase of
the pandemic and did not fully recover over the evalu-
ation periods. While we are not aware of the COVID-­19
pandemic having affected PCTs in one condition more
than another, this possibility cannot be ruled out. The
main change necessitated by the COVID-­19 pandemic was
the addition of information on use of telemedicine for
EBP delivery in the reach reports that all PCTs received.
A final and important limitation is that we did not inter-
view mentors or clinical staff from all conditions or track
the time mentors and champions spent in the different
quality improvement activities. A formative evaluation
involving staff in each condition would have helped to
understand the reasons for these findings and ways to
improve or tailor the quality improvement strategies for
different clinic contexts.
CONCLUSIONS
IF enacted by regional mentors improved access to EBPs
for PTSD in low-­performing PCTs more than other, less-­
intensive quality improvement interventions. IF focused
on improving EBP reach was also associated with greater
improvement in MBC. These findings support the training
and deployment of clinical staff with regional mentoring
responsibilities as facilitators for quality improvement in
7

Open access
PTSD specialty care when less-­intensive approaches are
not sufficiently effective. Finally, this evaluation illus-
trates the successful integration of an evidence-­ based
quality improvement intervention tailored for a specific
context through a research-­operation collaboration into
a national programme within a healthcare system. This
integration has the potential to improve intervention
scalability and sustainment.
Author affiliations
1
CCDOR, Minneapolis VA Health Care System, Minneapolis, Minnesota, USA
2
Department of Psychiatry and Behavioral Sciences, University of Minnesota,
Minneapolis, MN, USA
3 Practice, and Policy 2011;3:292–9.
Department of Medicine, University of Minnesota, Minneapolis, MN, USA
4
Executive Division, National Center for PTSD, White River Junction VA Medical
Center, White River Junction, Vermont, USA
Acknowledgements The authors would like to thank Sean Nugent for procuring
the data and all PTSD mentors and PCT staff.
Contributors Concept and design: NAS, KPM, SN. Acquisition, analysis, or
interpretation of data: all authors. Drafting of the manuscript: all authors. Obtained
funding: NAS. Administrative, technical, or material support: RJO. The authors
read and approved the final manuscript. NAS is the guarantor who accepts full
responsibility for this work, had access to the data and controlled the decision to
publish.
Funding This quality improvement project was funded through a U.S Department
of Veterans Affairs (VA), Health Services Research and Development (HSR&D) grant
(RVR 19-­479) awarded to the first author.
Disclaimer The views expressed in this article are those of the authors and do
not necessarily reflect the position or policy of the VA, HSR&D or U.S. Government.
Funders had no role in the design of the study and collection, analysis, and
interpretation of data and in writing the manuscript.
Competing interests The authors are VHA employees. NAS received VHA research
funding. The authors have no other competing interests to declare.
Patient and public involvement Patients and/or the public were not involved in
the design, or conduct, or reporting, or dissemination plans of this research.
Patient consent for publication Not applicable.
Ethics approval This evaluation was designed for internal purposes in support of
the VHA mission and findings were to be used within VHA for program improvement.
The data were collected as part of routine care and VHA operations. The project’s
primary operational partner (VHA’s Office of Mental Health and Suicide Prevention)
provided documentation that the project’s aims and this report involved non-­
research operations activities pursuant to US Department of Veterans Affairs Office
of Research and Development Program Guide 1200.21. The Minneapolis Veterans
Affairs Health Care System IRB determined that the activities did not meet the
definition of research. This was a quality improvement project (see above). We
used data collected as part of routine care to understand provider behaviour (the
provision of evidence-­based psychotherapies for PTSD). Data were retrospectively
collected.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement The datasets analysed for this evaluation are
available from the corresponding author on reasonable request. These data
will remain within the VA firewall and be housed on VINCI data servers. Outside
investigators can follow VA procedures and receive training and approval for access
within VA firewalls.
Open access This is an open access article distributed in accordance with
the Creative Commons Attribution Non Commercial (CC BY-­NC 4.0) license,
which permits others to distribute, remix, adapt, build upon this work non-­
commercially, and license their derivative works on different terms, provided the
original work is properly cited, appropriate credit is given, any changes made
indicated, and the use is non-­commercial. See: http://creativecommons.org/​
licenses/by-nc/4.0/.
ORCID iD
Nina A Sayer http://orcid.org/0000-0002-1068-1562
8 Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449
BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
REFERENCES
1 Sayer NA, Orazem RJ, Noorbaloochi S, et al. Iraq and Afghanistan
war veterans with reintegration problems: differences by
veterans affairs Healthcare user status. Adm Policy Ment Health
2015;42:493–503.
2 Greenberg G, Hoff R. FY. Veterans with PTSD Data Sheet: National,
VISN, and Healthcare System Tables. West Haven, CT: Northeast
Program Evaluation Center, 2021.
3 Karlin BE, Cross G. From the laboratory to the therapy room:
national dissemination and implementation of evidence-­based
Psychotherapies in the U.S. Am Psychol 2014;69:19–33.
4 Hamblen JL, Norman SB, Sonis JH, et al. A guide to guidelines for
the treatment of Posttraumatic stress disorder in adults: an update.
Psychotherapy (Chic) 2019;56:359–73.
5 Bernardy NC, Hamblen JL, Friedman MJ, et al. Implementation
of a Posttraumatic stress disorder mentoring program to improve
treatment services. Psychological Trauma: Theory, Research,
6 Maguen S, Holder N, Madden E, et al. Evidence‐Based
psychotherapy trends among Posttraumatic stress disorder patients
in a national Healthcare system, 2001–2014. Depress Anxiety
2020;37:356–64.
7 Rosen CS, Matthieu MM, Wiltsey Stirman S, et al. A review of
studies on the system-­wide implementation of evidence-­based
Psychotherapies for Posttraumatic stress disorder in the veterans
health administration. Adm Policy Ment Health 2016;43:957–77.
8 Sripada RK, Bohnert KM, Ganoczy D, et al. Documentation of
evidence-­based psychotherapy and care quality for PTSD in
the Department of veterans affairs. Adm Policy Ment Health
2018;45:353–61.
9 Sayer NA, Rosen CS, Bernardy NC, et al. Context matters: team
and organizational factors associated with reach of evidence-­based
Psychotherapies for PTSD in the veterans health administration. Adm
Policy Ment Health 2017;44:904–18.
10 Ritchie MJ, Dollar KM, Miller CJ, et al. Using implementation
Facilitation to improve care in the veterans health Administration
(version 3). In: Veterans Health Administration, Behavioral Health
Quality Enhancement Research Initiative (QUERI). 2020. Available:
https://www.queri.research.va.gov/tools/implementation/Facilitation-​
Manual.pdf
11 Stetler CB, Legro MW, Rycroft-­Malone J, et al. “Role of "external
Facilitation" in implementation of research findings: a qualitative
evaluation of Facilitation experiences in the veterans health
administration”. Implementation Sci 2006;1:23.
12 Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of
implementation strategies: results from the expert recommendations
for implementing change (ERIC) project. Implement Sci 2015;10:21.
13 Sayer NA, Bernardy NC, Yoder M, et al. Evaluation of an
implementation intervention to increase reach of evidence-­based
Psychotherapies for PTSD in US veterans health administration
PTSD clinics. Adm Policy Ment Health 2021;48:450–63.
14 Lewis CC, Boyd M, Puspitasari A, et al. Implementing measurement-­
based care in behavioral health: A review. JAMA Psychiatry
2019;76:324–35.
15 Resnick SG, Hoff RA. Observations from the National implementation
of measurement based care in mental health in the Department of
veterans affairs. Psychol Serv 2020;17:238–46.
16 Resick PA, Monson CM, Chard KM. Cognitive processing therapy for
PTSD: a comprehensive manual. New York, NY: Guilford Press, 2017.
17 Foa EB, Hembree E, Rothbaum B. Prolonged exposure therapy for
PTSD: therapist guide. In: Prolonged Exposure Therapy for PTSD:
Therapist Guide: Emotional processing of traumatic experiences.
Oxford University Press, 2007.
18 Ogrinc G, Davies L, Goodman D, et al. SQUIRE 2.0 (standards
for quality improvement reporting excellence): revised publication
guidelines from a detailed consensus process. BMJ Qual Saf
2016;25:986–92.
19 Kirchner JE, Ritchie MJ, Pitcock JA, et al. Outcomes of a partnered
Facilitation strategy to implement primary care-­mental health. J Gen
Intern Med 2014;29 Suppl 4(Suppl 4):904–12.
20 Kirchner JE, Dollar KM, Smith JL, et al. Development and preliminary
evaluation of an implementation Facilitation training program.
Implementation Research and Practice 2022;3:263348952210874.
21 Nix M, McNamara P, Genevro J, et al. Learning Collaboratives:
insights and A new Taxonomy from AHRQ’s two decades of
experience. Health Aff (Millwood) 2018;37:205–12.
22 Shiner B, Levis M, Dufort VM, et al. Improvements to PTSD
quality Metrics with natural language processing. J Eval Clin Pract
2022;28:520–30.
23 Bovin MJ, Marx BP, Weathers FW, et al. Psychometric properties of
the PTSD checklist for diagnostic and statistical Manual of mental

disorders–fifth edition (PCL-­5) in veterans. Psychological Assessment
2016;28:1379–91.
24 Mohr DC, Rosen CS, Schnurr PP, et al. The influence of team
functioning and workload on Sustainability of trauma-­focused
evidence-­based Psychotherapies. Psychiatr Serv 2018;69:879–86.
25 Hagedorn HJ, Gustavson AM, Ackland PE, et al. Advancing
pharmacological treatments for opioid use disorder (adapt-­OUD): an
implementation trial in eight veterans health administration facilities.
J Gen Intern Med 2022;37:3594–602.
Sayer NA, et al. BMJ Open Quality 2024;13:e002449. doi:10.1136/bmjoq-2023-002449
BMJ Open Qual: first published as 10.1136/bmjoq-2023-002449 on 12 January 2024. Downloaded from http://bmjopenquality.bmj.com/ on January 15, 2024 by guest. Protected by copyright.
Open access
26 Ritchie MJ, Parker LE, Edlund CN, et al. Using implementation
Facilitation to foster clinical practice quality and adherence to
evidence in challenged settings: a qualitative study. BMC Health Serv
Res 2017;17:294.
27 Bower P, Gilbody S. Stepped care in psychological therapies:
access, effectiveness and efficiency: narrative literature review. Br J
Psychiatry 2005;186:11–7.
28 Davison GC. Stepped care:doing more with less J Consult Clin
Psychol 2000;68:580–5
9