J Telemed Telecare OnlineFirst, published on November 30, 2016 as doi:10.

1177/1357633X16677676

Research Article

Journal of Telemedicine and Telecare

0(0) 1–11
Evaluation of a national telemedicine ! The Author(s) 2016
Reprints and permissions:
initiative in the Veterans Health sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/1357633X16677676

Administration: Factors associated jtt.sagepub.com

with successful implementation

Lauren Stevenson PhD1, Sherry Ball PhD1, Leah M Haverhals MA2,

David C Aron MD MS1 and Julie Lowery PhD3

Abstract
Background: The Consolidated Framework for Implementation Research was used to evaluate implementation facilitators and
barriers of Specialty Care Access Network-Extension for Community Healthcare Outcomes (SCAN-ECHO) within the
Veterans Health Administration. SCAN-ECHO is a video teleconferencing-based programme where specialist teams train
and mentor remotely-located primary care providers in providing routine speciality care for common chronic illnesses. The
goal of SCAN-ECHO was to improve access to speciality care for Veterans. The aim of this study was to provide guidance and
support for the implementation and spread of SCAN-ECHO.
Methods: Semi-structured telephone interviews with 55 key informants (primary care providers, specialists and support staff)
were conducted post-implementation with nine sites and analysed using Consolidated Framework for Implementation Research
constructs. Data were analysed to distinguish sites based on level of implementation measured by the numbers of SCAN-ECHO
sessions. Surveys with all SCAN-ECHO sites further explored implementation information.
Results: Analysis of the interviews revealed three of 14 Consolidated Framework for Implementation Research constructs that
distinguished between low and high implementation sites: design quality and packaging; compatibility; and reflecting and
evaluating. The survey data generally supported these findings, while also revealing a fourth distinguishing construct – leadership
engagement. All sites expressed positive attitudes toward SCAN-ECHO, despite struggling with the complexity of programme
implementation.
Conclusions: Recommendations based on the findings include: (a) expend more effort in developing and distributing educa-
tional materials; (b) restructure the delivery process to improve programme compatibility; (c) establish an audit and feedback
mechanism for monitoring and improving the programme; (d) engage in more upfront planning to reduce complexity; and
(e) obtain local leadership support for providing primary care providers with dedicated time for participation.

Keywords
Implementation research, speciality care, chronic illness, telehealth, telemedicine, healthcare training, healthcare education

Date received: 3 May 2016; Date accepted: 12 October 2016

process and to provide PCPs with some training in speci-

Introduction ality care.5,6 Speciality care refers to services provided by
Timely access to speciality care services within a reason- experts with extensive training and education in a particu-
able distance is an ongoing challenge in Veterans Health lar clinical area such as pain management, diabetes, or
Administration (VHA), especially for patients in rural or cardiology.
economically impoverished areas1 where speciality care is
in demand but in shorter supply.1–4 A VHA initiative to
improve access to speciality care is the Specialty Care 1
Louis Stokes Cleveland VA Medical Center, USA
2
Access Network - Extension for Community Health Eastern Colorado Health Care System, USA
3
Outcomes (SCAN-ECHO) programme, which is designed VA Ann Arbor Health Care System, USA
to strengthen relationships between speciality care pro-
Corresponding author:
viders (SCPs) and primary care providers (PCPs) by pro- Lauren Stevenson, Louis Stokes Cleveland VA Medical Center, 10701 East
viding infrastructure, mechanisms, and guidance to Boulevard, Cleveland, OH 44106, USA.
facilitate and enhance the speciality care consultation Email: lauren.stevenson@va.gov

Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016

2 Journal of Telemedicine and Telecare 0(0)
Specifically, SCAN-ECHO is a medical education and
care delivery programme that uses a video-teleconferen-
cing-based platform and includes a 12–18 month curricu-
lum of didactic presentations. In addition, PCPs present
their complex patient cases and receive real-time consult-
ations from SCPs and other speciality care team members
such as clinical pharmacists, nurses, and health psycholo-
gists. Thus, specific patient needs are addressed while par-
ticipating PCPs gain knowledge and skills that allow them
to treat complex patients independently and offer
advice to fellow PCPs. This model has proven to be an
effective way to enable PCPs to provide care for complex
chronic conditions such as hepatitis C,7 diabetes8 and
chronic pain.9
A programme evaluation was funded to understand
what factors distinguished successful implementation
from less successful implementation, in order to inform
VHA’s dissemination efforts following the initial imple-
mentation. We applied the Consolidated Framework for
Implementation Research (CFIR) to categorise and quan-
tify distinguishing factors.10,11 The CFIR provides a com-
prehensive taxonomy of constructs from multiple
disciplinary domains that are likely to influence implemen-
tation of complex programmes, thereby offering a check-
list of potential areas for attention during the
implementation process, a systematic method for analysis,
and a set of common constructs that have been used in
previous implementation studies (Table 1).10,11
Methods
Data collection
A convergent mixed methods approach12 involving two
quantitative surveys and qualitative interviews was used
to examine implementation of SCAN-ECHO within
VHA. First, a Web-based survey was disseminated to all
SCAN-ECHO site leaders (n ¼ 52) (Supplementary
Material, Appendix A) in June 2012 to identify their per-
ceptions of the most important of the 39 CFIR constructs
related to successful implementation of SCAN-ECHO. A
semi-structured interview guide (Supplementary Material,
Appendix B) was then developed that focused on the
CFIR constructs viewed as important or very important
by the majority of the respondents to the initial survey.
Interviews were conducted with key informants approxi-
mately one-year post-introduction of the initiative. The
respondents were identified from nine sites (geographic-
site/speciality combinations), which were selected in a
stratified random sample from the initial 37 SCAN-
ECHO sites to reflect different degrees of implementation
success. Implementation success was defined as the
number of SCAN-ECHO sessions that had been con-
ducted by each of these sites at the time they were selected
to be part of the evaluation in July 2012 (Table 2). A
modified snowball sampling approach was used to recruit
interview participants; programme leads and primary care
directors were asked to identify SCPs, PCPs, other
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
providers (nurse practitioners, pharmacists), and support
staff (medical support assistants), who would be able to
provide information on implementation (Table 3).
The interviews (n ¼ 55) were conducted between
September 2012 and January 2013 via telephone by pairs
of analysts, one conducting the interview and the other
taking detailed notes. Interviews were digitally recorded
but not transcribed. The pairs reviewed and clarified inter-
view notes, referring to the recordings as needed.
A survey was then developed (Supplementary Material,
Appendix C), which consisted of questions addressing
those constructs viewed very positively or negatively by
any of the sites participating in the interviews (see Data
analysis section below). The primary objective of fielding
the survey was to determine if the qualitative findings were
confirmed by the quantitative survey findings (convergent
mixed methods design) from a larger sample of SCAN-
ECHO participants from all 77 sites participating in
SCAN-ECHO at the time of survey administration.
This evaluation was a quality improvement project
funded by VHA’s Office of Specialty Care Services/
Specialty Care Transformation (OSCT), conducted in
accordance with VHA guidelines for non-research projects
(VHA Handbook 1058.05), which do not require IRB
approval. Participation in the surveys and interviews
was completely voluntary, and the confidentiality of all
responses has been maintained.
Data analysis
A consensual qualitative analysis12 was used for the inter-
view responses. Interviewer pairs independently coded
interview notes by assigning responses to CFIR constructs
and rating the influence of each construct in the organisa-
tion (positive or negative) and magnitude or strength of its
influence10 (–2, –1, 0, þ1, þ2). Constructs not specifically
mentioned (missing) were distinguished from those with
ambiguous or neutral effects (rated 0). Pairs then met to
discuss and reach consensus on coding discrepancies.13–15
Ratings for each participant were aggregated to a site-
level rating for each CFIR construct. These ratings were
calculated as an average of ratings across respondents
within a site; ratings were given more or less weight in
the calculations depending on respondents’ knowledge of
SCAN-ECHO and their ability to provide detailed
responses. Significant variability (a difference of at least
two points) was noted if present among responses for a
given construct. This approach allowed for a systematic,
rapid comparison across sites.10 A site-ratings matrix for
each of the 14 CFIR constructs examined the extent to
which constructs were consistently coded as negative or
had high variability for sites with low use of SCAN-
ECHO, and coded as positive at sites with high use of
SCAN-ECHO.15,16
For the follow-up survey, overall percentages of
respondents agreeing or strongly agreeing with survey
statements were calculated, and low implementation sites
were compared to medium and high implementation sites.
Stevenson et al.
Table 1. Consolidated Framework for Implementation Research (CFIR) constructs.
Topic/Description
I. INTERVENTION CHARACTERISTICS
A Intervention Source
B Evidence Strength & Quality
C Relative advantage
D Adaptability
E Trialability
F Complexity
G Design Quality & Packaging
H Cost
II. OUTER SETTING
A Patient Needs & Resources
B Cosmopolitanism
C Peer Pressure
D External Policy & Incentives
III. INNER SETTING
A Structural Characteristics
B Networks & Communications
C Culture
D Implementation Climate
1 Tension for Change
2 Compatibility
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
3
Short Description
PERCENTAGEþ
54 Perception of key stakeholders about whether the intervention is
externally or internally developed.
Stakeholders’ perceptions of the quality and validity of evidence
supporting the belief that the intervention will have desired
outcomes.
88 Stakeholders’ perception of the advantage of implementing the
intervention versus an alternative solution.
97 The degree to which an intervention can be adapted, tailored,
refined, or reinvented to meet local needs.
77 The ability to test the intervention on a small scale in the organ-
ization [8], and to be able to reverse course (undo implemen-
tation) if warranted.
88 Perceived difficulty of implementation, reflected by duration,
scope, radicalness, disruptiveness, centrality, and intricacy and
number of steps required to implement
88 Perceived excellence in how the intervention is bundled, pre-
sented, and assembled
73 Costs of the intervention and costs associated with implementing
that intervention including investment, supply, and opportunity
costs.
85 The extent to which patient needs, as well as barriers and facili-
tators to meet those needs are accurately known and prioritized
by the organization.
66 The degree to which an organization is networked with other
external organizations.
65 Mimetic or competitive pressure to implement an intervention;
typically because most or other key peer or competing organ-
izations have already implemented or in a bid for a competitive
edge.
64 A broad construct that includes external strategies to spread
interventions including policy and regulations (governmental or
other central entity), external mandates, recommendations and
guidelines, pay-for-performance, collaboratives, and public or
benchmark reporting.
66 The social architecture, age, maturity, and size of an organization.
88 The nature and quality of webs of social networks and the nature
and quality of formal and informal communications within an
organization.
89 Norms, values, and basic assumptions of a given organization.
The absorptive capacity for change, shared receptivity of involved
individuals to an intervention and the extent to which use of that
intervention will be rewarded, supported, and expected within
their organization.
76 The degree to which stakeholders perceive the current situation
as intolerable or needing change.
85 The degree of tangible fit between meaning and values attached to
the intervention by involved individuals, how those align with
individuals’ own norms, values, and perceived risks and needs,
and how the intervention fits with existing workflows and
systems.
(continued)
4 Journal of Telemedicine and Telecare 0(0)

Table 1. Continued

Topic/Description Short Description

3 Relative Priority 88 Individuals’ shared perception of the importance of the imple-

4 Organizational Incentives &
Rewards
5 Goals and Feedback
6 Learning Climate
E Readiness for Implementation
1 Leadership Engagement
2 Available Resources
Code separately:
 Funding
 Staffing
 HR processes/support
 Training
 IT Support
 Administrative Support
3 Access to Knowledge &
Information
mentation within the organization.
61 Extrinsic incentives such as goal-sharing awards, performance
reviews, promotions, and raises in salary and less tangible
incentives such as increased stature or respect.
80 The degree to which goals are clearly communicated, acted upon,
and fed back to staff and alignment of that feedback with goals.
92 A climate in which: a) leaders express their own fallibility and need
for team members’ assistance and input; b) team members feel
that they are essential, valued, and knowledgeable partners in
the change process; c) individuals feel psychologically safe to try
new methods; and d) there is sufficient time and space for
reflective thinking and evaluation.
Tangible and immediate indicators of organizational commitment
to its decision to implement an intervention.
92 Commitment, involvement, and accountability of leaders and
managers with the implementation.
96 The level of resources dedicated for implementation and on-going
operations including money, training, education, physical space,
and time.
 Availability of funding for the new program.
 Are the needed people in place (regardless of whether they are
newly hired or existing staff who have been re-assigned)?
 Has HR provided timely and helpful support for hiring any new
people needed?
 Has there been helpful training?
 Has IT support been readily available and helpful?
 Is administrative support (e.g., coordinating and scheduling
meetings, working with IT and HR, etc.) provided?
85 Ease of access to digestible information and knowledge about the
intervention and how to incorporate it into work tasks.

IV. CHARACTERISTICS OF INDIVIDUALS

A Knowledge & Beliefs about the 85
Intervention
B Self-efficacy
C Individual Stage of Change
D Individual Identification with
Organization
E Other Personal Attributes
Individuals’ attitudes toward and value placed on the intervention
as well as familiarity with facts, truths, and principles related to
the intervention.
84 Individual belief in their own capabilities to execute courses of
action to achieve implementation goals.
81 Characterization of the phase an individual is in, as he or she
progresses toward skilled, enthusiastic, and sustained use of the
intervention.
72 A broad construct related to how individuals perceive the
organization and their relationship and degree of commitment
with that organization.
76 A broad construct to include other personal traits such as toler-
ance of ambiguity, intellectual ability, motivation, values, com-
petence, capacity, and learning style.

V. PROCESS
A Planning 96 The degree to which a scheme or method of behavior and tasks
for implementing an intervention are developed in advance and
the quality of those schemes or methods.
B Engaging Attracting and involving appropriate individuals in the implemen-
tation and use of the intervention through a combined strategy
of social marketing, education, role modeling, training, and other
similar activities.
(continued)

Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016

Stevenson et al. 5

Table 1. Continued

Topic/Description Short Description

1 Opinion Leaders 92 Individuals in an organization who have formal or informal influ-

2 Formally appointed internal
implementation leaders
3 Champions
4 External Change Agents
5 Key Stakeholders
C Executing
D Reflecting & Evaluating
a
Percentage of respondents (n ¼ 22) who rated potential implementation factors as important or very important.
Source: Damschroder L, Aron D, Keith R, et al. Fostering implementation of health services research findings into practice: A consolidated framework for
advancing implementation science. Implement Sci 2009; 4: 50.17
ence on the attitudes and beliefs of their colleagues with respect
to implementing the intervention
88 Individuals from within the organization who have been formally
appointed with responsibility for implementing an intervention as
coordinator, project manager, team leader, or other similar role.
88 ‘‘Individuals who dedicate themselves to supporting, marketing,
and ‘driving through’ an [implementation]’’ [101](p. 182), over-
coming indifference or resistance that the intervention may
provoke in an organization.
44 Individuals who are affiliated with an outside entity who formally
influence or facilitate intervention decisions in a desirable
direction.
88 The inclusion of stakeholders in the development and implemen-
tation of the intervention/program. A stakeholder is a person
who can affect or be affected by the intervention/program.
92 Carrying out or accomplishing the implementation according
to plan.
88 Quantitative and qualitative feedback about the progress and
quality of implementation accompanied with regular personal
and team debriefing about progress and experience.

Table 2. Number of Specialty Care Access Network-Extension for Table 4. Number of Specialty Care Access Network-Extension
Community Healthcare Outcomes (SCAN-ECHO) sessions at nine for Community Healthcare Outcomes (SCAN-ECHO) sessions/
sites participating in semi-structured interviews. number of outpatient visits at sites participating in surveys.
Implementation level Degree of Number of Minimum Maximum
Implementation site/speciality units rate rate
Site Specialty Low Medium High
Low rate 8 0.00022 0.00188
Hepatitis C 5 56 153
Medium rate 8 0.00269 0.00955
Pain 11 48 74
High rate 8 0.00970 0.04459
Diabetes 13 29 154

Degree of implementation was measured as number of

Table 3. Number and roles of interview respondents by site. SCAN ECHO sessions conducted from 12/2012 through
2/2013 as a percentage of outpatient visits at each facility,
Site (implementation
level/speciality) Participants Role
as shown in Table 4. Only sites with at least one SCAN-
ECHO session, and only respondents from these sites,
Low/hep C 3 2 CL, 1 SCP were included in the analysis.
Low/pain 7 1 CL, 2 OS, 2 PCP, 2 SCP
Low/diabetes 9 2 CL, 3 PCP, 2 SCP, 2 OS
Results
Medium/hep C 4 1 CL, 1 PCP, 2 SCP
Medium/pain 6 1 CL, 1 OP, 2 PCP, 1 OS, 1 SCP Twenty-two SCAN-ECHO leaders responded to the ini-
Medium/diabetes 5 1 CL, 1 OS, 1 PCP, 1 SCP, 1 OP tial survey, for a response rate of 42%. Fourteen of the 39
CFIR constructs were viewed as important or very
High/hep C 6 2 CL, 1 OS, 2 PCP, 1 SCP
important by at least 88% of the respondents; these are
High/pain 8 1 CL, 1 OS, 3 PCP, 2 SCP, 1 OP
shown in Table 1, and were used to design the interview
High/diabetes 7 1 CL, 1 OS, 3 PCP, 2 SCP
guide.
Total 55 Analysis of interviews revealed three CFIR constructs
CL: clinical leader; OP: other provider; OS: other support staff; PCP: primary that distinguished between low and high implementation
care provider; SCP: speciality care provider. sites: design quality and packaging, compatibility, and

Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016

6 Journal of Telemedicine and Telecare 0(0)
Table 5. Results matrix for the most relevant Consolidated Framework for Implementation Research (CFIR) constructs.
CFIR construct Low
a
Compatibility 0 0
Reflecting and evaluatinga 0 0
Design quality and packaginga 1 1
Complexity 1 2
Knowledge and beliefs 1 1
Leadership engagement 0 1
Complexity and Knowledge and beliefs were deemed to be important constructs because they were rated consistently as negative or positive (respectively)
across both low and high implementation sites. Leadership engagement was a distinguishing construct based on the survey responses, but not the interview
responses (rated inconsistently across sites, see matrix above).
a
To be a distinguishing construct, a construct had to be rated as negative or 0 across all low sites, and have at least two positive ratings in the high sites.
reflecting and evaluating. Complexity was a challenge (i.e.
rated negatively) for all sites; in contrast, knowledge and
beliefs about the intervention were consistently positive
across all sites. Table 5 shows the interview results
matrix for these constructs. A summary of sites’ percep-
tions of these constructs is provided below, and Table 6
presents quotations from low versus high implementation
sites for each of these constructs.
Survey responses were received from 198 of 450 pro-
viders (44%), but only 81 (18%) were from sites that had
conducted at least one SCAN-ECHO session from
December 2012–February 2013. The interview responses
were generally supported by the survey responses, with the
exception of Leadership Engagement. This construct did
not show up as an important construct in the interview
responses, but was a distinguishing factor in the survey
responses. The extent to which the survey responses sup-
ported the interview findings is presented for each of the
important constructs below.
Design quality and packaging
The quality of a programme’s design and the way in which
it is presented can contribute to implementation success (if
done well) or failure (if done poorly). Low implementa-
tion sites reported a failure to understand what SCAN-
ECHO should look like or expressed confusion about the
programme. High implementation sites felt that the infor-
mation they received was sufficiently clear for them to
successfully implement the programme.
These results were somewhat supported by survey data,
which revealed differences in understanding of SCAN-
ECHO among SCPs across sites: 75% of SCPs (who
were responsible for planning and conducting training ses-
sions) at low implementation sites reported the guidelines
and expectations of the SCAN-ECHO initiative were
clear, compared with 83% at medium implementation
sites, and 100% at high implementation sites. However,
this difference was not significant. In addition, 69% of
SCP respondents at low implementation sites, 71% at
medium implementation sites, and 93% at high implemen-
tation sites agreed with the statement, ‘I received adequate
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
High
0 1 1 1
1 0 1 1
0 1 1 0
2 2 1 2
1 1 2 2
1 1 2 0
instruction and support in terms of how to organise and
implement the SCAN-ECHO initiative’ (p ¼ 0.10).
Compatibility
Compatibility, or the extent to which a programme fits
with existing workflows and systems, is important for
facilitating successful implementation. SCPs at high
implementation sites believed that SCAN-ECHO was
compatible with the existing delivery model, improving
its efficiency. However, at low implementation sites there
was concern that the programme might interfere with the
role of SCPs or duplicate efforts.
A large majority of survey respondents (94%) agreed
that the SCAN-ECHO programme ‘is appropriate for our
practice setting’ and ‘is a good model for how medicine
should be practiced’ (88%), but there was no noticeable
trend across implementation level. There was a slight
trend across implementation level in the percentage of
providers who agreed with the statement, ‘SCAN-ECHO
has improved the efficiency of speciality care referrals’
(69%, 74% and 85% agreeing from low, medium, and
high implementation sites, respectively, p ¼ ns).
Reflecting and evaluating
Obtaining feedback about a programme enables sites to
adjust/adapt as needed. High implementation sites
engaged in reflection and evaluation, including conducting
surveys for informing programme adjustments, while low
implementation sites did not regularly do so. No survey
questions addressed the construct of reflecting and
evaluating.
Complexity
Intervention complexity leads to implementation chal-
lenges. Responses regarding complexity were consistently
rated as negative (high complexity) across all sites.
Complexity included acquiring new tele-health equipment,
scheduling, hiring new staff or changing job duties,
and establishing access to video-teleconferencing.
Stevenson et al.
Table 6. Illustrative quotes comparing relevant Consolidated Framework for Implementation Research (CFIR) constructs at low vs high
implementation sites.
CFIR construct
Design quality and packaginga
Compatibilitya
‘However, I have heard some SCPs expressing
Reflecting and evaluatinga
Complexityb
Knowledge and beliefs about
the interventionc
Low
‘The whole programme seems to have put the
cart before the horse. To me it seems there
wasn’t enough stuff decided and concrete at the
national level before they started implement-
ing . . . . When I got materials it was much later
on and I didn’t really look at them. It didn’t feel
like there was any guidance’.
‘I think that, boy, we can’t all be mini-specialists in
everything and I’m a specialist in rheumatology
in addition to being a primary care doc and that
took 3 years of training. I think that a mini-
specialist, I just think that’s rhetoric. We all
need to be good generalists and know when to
refer to specialist. To be mini-specialist that’s
not credible to me’.
concerns about making themselves obsolete,
are we going to be out of a job if we train all
these PCPs?’.
‘Our experience is so small I really haven’t had
any feedback other than they keep participating
every month so they must be getting something
out of it’.
‘It was extremely difficult because we didn’t have
a good business plan. Now that we’ve made all
the mistakes and solved it we have helped other
groups. We were pioneers and there wasn’t a
good plan to follow’.
‘I’m a Veteran myself and saw the issues i.e. access
is the biggest issue we have. The old days are
gone, but still a lot of animosity there. SCAN is
fixing that, I really believe that’s what it’s
doing. . .. It’s really great. And yet it’s been
hampered by logistics. It’s forward thinking’.
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
7
High
Respondent noted that the guidelines sent out by
VA [Veteran Affairs] Central Office were ‘rea-
sonable’. ‘We know what they wanted’. ‘Monthly
calls were helpful, to know the direction’.
‘They did give us some wall hangings i.e. the spine so
I can explain to patients where the problem is,
and they gave handouts to give to patients. These
things were helpful, I didn’t know what to expect
but the quality seems good’.
‘It [SCAN-ECHO] seems to integrate very well.
There was some question about how it would
integrate with e-consults. What we found is
that the providers don’t see it as duplicative at
all . . . with SCAN-ECHO it seems that, through
the discussion and the back and forth, PCPs are
actually learning what to do and next time they
know what to do, so the consult doesn’t come
back again for new patient. That’s fabulous and
there is a place for both SCAN-ECHO and e-
consults. It’s good that in VA you can convert
consult from one type to another. Everybody
learns, it integrates quite well into workflow’.
‘The feedback is from the surveys we send out
weekly. We have received emails from some of
our participants at CBOCs [VA community-
based outpatient clinics] or other facilities that
don’t have the specialists we have here that they
have improved the service they’re giving to the
vets by the knowledge they’ve gained and edu-
cation received [through SCAN-ECHO]. The
response has been overwhelmingly positive’.
‘Initially we had hurdles. I would say it was com-
plicated. Right now it’s seamless because we
have an infrastructure. If anything it has
improved communications. I don’t want to put
SCAN and complex in the same statement’.
‘Technological, as with any new process it took a
while to get the bugs worked out. There were
technical errors made, a rough start, had to
develop a communication protocol. Things have
been working dandy ever since’.
‘Hindsight’s 20/20 and now that we have 5 clinics
up and running it gets easier every time. When
we first started, it was a 5 [very difficult]. With
each clinic it’s gotten easier, it’s now down to a
2 or 1.5 now [with 1 being least difficult]. [You]
learn from your experience and you don’t
stumble at that point the next time’.
’I think that this is a solid idea. Essentially in a
system we have some of our providers, par-
ticularly rural, it’s really the right idea to help
them. Even if you take away all the other factors
you get patients living in rural setting with not
much funding is a lot of travel costs for VA to
(continued)
8 Journal of Telemedicine and Telecare 0(0)
Table 6. Continued
CFIR construct Low
Leadership engagementd ‘We really do need full buy-in from leadership at
all levels. Without that, I really do believe the
programme will fail, need to have the pro-
gramme solidified like that’.
‘To be quite frank with you, the SCAN programme
hasn’t taken root yet at our facility. We have
some providers who would like to participate
but they can’t because of time’.
PCP: primary care provider; SCAN: Specialty Care Access Network; SCAN-ECHO: Specialty Care Access Network-Extension for Community Healthcare
Outcomes; SCP: speciality care provider.
a
Distinguishing constructs between low implementation sites and high implementation sites; bconsistently negative across sites; cconsistently positive across
sites; dinconsistent across sites in interview responses: distinguishing construct in survey responses.
Although not a distinguishing characteristic between sites,
40% of all survey respondents agreed with the statement,
‘The hiring process at this facility has impeded our ability
to hire necessary SCAN-ECHO staff in a timely manner’.
Knowledge and beliefs about the intervention
Individuals’ positive attitudes toward and value placed on
a programme can aid in its implementation. Conversely,
negative impressions can raise significant barriers.
Fortunately for SCAN-ECHO, most PCPs saw benefits
of acquiring new skills, honing knowledge, and having
access to speciality expertise.
Of all surveyed, 86% agreed that SCAN-ECHO
increased PCPs’ knowledge and competencies, with no
noticeable trends across level of implementation. PCPs
generally believed they were able to apply knowledge
gained, with >75% agreeing or strongly agreeing with
the following statements: (a) ‘With the information I
receive from the SCAN-ECHO sessions, I can provide
care for the typical patient in that speciality without fur-
ther referrals’, (b) ‘The information I’ve received from
SCAN-ECHO cases has been useful in treating other
patients in my panel’, and (c) ‘SCAN-ECHO has
improved my ability to manage and treat patients with
this condition’.
Leadership engagement
Leadership engagement was not identified as a distin-
guishing construct based on the interview responses;
there were both positive and negative comments across
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
High
transport them or postpone or avoid treatment
and things will be further along and harder to
manage’.
‘Front office is innovative, embraces technology.
So for them this is a new wave of the future so
they hopped on and drove that train’.
‘The administration of the medical center has gone
out of their way both to try to recruit and build
interest in the program. We’ve had retreats
both last year and this year about the SCAN-
ECHO project, bringing all the participants to
one place, at its kickoff and this year to gain
feedback and explain next steps. Without a
doubt the retreats helped the programme
progress, recruit, and maintain participation’.
‘Much easier when their leadership approves and
agrees to it. Makes all the difference. If leader-
ship gives them time to come, then they almost
always continue’.
the sites about support from leadership. For example,
leadership provided technological support and resources
to attend training sessions, but not dedicated time to par-
ticipate in the SCAN-ECHO sessions on a routine basis or
to set up ‘mini-speciality’ clinics.
Survey responses indicated that limited protected time
was indeed a barrier to participation for PCPs and likely a
major contributor to concerns with developing and apply-
ing speciality expertise. Nearly half (45%) of respondents
believed the programme has increased their workload.
Only 16% of PCPs said they had protected time to serve
as mini-specialists. Survey responses showed differences
across sites in protected time for PCPs to attend SCAN-
ECHO sessions: 83% and 82% of PCPs from low and
medium implementation sites, respectively, said they had
protected time, compared with 92% at high implementa-
tion sites (p ¼ 0.08). This was further supported by agree-
ment with ‘Our clinical leadership considers SCAN-
ECHO a valuable use of my time’ at 69%, 67% and
88% at low, medium, and high implementation sites,
respectively (p ¼ 0.12).
Discussion
The survey and interview data consistently identified con-
structs that differentiated lower from higher implementa-
tion sites, although differences across sites in the survey
data were not statistically significant. These constructs
included issues of compatibility with existing care
models, lack of feedback on the programme, complexity
of implementation, and failure of leadership to provide
protected time for participation. Sites were consistently
Stevenson et al.
positive in beliefs about programme benefits, something
needed for successful implementation, but insufficient to
guarantee it.
Implementation challenges identified from the inter-
views and surveys point toward suggestions for improving
implementation processes in low performing sites as well
as in new sites who wish to start SCAN-ECHO pro-
grammes. Powell et al.18 provide a comprehensive list of
implementation strategies, classified into six categories
representing larger implementation processes: planning,
educating, restructuring, financing, managing quality,
and attending to the policy context. Specific strategies
in five of these categories are presented below for address-
ing issues identified in this study.
Design quality and packaging
This falls under the category of ‘educating.’ Potential stra-
tegies to address issues cited by participants include
developing and distributing effective educational materials
(guidelines, manuals, toolkits, etc.) ‘in ways that make
it easier for stakeholders to learn about the innovation
and for clinicians to learn how to deliver the clinical
innovation’.10
Compatibility
Interview and survey data showed a number of providers
had concerns about PCPs taking on responsibilities trad-
itionally handled by SCPs and worried there might be
differences in the degree of integration of SCAN-ECHO
into existing delivery processes across sites. These con-
cerns could potentially be addressed under ‘restructure’
strategies, which ‘facilitate implementation by altering
staffing, professional roles, physical structures, equip-
ment, and data systems’ (p ¼ 135).18 Adjusting primary
care staffing and team composition may be helpful, such
as engaging more nurse practitioners in trainings. In add-
ition, finding ways to integrate the programme into the
existing consultation process using quality improvement
strategies could avoid duplication of effort and improve
efficiency of the process for requesting and obtaining
consultations from SCPs, thereby improving
compatibility.
Reflecting and evaluating
This issue is most appropriately addressed using a ‘quality
management’ strategy, such as quality monitoring or audit
and feedback.18 Sites would benefit from an examination
of data and feedback from their own SCAN-ECHO pro-
grammes and also from sharing data with other sites.
Complexity
Tackling a complex intervention generally requires
increased attention to planning strategies, which include
conducting local needs assessment, assessing for readiness,
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
9
identifying specific barriers, and developing a formal
implementation blueprint.10 It is important that sites
devote sufficient time and energy to these processes.
Leadership support
Obtaining buy-in from leaders, especially in the form of
protected time for providers participating in SCAN-
ECHO, also falls under the category of planning. One of
the strategies for obtaining necessary support is to obtain
formal commitments from leaders, which should be done
prior to devoting resources to implementation.
Our findings are consistent with those from other stu-
dies examining the factors associated with the successful
implementation of telehealth programmes. For example,
in an in-depth literature review of 225 articles focused on
videoconferencing in clinics, one finding was the import-
ance of developing protocols for using the system, which is
a component of packaging the programme19 and other
studies have supported these findings.20 Perceived useful-
ness of and interest in telemedicine has previously shown
to increase probability of PCPs using telemedicine
options.21 Previous research has also shown the import-
ance of reflecting and evaluating related to telehealth nur-
sing services22 and the challenges associated with
integrating new and ‘different’ models of practice into
deeply embedded local working structures (compatibil-
ity).23 Past telehealth efforts by the VHA have been
successful when focusing on planning to reduce the
complexity of implementation.24 Other research studying
telemedicine has noted leadership support as a key factor
to success.25
However, most of these other studies did not conduct a
systematic examination of the association between organ-
isational factors of interest and a quantitative measure of
implementation success, as we did. Instead, previous stu-
dies that have conducted qualitative assessments of the
key factors affecting implementation success have tended
to identify these factors based on the judgement of the
interview respondents,25 i.e. the respondents are asked to
make an assessment of which factors they perceive are
most important, which is essentially what we did with
our initial survey of SCAN-ECHO leaders. However, we
took the next step of trying to link these factors with
actual implementation outcomes, to verify (or not) the
judgement of the participants. Furthermore, reviews of
previously published studies often do not provide details
on the specific methods employed by the studies included
in the review,26 so it is impossible to determine whether
the key factors identified from these reviews are based on
perceptions or empirical data. In their literature review of
successful telemedicine interventions, Broens et al. noted
‘Currently, there is no generally accepted methodology to
systematically identify and score determinants for tele-
medicine implementations’ (p ¼ 307).26 We hope that
our approach provides one option for such a systematic
approach. Findings from our study and others that
employ a similar methodology can meet the need for
10
research to ‘prove the catalytic validity of our explanatory
propositions and their predictive power’.23
Limitations
Limitations of this study include the use of only nine sites
for our qualitative interviews, which limits generalisability
of our findings. A mixed methods approach with surveys
sent to all SCAN-ECHO participants was used to partially
address this and allowed for confirmation of the qualita-
tive findings. Nevertheless, although our survey data
included a larger sample, it was still insufficient for achiev-
ing significance of most of our findings. Furthermore, this
project was conducted within a large, integrated health-
care system; it is not known if similar findings would be
obtained in smaller, community healthcare systems.
Identifying constructs associated with SCAN-ECHO
implementation success from both interview and survey
responses focuses recommendations for improving dis-
semination of SCAN-ECHO on the following (a) expend
more effort in developing and distributing educational
materials; (b) restructure the delivery process to improve
programme compatibility; (c) establish an audit and feed-
back mechanism for monitoring and improving the pro-
gramme; (d) engage in more upfront planning to reduce
complexity; and (e) obtain local leadership support for
providing PCPs with dedicated time for participation.
Implications for practice
While the key constructs affecting implementation success
that were identified in this study have been identified in
other studies, our approach for identifying these con-
structs is unique and can be replicated to provide a sys-
tematic, efficient means for prioritising efforts to improve
implementation success. As the largest integrated health-
care system in the USA, the VHA is uniquely positioned
to conduct such an evaluation of large-scale implementa-
tion efforts.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) disclosed receipt of the following financial support
for the research, authorship, and/or publication of this article:
This material is based upon work supported by the US
Department of Veterans Affairs, the Office of Specialty Care
Transformation and the Office of Research and Development
Quality Enhancement Research Initiative.
References
1. McQueen L, Mittman BS and Demakis JG. Overview of
the Veterans Health Administration (VHA) Quality
Enhancement Research Initiative (QUERI). J Am Med
Inform Assoc 2004; 11: 339–343.
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
Journal of Telemedicine and Telecare 0(0)
2. Rosenblatt RA, Andrilla CH, Catlin M, et al. Geographic
and specialty distribution of US physicians trained to treat
opioid use disorder. Ann Fam Med 2015; 13: 23–26.
3. Baldwin LM, Andrilla CH, Porter MP, et al. Treatment of
early-stage prostate cancer among rural and urban patients.
Cancer 2013; 119: 3067–3075.
4. Rosenblatt RA, Andrilla CH, Curtin T, et al. shortages of
medical personnel at community health centers: Implications
for planned expansion. JAMA 2006; 295: 1042–1049.
5. Kirsh S, Ho P and Aron D. Providing specialty consultant
expertise to primary care: An expanding spectrum of mod-
alities. Mayo Clin Proc 2014; 89: 1416–1426.
6. Kirsh S, Su G, Sales A, et al. Access to outpatient specialty
care: Solutions from an integrated health care system. Am J
Med Qual 2014; 30(1): 88–90.
7. Arora S, Geppert CM, Kalishman S, et al. Academic
health center management of chronic diseases through
knowledge networks: Project ECHO. Acad Med 2007; 82:
154–160.
8. Colleran K, Harding E, Kipp BJ, et al. Building capacity to
reduce disparities in diabetes: Training community health
workers using an integrated distance learning model.
Diabetes Educ 2012; 38: 386–396.
9. Dubin RE, Flannery J, Taenzer P, et al. ECHO Ontario
chronic pain & opioid stewardship: Providing access and
building capacity for primary care providers in underser-
viced, rural, and remote communities. Stud Health Technol
Inform 2015; 209: 15–22.
10. Damschroder LJ and Lowery JC. Evaluation of a large-scale
weight management program using the consolidated frame-
work for implementation research (CFIR). Implement Sci
2013; 8: 51.
11. Kirk MA, Kelley C, Yankey N, et al. A systematic review of
the use of the Consolidated Framework for Implementation
Research. Implement Sci 2016; 11–72.
12. Cresswell JW. A concise introduction to mixed methods
research. Los Angeles: Sage, 2015.
13. Hill CE, Knox S, Thompson BJ, et al. Consensual qualita-
tive research: An update. J Couns Psychol 2005; 52: 196–205.
14. Hill CE, Thompson BJ and Williams EN. A guide to con-
ducting consensual qualitative research. Couns Psychol 1997;
25: 517–572.
15. Rihoux B and Ragin CC. Why compare? Why configur-
ational comparative methods? In: Rihoux B and Ragin CC
(eds) Configurational comparative methods Los Angeles:
Sage, 2009, pp.xvii–xx.
16. Averill JB. Matrix analysis as a complementary analytic
strategy in qualitative inquiry. Qual Health Res 2002; 12:
855–866.
17. Damschroder L, Aron D, Keith R, et al. (2009) Fostering
implementation of health services research findings into
practice: A consolidated framework for advancing imple-
mentation science. Implement Sci 4(1): 50.
18. Powell BJ, McMillen C, Proctor EK, et al. A compilation of
strategies for implementing clinical innovations in health
and mental health. Med Care Res Rev 2012; 69: 123–157.
19. Jarvis-Selinger S, Chan E, Payne R, et al. Clinical telehealth
across the disciplines: Lessons learned. Telemed J E Health
2008; 14: 720–725.
20. Lee A and Billings M. Telehealth implementation in a skilled
nursing facility: Case report for physical therapist practice in
Washington. Phys Ther 2016; 96: 252–259.
Stevenson et al.
21. Saigi-Rubı́ó F, Jiménez-Zarco A and Torrent-Sellens J.
Determinants of the intention to use telemedicine:
Evidence from primary care physicians. Int J Technol
Assess Health Care 2016; 32: 29–36.
22. Vinson M, McCallum R, Thornlow DK, et al. Design,
implementation, and evaluation of population-specific tele-
health nursing services. Nurs Econ 2011; 29: 265.
23. May C, Harrison R, Finch T, et al. Understanding the nor-
malization of telemedicine services through qualitative
evaluation. J Am Med Inform Assoc 2003; 10: 596–604.
24. Ohl M, Dillon D, Moeckli J, et al. Mixed-methods evalu-
ation of a telehealth collaborative care program for persons
Downloaded from jtt.sagepub.com at CORNELL UNIV on December 6, 2016
11
with HIV infection in a rural setting. J Gen Intern Med 2013;
28: 1165–1173.
25. Wade VA, Taylor AD, Kidd MR, et al. Transitioning a
home telehealth project into a sustainable, large-scale ser-
vice: A qualitative study. BMC Health Serv Res 2016; 16:
183–193.
26. Broens THF, Huis in’t Veld RMHA, Vollenbroek-Hutten
MMR, et al. Determinants of successful telemedicine imple-
mentations: A literature study. J Telemed Telecare 2007; 13:
303–309.