Journal of the American Medical Informatics Association, 00(0), 2021, 1–10

doi: 10.1093/jamia/ocab010
Review

Review

Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021

Toolkits for implementing and evaluating digital health: A
systematic review of rigor and reporting
Myron Anthony Godinho ,1 Sameera Ansari,1 Guan Nan Guo ,1,2 and
Siaw-Teng Liaw 1

1
WHO Collaborating Centre on eHealth (AUS-135), School of Population Health, UNSW Sydney, New South Wales, Australia, 2Na-
tional Drug and Alcohol Research Centre, University of New South Wales, Sydney, New South Wales, Australia

Corresponding Author: Emeritus Prof. Dr. Siaw-Teng Liaw, Collaborating Centre on eHealth (AUS-135), School of Popula-
tion Health, UNSW Sydney, NSW 2052, Australia (siaw@unsw.edu.au)
Received 16 September 2020; Revised 8 January 2021; Editorial Decision 11 January 2021; Accepted 15 January 2021

ABSTRACT

Objective: Toolkits are an important knowledge translation strategy for implementing digital health. We studied
how toolkits for the implementation and evaluation of digital health were developed, tested, and reported.
Materials and Methods: We conducted a systematic review of toolkits that had been used, field tested or evalu-
ated in practice, and published in the English language from 2009 to July 2019. We searched several electronic
literature sources to identify both peer-reviewed and gray literature, and records were screened as per system-
atic review conventions.
Results: Thirteen toolkits were eventually identified, all of which were developed in North America, Europe, or
Australia. All reported their intended purpose, as well as their development process. Eight of the 13 toolkits in-
volved a literature review, 3 did not, and 2 were unclear. Twelve reported an underlying conceptual framework,
theory, or model: 3 cited the normalization process theory and 3 others cited the World Health Organization and
International Telecommunication Union eHealth Strategy. Seven toolkits were reportedly evaluated, but details
were unavailable. Forty-three toolkits were excluded for lack of field-testing.
Discussion: Despite a plethora of published toolkits, few were tested, and even fewer were evaluated. Methodo-
logical rigor was of concern, as several did not include an underlying conceptual framework, literature review,
or evaluation and refinement in real-world settings. Reporting was often inconsistent and unclear, and toolkits
rarely reported being evaluated.
Conclusion: Greater attention needs to be paid to rigor and reporting when developing, evaluating, and report-
ing toolkits for implementing and evaluating digital health so that they can effectively function as a knowledge
translation strategy.

Key words: digital health, toolkit, framework, implementation, evaluation, eHealth

INTRODUCTION
The 2018 World Health Assembly Resolution 71.7 on digital health
recognized the role of digital technologies in achieving universal
health coverage and other targets of the Sustainable Development
Goals, urging stakeholders “to assess their use of digital technolo-
gies for health. . . and to prioritise [their] development, evaluation,
implementation, scale-up and greater use. . .”1 To this end, toolkits
are being increasingly used to guide the implementation and evalua-
tion of digital health interventions and systems. Toolkits are a

C The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association.
V
All rights reserved. For permissions, please email: journals.permissions@oup.com
1
2 Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0
knowledge translation (KT) strategy used to communicate messages
or share decision aids, tools, or goods to improve health, educate,
or change practice or behavior among diverse populations—includ-
ing patients, carers, clinical and managerial health professionals,
policymakers, community and health organizations—and the health
system.2
In 2012, the World Health Organization (WHO) and Interna-
tional Telecommunication Union (ITU) developed the National
eHealth Strategy Toolkit, which provides guidance to governments
on the tools and processes to be considered in the development, im-
plementation, monitoring and evaluation of a national strategy.3
This WHO-ITU toolkit has prompted many public and private
organizations, including global aid agencies, to design domain-
specific toolkits, interactively mapping and addressing the digital
health strategy, universal health coverage, and Sustainable Develop-
ment Goals. Examples of such toolkits are the COCIR eHealth Tool-
kit by the European Coordination Committee of the Radiological,
Electromedical and Healthcare IT Industry4 and the Global Digital
Health Index Indicator Guide.5 However, little is known about the
success or otherwise of the use of these toolkits. Significant resources
are needed to produce such toolkits, underscoring the need that they
be of adequate quality and have evidence from user and utility testing
and evaluation that they actually “work” in the real world. The
WHO Collaborating Centre for eHealth (AUS-135) was tasked with
conducting a systematic review on implementation and evaluation of
toolkits for digital health, given the emerging evidence and lack of
syntheses on this topic. The review is the first part of a wider project
to support WHO activities in digital health strategy development, im-
plementation, capacity building, and evaluation.6
Background
A 2014 scoping review of toolkits used to disseminate health knowl-
edge or support practice change found that only 31 (37%) of the 83
toolkits included had been evaluated to any extent.2 The majority
(70%) did not specify the evidence base from which they draw, and
their effectiveness as a KT strategy is rarely assessed. To truly inform
health and health care, toolkits should include comprehensive
descriptions of their content, be explicit regarding content that is evi-
dence based, and include an evaluation of their effectiveness as a KT
strategy, addressing both clinical and implementation outcomes.2
This message is reinforced by a 2015 systematic review that con-
cluded that toolkits should be informed by high-quality evidence and
theory, and should be evaluated using rigorous study designs to ex-
plain the factors underlying their effectiveness and successful imple-
mentation.7 A recent qualitative study of clinic and community
perceptions of a general intervention toolkit asserted that “unless the
toolkit is used, it won’t help solve the problem.” The authors recom-
mended that studies be conducted to determine when and how tool-
kits are used. Funders, policymakers, researchers, and leaders in
primary care and public health should allocate resources to foster
toolkit development, testing, implementation, and evaluation.8
A cursory search of the literature found no reviews of digital
health toolkits, including for implementation or evaluation. To the
best of our knowledge, there are currently no defined standards or
critical appraisal checklists for measuring or evaluating the quality
of digital health toolkits.
Defining key terms
The Agency for Healthcare Research and Quality (AHRQ) defines a
toolkit as “a collection of related information, resources, or tools
that together can guide users to develop a plan or organise efforts to
follow evidence-based recommendations or meet evidence-based
specific practice standards.”9 The AHRQ also defines a tool as “an
instrument (e.g., survey, guidelines, or checklist) that helps users ac-
complish a specific task that contributes to meeting a specific
evidence-based recommendation or practice standard.” As a KT
strategy, toolkits should be concise, understandable and clearly fo-
cused, to enable the user to understand and operationalize or evalu-
ate digital health strategies through a clear step-by-step process.
Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021
As defined by the WHO Global Strategy on Digital Health 2020-
2025, digital health is “the field of knowledge and practice associ-
ated with the development and use of digital technologies to im-
prove health.”10 It is “a broad umbrella term encompassing eHealth
(which includes mHealth)” and “expands the concept of eHealth to
include digital consumers, with a wider range of smart and con-
nected devices, such as the Internet of Things, advanced computing,
big data analytics, artificial intelligence (AI) including machine
learning, and robotics.”10
Review objectives
For toolkits to function as a robust and trustworthy KT strategy,
toolkits should (1) be developed with adequate methodological rigor
and (2) transparently report the process of their development, test-
ing, and refinement.
This review focused on knowledge-to-action digital health tool-
kits, studied their development process for both rigor and reporting,
their intended use, and their level of application. We also considered
whether toolkit evaluations provided evidence for their effectiveness
as a KT strategy and thus identified determinants of successful tool-
kits. The review addressed the following questions about toolkits for
implementation and evaluation of digital health that have been prac-
tically used, tested, or evaluated in any way:
1. What are the toolkit’s characteristics?
• What is the purpose of the toolkit?
• Who is it meant to be used by?
• Which operational level is it intended to be applied at?
2. Is the toolkit development process methodologically rigorous and
adequately reported?
• Is the toolkit informed by a literature review or evidence syn-
thesis?
• Is the toolkit based on an underlying conceptual framework?
• Is the toolkit informed by expert consensus?
• Has the toolkit been evaluated?
• Has the toolkit reported the process of its development, test-
ing, evaluation, and refinement?
MATERIALS AND METHODS
This systematic review was conducted from March 2019 to May
2020. Our methods were informed by the PRISMA (Preferred
Reporting Items for Systematic Reviews and Meta-Analyses) guide-
lines, and modified to suit this review’s meta-research objectives.11
Search methods and strategy
We searched several electronic databases for literature published
from 2009 to July 2019, including Global Health, Scopus, Pro-
Quest, Web of Science, and PubMed. Relevant gray literature (eg,
technical reports, dissertations, patents, meeting reports, annual
reports, government publications) was also identified using Google
Advanced Search, and the first 10 pages of relevance-sorted results
Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0
were perused for relevant material. Other highly relevant sources in
the public domain that were specifically searched included: WHO
Institutional Repository for Information Sharing, Digital Impact Al-
liance,12 Asian Development Bank, Asia eHealth Information Net-
work, MEASURE Evaluation and Health Data Collaborative,
Digital Square Global Goods,13 Health Metrics Network, AHRQ,
COCIR, Joint Learning Network, Healthcare Information and Man-
agement Systems Society, U.S. Office of the National Coordinator,
and The Open Group. Gray literature is being identified on an ongo-
ing basis, as part of the WHO Collaborating Centre activities.6
Records were pooled using Covidence to create a single literature
corpus and duplicates removed.14 The search strategy was com-
prised of groups of similar search terms combined into strings using
Boolean operators, for example, (toolkit OR tool) AND (eHealth
OR mhealth OR digital health) AND (evaluation OR assessment).
The search strings are listed in Supplementary Appendix 1.
Screening process
Search results were pooled in EndNote X7 library (Clarivate, Phila-
delphia, PA) for reference management and exported to Covidence
for screening. Duplicates were removed using both Endnote X7 and
Covidence. All records pertaining to the same toolkit and its testing
or evaluation were grouped together. Records were screened for in-
clusion in 2 stages: (1) title and abstract and (2) full text. Using Cov-
idence, each record was screened by 2 reviewers working
independently, choosing to include or exclude records based on se-
lection criteria. During title and abstract screening, records were ex-
cluded or progressed to full-text screening if both reviewers agreed.
During full-text screening, records were excluded or included if both
reviewers agreed. Disagreements between reviewers at either screen-
ing stage were resolved through arbitration and consensus with a
third reviewer.
Selection criteria
We included any toolkit for the implementation and evaluation of
digital health that had been used or field tested or evaluated in prac-
tice. We excluded non-English literature, conference abstracts,
reviews, practice guidelines, and checklists. Conceptual frameworks
are sometimes described as a tool or toolkit because they are used to
guide approaches to implementation and evaluation; we restricted
our selection to the AHRQ instrumental definition of tool and tool-
kit and excluded conceptual frameworks from this review.
Data extraction and quality appraisal
A data extraction template was developed to capture data of rele-
vance to the reviews questions. Two independent reviewers
extracted data from each full text, and all authors reviewed com-
pleted data extraction sheets, and resolved disagreements through
discussion and consensus. As there are currently no critical appraisal
checklists for toolkit quality, during data extraction we captured
any available information regarding toolkit development methods to
serve as an indicator of methodological rigor. Specifically, this in-
cluded the conduct of a literature review, the incorporation of expert
consensus, and the use of toolkit field testing (eg, pilot testing, use,
evaluation). Recognizing that there are currently no reporting crite-
ria for toolkits, we extracted the reporting characteristics of toolkits
that reported their development processes, an approach informed by
our prior work on reporting completeness.15 The details captured by
the data extraction form fields are outlined in Table 1.
3
RESULTS
Of the 1473 records sourced from searches of electronic databases,
gray literature, and perusal of relevant journals, 138 duplicates were
removed and 1335 records were screened at the title and abstract
stage (Figure 1). Of these, 1231 records were excluded and 104
articles proceeded to full-text screening. During full text screening,
88 full-text articles were excluded (reasons for these are outlined in
Figure 1) and 16 full-text articles (13 toolkits) were included for
data extraction.
Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021
The earliest toolkit we included was developed in 2010, with
more being developed each subsequent year (Table 2). All toolkits
reported their intended purpose 7 were for digital health implemen-
tation, 11 for digital health evaluation, and 5 addressed both. While
toolkits for use at the national or sectoral level were developed by
international agencies, the U.S. Agency for International Develop-
ment developed most of the toolkits for health organization use. Al-
most all the toolkits were developed in North America or Europe,
except for one from Australia.
Methods of development
All of the 13 included toolkits reported their development process
(Table 3), all of which involved expert consensus. Eight of 13 tool-
kits reported that their development process involved a literature re-
view, while 3 did not, and a further 2 were unclear. Twelve reported
basing the toolkit on an underlying conceptual framework, theory,
or model, and of them, 3 cited the use of normalization process the-
ory and 3 others cited the WHO-ITU eHealth Strategy; only 1 tool-
kit did not report using an underlying framework. The intended user
groups for each toolkit included government policymakers, imple-
menting partners, donors, system managers, system users, and care
program managers. Because evidence of the toolkit being used or
field tested in practice was required for inclusion of a toolkit, all the
toolkits we included necessarily reported this. Seven toolkits were
reportedly evaluated,16–20,27,28,32–35,41–45 but detailed evaluation
results were only available for 316–20,27,28 and, when available,
addressed usability, relevance, and facilitators and barriers to toolkit
use. The PRISM (Performance of Routine Information System Man-
agement) toolkit is a good example of toolkit development and
reporting, as it is based on a literature review, an underlying frame-
work, and stakeholder consensus,18 and has been evaluated in sev-
eral real-world settings, the findings of which have been
reported.16,17 The extracted data for included toolkits is available in
Supplementary Appendix 2.
DISCUSSION
What we learned
Very few of the potentially relevant toolkits met the inclusion crite-
rion of field testing, let alone evaluation; this could indicate a lack
of field testing or poor reporting of field testing. The description of
the development of the toolkits was often inconsistent and unclear,
possibly reflecting an inconsistent methodology.
The toolkits for implementing and evaluating digital health
broadly fell into 3 categories, based on the organizational level they
applied to: the national health system level, health organization or
facility level (eg, health information system), or user level (eg, indi-
vidual or health professional) (Table 4). These align closely with the
organizational levels identified by the “Framework of e-health for
improved health service delivery,”47 which might offer a common
conceptual framework to enable a multiperspective and interprofes-
4 Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0
Table 1. Data extraction form fields and description
Toolkit
Year
Organization(s)
Summary (rationale and scope)
Purpose: Implementation
Purpose: Evaluation
Target user group
Toolkit development overview
Reported process
Literature review
Conceptual framework
Expert consensus
Field testing
Evaluation
Organizational level
Figure 1. Flow diagram of study selection process. Ab: Abstract; DH: digital health; DHI: digital health implementation; Ti: Title.
sional approach to implementation and evaluation across the levels.
Furthering this line of thinking, more coordinated implementation
of digital health could be achieved by better conceptual synchronic-
ity between toolkits of all levels.
Following the release of the WHO-ITU National eHealth Strat-
egy toolkit, many domain-specific operational and technical toolkits
have emerged and are continuing to emerge. The plethora of largely
untested toolkits we excluded risks “cognitive overload” among
intended users; a limited number of focused, instructive, well-tested,
and clearly reported toolkits would likely be a more effective KT
strategy to guide implementation of digital health interventions and
systems. Our findings indicate that increasingly more toolkits were
made available over time. In light of the burgeoning demand for
health services,48,49 many more such toolkits will likely be devel-
oped as emerging digital technologies (eg, artificial intelligence, ro-
botics, “-omics”) are more widely adopted to support increasingly
integrated models of care.50–53 However, generic frameworks or
guidance for methodologically rigorous toolkit development are few
and far between. The AHRQ offers some guidelines,9 but these are
What is the toolkit called?
In which year was the toolkit published?
Which organization(s) developed the toolkit?
A brief summary of the toolkit’s rationale, purpose, aim/scope, and
tools/components, if described.
Is the toolkit intended for use in implementation?
Is the toolkit intended for use in evaluation?
Who are the target user group, as specified by the toolkit itself?
Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021
Description of the stages and processes involved in toolkit development
Did the toolkit report its development process?
Did the development process include a literature review or evidence syn-
thesis?
Did the development process include a conceptual framework?
Did the development process include expert consensus?
Did the development process include field testing?
Was the toolkit evaluated?
Which organizational level does the toolkit address? (ie, the national
health system level, health facility level, or operational/staff/user
level).
mainly related to reporting and communication quality, rather than
to rigor.
Recommendations for developing and reporting a
digital health toolkit
For toolkits to successfully fulfill their purpose as a tool for KT, it is
imperative that they demonstrate adequate methodological rigor, to
convey “true” knowledge (epistemology) about a given implementa-
tion reality (ontology). This process should also include their evalua-
tion, which ought to test and demonstrate the toolkits’ real-world
applicability and usability. Furthermore, they should be trustwor-
thy, and transparently and completely report their development pro-
cess. Users should understand how a toolkit was developed in order
to meaningfully appraise its relevance and applicability to their con-
text and situation.
Starting by asking, “What makes a good toolkit?,” we build on
this logic, pooling together our study findings, and guidance on
guideline development,54 to propose a preliminary standard ap-
Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0 5

Table 2. Characteristics of included toolkits (listed chronologically)

Year Toolkit, Case Examples, Institution Purpose Summary

and Evaluation

2010-2019 Performance of Routine In- MEASURE Evaluation Implementation and evalu- Conceptual framework and
formation System Man- ation associated data collection
agement (PRISM) and analysis tools to as-
Toolkit16–18 sess, design, strengthen,
and evaluate RHIS

Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021

2011 eHealth Implementation eHealth Unit, University Implementation and evalu- Toolkit to enable senior
toolkit19,20 College London ation staff to analyze chal-
lenges likely to arise
when implementing an e-
health initiative.
Intended to promote crit-
ical thinking, not to re-
place it (ie, not a “tick-
box” tool).
2012 National eHealth Strategy WHO-ITU Implementation Comprehensive practical
Toolkit3,21–24 guidance for develop-
ment of a national
eHealth strategy, action
plan and monitoring
framework.
2015 mHealth Health Assess- WHO Implementation and evalu- Comprehensive self-assess-
ment and Planning for ation ment and planning guide
Scale (MAPS) toolkit25,26 to improve the potential
for scaling up and
achieving long-term sus-
tainability in mHealth
2016 HIS assessment support WHO Regional Office for Evaluation European-specific version
tool27,28 Europe of the National HIS As-
sessment toolkit,29 which
uses the HMN frame-
work to guide HIS evalu-
ation, for the
achievement of HMN
goals.
2017 Evaluating Person-Centred Computer and Information Implementation and evalu- Flexible toolkit used to
Digital Health and Well- Sciences, University of ation evaluate an evolving
ness at Scale30 Strathclyde large scale, national digi-
tal health project.
2017 Informatics Capability Academic GP Unit, UNSW Evaluation Assisted self-assessment
Maturity Toolkit31 Medicine tool to reflect on and
document the informat-
ics capability maturity of
a health facility.
2018 Routine HIS Rapid Assess- MEASURE Evaluation Evaluation Toolkit for assisting health
ment Tool32,33 information system man-
agers identify RHIS gaps
using global standards to
identify where resources
should be invested for
system strengthening.
2018 eHealth Literacy Assess- Department of Public Implementation and evalu- Toolkit for assessing indi-
ment Toolkit34,35 Health, University of Co- ation viduals’ health literacy
penhagen and digital literacy across
7 dimensions, using a
mix of existing and
newly developed scales.
2019 Coordinating Digital Digital Square, PATH Implementation Toolkit for implementing
Transformation practical strategies, en-
Toolkit36–40 abling approaches and
best practices for success-
fully coordinating the
digital health sector.

(continued)
6 Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0

Table 2. continued

Year Toolkit, Case Examples, Institution Purpose Summary

and Evaluation

2019 Global Digital Health Global Digital Health Evaluation Index for tracking, moni-
Index54 Index toring, and evaluating
progress in digital health
technology at the country
level across the 7 compo-
nents of the WHO-ITU

Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021

2019 HIS Interoperability Matu-
rity (HISIM) Toolkit41–43
2019 HIS Stages of Continuous
Improvement (HISSCI)
Toolkit44,45
Health Data Collaborative
and MEASURE Evalua-
tion
Health Data Collaborative
and MEASURE Evalua-
tion
eHealth strategy frame-
work.
Evaluation Self-administered toolkit
designed to monitor,
evaluate, and report on
domains and compo-
nents required for a
country’s digital HIS to
exchange data (intero-
perate) with other health
systems, to inform a plan
for a strong, responsive,
and sustainable national
HIS.
Evaluation Toolkit to help countries or
organizations holistically
assess, plan, and priori-
tize interventions and
investments to strengthen
an HIS.

HIS: health information system; HMN: Health Metrics Network; ITU: International Telecommunication Union; mHealth: mobile health; RHIS: routine health
information system; WHO: World Health Organization.

proach to developing, testing and reporting toolkits for implement-
ing and evaluating digital health interventions (Table 5). Developing
a good toolkit should include robust methodological rigor and com-
plete and transparent reporting. These, in turn, are characterized by
several characteristics (Table 5, left column) that can be operational-
ized through the recommendations provided (Table 5, right col-
umn). These recommendations are not intended as gospel, but
rather as a starting point for discourse among the digital health re-
search and implementation community on this important matter. By
this review’s own critical standards, we recommend that these be
subjected to expert review and consensus, to create guiding criteria
for developing and reporting DHI toolkits. As there are currently no
critical appraisal checklists for toolkit quality, we recommend that
similar principles be used to inform the development of a standard-
ized tool for assessing toolkits’ quality and rigor, with a scoring sys-
tem, for example (Table 5).
included demonstrated use cases, methodological rigor was of con-
cern, as several did not include an underlying conceptual frame-
work, or literature review, or evaluation and refinement in real-
world settings. Reporting of approaches and methods was often in-
consistent and unclear, and toolkits rarely reported being evaluated.
Toolkit development should exhibit greater methodological rigor,
whether in the evidence base (eg, literature review), theoretical
grounding (ie, underlying conceptual framework), participatory ap-
proach (eg, co-creation, consensus processes), or in the testing, eval-
uation, and refinement of the toolkits in real-world settings.
As a vital component of the widespread, global rollout of digi-
tal health, it is imperative that, as a knowledge translation strat-
egy, toolkits fulfill their function efficiently and effectively.
Greater attention needs to be paid to developing, evaluating, and
reporting toolkits to ensure that they effectively perform their
intended function.

Limitations
Our study did not include non-English literature; it would add rele-
FUNDING
vance and validity to this study to include literature published in None.
other languages.

CONCLUSION
The findings of this review raise concerns regarding the potential of
toolkits to effectively facilitate digital health implementation and
evaluation. Despite a plethora of published digital health toolkits,
very few demonstrated their application in real-world testing, and
even fewer had any evidence of having been evaluated. Of those that
AUTHOR CONTRIBUTIONS
MAG led the review and wrote the first draft of the manuscript. SA
coordinated the review and conducted the database searches. All
authors conceptualized the work, contributed to data collection,
participated in screening search results, critically revised the manu-
script, approved the final version to be published; and agree to be
accountable for all aspects of the work. MAG, S-TL and SA partici-
Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0 7

Table 3. Toolkit development approach and methods

Year Toolkit Literature Review Expert Consensus Underlying Frame- Intended User Evaluated
work Group

2010-2019 PRISM Yes Yes PRISM framework Managers, system Yes

Toolkit16–18 evaluators, and
policy makers
2011 eHealth Implemen- Yes Yes Normalization Senior staff and Yes
tation tool- process theory Managers of
kit19,20

Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021

eHealth imple-
mentation
2012 National eHealth Unclear Yes WHO-ITU frame- Policymakers No
Strategy Tool- work
kit3,21–24
2015 MAPS toolkit25,26 Yes Yes MAPS framework Project managers No
and project
teams
2016 HIS assessment No Yes European Health HIS evaluators Yes
support tool27,28 Information Ini-
tiative frame-
work
2017 Evaluating Person- No Yes Normalization Care managers No
Centred Digital process theory
Health and
Wellness at
Scale30
2017 Informatics Capa- Yes Yes Capability matu- Care providers and No
bility Maturity rity models managers of
Toolkit31 health organiza-
tions
2018 Routine HIS Rapid Yes Yes Health Facility and Managers of HIS, Yes
Assessment Community In- programs, and
Tool32,33 formation Sys- data.
tem Standards
2018 eHealth Literacy No Yes The eHealth Liter- eHealth interven- Yes
Assessment acy Framework tion Users
Toolkit34,35
2019 Coordinating Digi- Yes Yes Not reported Governments, No
tal Transforma- implementing
tion partners, donors
Toolkit36–40
2019 Global Digital Unclear Yes WHO-ITU Frame- Policymakers No
Health Index46 work
2019 HIS Interoperabil- Yes Yes Principles of digital Policymakers in Yes
ity Maturity development; low-resource
(HISIM) several maturity settings
Toolkit41–43 models and as-
sessment tools
2019 HIS Stages of Con- Yes Yes National eHealth Policymakers Yes
tinuous Im- Strategy Toolkit
provement (WHO) toolkit,
(HISSCI) Health Metrics
Toolkit44,45 Network Assess-
ment (WHO);
HIS Strengthen-
ing Model
(MEASURE
Evaluation); De-
mand and Read-
iness Tool.

HIS: health information system; ITU: International Telecommunication Union; MAPS: mHealth Health Assessment and Planning for Scale; PRISM: Perfor-
mance of Routine Information System Management; WHO: World Health Organization.
8 Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0

Table 4. Generic classification of toolkits by organizational level

Organizational level Toolkit

Health system level Coordinating Digital Transformation Toolkit36–40

Global Digital Health Index46
National eHealth Strategy Toolkit3,21–23
Health organization/facility level HIS Interoperability Maturity (HISIM) Toolkit41–43
HIS Stages of Continuous Improvement (HISSCI) Toolkit44,45
Routine HIS Rapid Assessment Tool32,33
HIS assessment support tool27,28

Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021

Users (care providers and patients) level
Performance of Routine Information System Management (PRISM)
Toolkit16,17
Informatics Capability Maturity Toolkit31
eHealth Literacy Assessment Toolkit34,35
Evaluating Person-Centred Digital Health and Wellness at Scale30
mHealth Health Assessment and Planning for Scale (MAPS) toolkit25,26
eHealth Implementation toolkit19,20

HIS: health information system.

Table 5. Preliminary considerations in developing toolkits for implementing and evaluating digital health

What Makes a Good Toolkit? Recommendations for Digital Health Intervention Toolkits

Rigor
• Philosophically grounded • Make the philosophical stance, assumptions and motivations explicit.
• Evidence based • Include a literature review.
• Theoretically grounded • Include a conceptual framework, model, or theory.
• Describe the framework building process.
• Participatory methods (co-creating toolkits with users) Include all stakeholders in development process, especially intended user
groups (eg, during needs assessment, establishing expert consensus).
• Use a methodology and study design that can incorporate appropriate
methods (eg, case study or mixed methods approaches).
• Testing and refinement • Ensure to describe the type of testing that the toolkit (eg, usability
testing, pilot testing, expert review, focus group with users).
• Describe the context in which the toolkit was tested, and how gener-
alizable this is to other contexts (eg, realist evaluation).
• Explain how the findings of testing informed the toolkit or frame-
work development process.
• Toolkit synthesis • Describe the research design and logic being used for framework syn-
thesis.
• Outline the resources used in developing the toolkit.
• Clearly detail the case(s) being used to develop and test the toolkit.
• Ensure alignment of objectives between review, expert consensus and
evaluation.
• Describe the approach used for triangulation of findings from differ-
ent sources, and integration (synthesis) of findings into the final
framework.
• Toolkit evaluation • Outline how the toolkit was evaluated (ie, the approach, study de-
sign, program logic, etc.).
• Describe the unintended consequences of the toolkit’s use in the con-
text of complex adaptive systems (ie, both benefits and harms).
• Describe how and why the toolkit was deemed to be “fit” for its
intended purpose.
Reporting
• Underlying logic and reasoning • Report the toolkit’s purpose and motivation.
• Trustworthy • Explicitly report the development process in context.
• Instructional clarity • Clearly describe the step-by-step process of how to use the toolkit.
• Personalized communication • Ensure clarity of communication to all intended users.
• Applicability • Ensure that the toolkit applies to meet needs of the system and/or in-
dividual users.
• Ensure that the toolkit applies to meet needs of organizational users
(ie, clinical, managerial, and technical staff).
• Ensure that the toolkit applies to meet needs of community and/or
citizen users.
Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0
pated in data extraction. S-TL guided the overall direction of the
work.
DATA AVAILABILITY STATEMENT
The data underlying this article are available in the article and in its
online supplementary material.
ACKNOWLEDGMENTS
We thank Dr Jitendra Jonnagaddala and Dr Padmanesan Narasim-
han for their insights toward conceptualizing the review, and Ms
Donna Medeiros for her recommendations on sources of toolkits.
CONFLICT OF INTEREST STATEMENT
This review was conducted to meet the Terms of Reference for the
WHO Collaborating Centre for eHealth (AUS-135).
REFERENCES
1. World Health Organization. Seventy-First World Health Assembly,
Agenda item 12.4, Digital Health. Secondary Seventy-First World Health
Assembly, Agenda item 12.4, Digital Health 2018. https://apps.who.int/
gb/ebwha/pdf_files/WHA71/A71_R7-en.pdf Accessed September 15,
2020.
2. Barac R, Stein S, Bruce B, Barwick M. Scoping review of toolkits as a
knowledge translation strategy in health. BMC Med Inform Decis Mak
2014; 14 (1): 121.
3. World Health Organization and International Telecommunication Union.
National eHealth Strategy Toolkit. 2012. https://apps.who.int/iris/bit-
stream/handle/10665/75211/9789241548465_eng.pdf?
sequence¼1&isAllowed¼y Accessed September 15, 2020.
4. European Coordination Committee of the Radiological, Electromedical,
and Healthcare IT Industry. COCIR eHealth Toolkit. Integrated Care:
Breaking the Silos. 2015. https://www.cocir.org/uploads/media/15013.
COC_2.pdf Accessed September 15, 2020.
5. Global Digital Health Index. Global Digital Health Index Indicator
Guide. https://static1.squarespace.com/static/5ace2d0c5cf-
d792078a05e5f/t/5c1153d1352f53f8337b8dfb/1544639443105/GDHI-
IndicatorþGuide.pdf Accessed September 15, 2020.
6. WHO Collaborating Centre for eHealth. Terms of Reference: AUS 135
WHO Collaborating Centre for eHealth. eHealth 2019. https://sphcm.
med.unsw.edu.au/sites/default/files/sphcm/Centres_and_Units/eHealth_
terms_reference.pdf Accessed March 20, 2019.
7. Yamada J, Shorkey A, Barwick M, Widger K, Stevens BJ. The effectiveness
of toolkits as knowledge translation strategies for integrating evidence
into clinical care: a systematic review. BMJ Open 2015; 5 (4): e006808.
8. Davis MM, Howk S, Spurlock M, McGinnis PB, Cohen DJ, Fagnan LJ. A
qualitative study of clinic and community member perspectives on inter-
vention toolkits: “Unless the toolkit is used it won’t help solve the prob-
lem. BMC Health Serv Res 2017; 17 (1): 497.
9. Agency for Healthcare Research and Quality (AHRQ). AHRQ Publishing
and Communications Guidelines, Section 6: Toolkit Guidance. 2013.
https://www.ahrq.gov/sites/default/files/publications/files/pcguide6.pdf
Accessed September 15, 2020.
10. World Health Organization. Global Strategy for Digital Health 2020-
2024. 2019. https://extranet.who.int/dataform/upload/surveys/183439/
files/Draft%20Global%20Strategy%20on%20Digital%20Health.pdf
Accessed September 15, 2020.
11. Ansari S, Godinho MA, Jonnagaddala J, Narasimhan P, Guo G, Liaw S-T.
A systematic review of toolkits for implementation and evaluation of digi-
tal health interventions. 2019 https://www.crd.york.ac.uk/prospero/dis-
play_record.php?ID¼CRD42019147273 Accessed September 15, 2020.
9
12. Digital Impact Alliance. Principles for Digital Development. 2019. https://
digitalprinciples.org/ Accessed September 15, 2020.
13. Digital Square. Global Goods Guidebook: PATH. 2019. https://
digitalsquare.org/s/Global-Goods-Guidebook_V1.pdf Accessed Septem-
ber 15, 2020.
14. Veritas Health Innovation. Covidence systematic review software. 2019.
www.covidence.org Accessed September 15, 2020.
15. Godinho MA, Gudi N, Milkowska M, Murthy S, Bailey A, Nair NS.
Completeness of reporting in Indian qualitative public health research: a
systematic review of 20 years of literature. J Public Health 2019; 41 (2):
Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021
405–11.
16. Hotchkiss DR, Aqil A, Lippeveld T, Mukooyo E. Evaluation of the perfor-
mance of routine information system management (PRISM) framework:
evidence from Uganda. BMC Health Serv Res 2010; 10 (1): 188.
17. MEASURE Evaluation. PRISM Case Studies: Strengthening and Evaluating
RHIS. Chapel Hill, NC: University of North Carolina at Chapel Hill; 2008.
18. Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for
designing, strengthening and evaluating routine health information sys-
tems. Health Policy Plan 2009; 24 (3): 217–28.
19. MacFarlane A, Clerkin P, Murray E, et al. The e-health implementation
toolkit: qualitative evaluation across four European countries. Implement
Sci 2011; 6 (1): 122.
20. Murray E, May C, Mair F. Development and formative evaluation of the
e-Health Implementation Toolkit (e-HIT). BMC Med Inform Decis Mak
2010; 10 (1): 61.
21. Darcy N, Elias M, Swai A, Danford H, Rulagirwa H, Perera S. eHealth
strategy development: a case study in Tanzania. J Health Inform Africa
2014; 2 (2). doi: 10.12856/JHIA-2014-v2-i2-107
22. Ali S. Formulation of a National e-Health Strategy Development Frame-
work for Pakistan. Graduate Studies [master’s thesis]. Calgary, Alberta,
Canada, University of Calgary; 2013.
23. Riazi H, Jafarpour M, Bitaraf E. Towards National eHealth Implementa-
tion–a comparative study on WHO/ITU National eHealth Strategy Tool-
kit in Iran. Stud Health Technol Inform 2014; 205: 246–50.
24. Hamilton C. The WHO-ITU national eHealth strategy toolkit as an effec-
tive approach to national strategy development and implementation. Stud
Health Technol Inform 2013; 192: 913–6.
25. World Health Organization. The MAPS Toolkit: mHealth Assessment and
Planning for Scale. Geneva, Switzerland: World Health Organization; 2015.
26. Labrique AB, Wadhwani C, Williams KA, et al. Best practices in scaling
digital health in low and middle income countries. Global Health 2018;
14 (1): 103.
27. Verschuuren M, Diallo K, Calleja N, Burazeri G, Stein C. World health or-
ganization. First experiences with a WHO tool for assessing health infor-
mation systems. Public Health Panor 2016; 2 (3): 379–82.
28. World Health Organization Regional Office for Europe. Support Tool to
Assess Health Information Systems and Develop and Strengthen Health
Information Strategies. Copenhagen, Denmark: WHO Regional Office
for Europe; 2015.
29. World Health Organization. Assessing the National Health Information
System: An Assessment Tool. Version 4.00. Geneva, Switzerland: World
Health Organization; 2008.
30. McGee-Lennon M, Bouamrane M-M, Grieve E, et al. A flexible toolkit for
evaluating person-centred digital health and wellness at scale. In: Duffy
VG, Lightner N, eds. Advances in Human Factors and Ergonomics in
Healthcare. New York, NY: Springer; 2017: 105–18.
31. Liaw S-T, Kearns R, Taggart J, et al. The informatics capability maturity
of integrated primary care centres in Australia. Int J Med Inform 2017;
105: 89–97.
32. MEASURE Evaluation. Validating the Effectiveness of a Rapid Assess-
ment Tool for Routine Health Information Systems. Chapel Hill, NC:
MEASURE Evaluation, University of North Carolina., 2018.
33. MEASURE Evaluation. Routine Health Information System Rapid As-
sessment Tool: Implementation Guide. Chapel Hill, NC: MEASURE Eval-
uation, 2018.
34. Karnoe A, Furstrand D, Christensen KB, Norgaard O, Kayser L. Assessing
competencies needed to engage with digital health services: development
10 Journal of the American Medical Informatics Association, 2021, Vol. 00, No. 0

of the eHealth literacy assessment toolkit. J Med Internet Res 2018; 20 1565379490219/StateþofþDigitalþHealthþ2019.pdf Accessed Septem-
(5): e178. ber 15, 2020.
35. Knudsen AK, Kayser L. Validation of the eHealth Literacy Assessment 47. World Health Organization Regional office for South-East Asia. Regional Strat-
tool (eHLA). Int J Integr Care 2016; 16 (6): 349. egy for Strengthening eHealth in the South-East Asia Region WHO (2014-
36. PATH. Coordinating Digital Transformation: Ethiopia. Seattle, WA: Dig- 2020). 2015. https://apps.who.int/iris/bitstream/handle/10665/160760/SEA-
ital Square; 2019. HSD-366%20Rev.pdf?sequence¼1&isAllowed¼y Accessed September 15,
37. PATH. Coordinating Digital Transformation: Nepal. Seattle, WA: Digital 2020.
Square; 2019. 48. Lozano R, Fullman N, Mumford JE, et al. Measuring universal health cov-
38. PATH. Coordinating Digital Transformation: Tanzania. Seattle, WA: erage based on an index of effective coverage of health services in 204
Digital Square; 2019. countries and territories, 1990–2019: a systematic analysis for the Global

Downloaded from https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocab010/6146672 by guest on 23 February 2021

39. PATH. Coordinating Digital Transformation: Replication Guide. Seattle,
WA: Digital Square; 2019.
40. PATH. Coordinating Digital Transformation: Overview. Seattle, WA:
Digital Square; 2019.
41. MEASURE Evaluation. Health Information Systems Interoperability Ma-
turity Toolkit. 2019. https://www.measureevaluation.org/resources/tools/
health-information-systems-interoperability-toolkit Accessed September
15, 2020.
42. MEASURE Evaluation. Building a Strong and Interoperable Digital
Health Information System for Uganda. Chapel Hill, NC: MEASURE
Evaluation; 2018.
43. MEASURE Evaluation. Building a Strong and Interoperable Health Infor-
mation System for Ghana. Chapel Hill, NC: MEASURE Evaluation;
2018.
44. MEASURE Evaluation. HIS Stages of Continuous Improvement Toolkit.
Chapel Hill, NC: MEASURE Evaluation; 2017.
45. MEASURE Evaluation. Mapping a Path to Improve Uganda’s Health In-
formation System Using the Stages of Continuous Improvement Toolkit -
Workshop Report. Chapel Hill, NC: MEASURE Evaluation; 2019.
46. Mechael P, Ke Edelman J. The State of Digital Health 2019: Global Devel-
opment Incubator, 2019. https://static1.squarespace.com/static/
5ace2d0c5cfd792078a05e5f/t/5d4dcb80a9b3640001183a34/
Burden of Disease Study 2019. Lancet 2020; 396 (10258): 1250–84. doi:
https://doi.org/10.1016/S0140-6736(20)30750-9
49. Murray CJL, Abbafati C, Abbas KM, et al. Five insights from the global
burden of disease Study 2019. Lancet 2020; 396 (10258): 1135–59.
50. Marcelo A, Medeiros D, Ramesh K, Roth S, Wyatt P. Transforming
Health Systems Through Good Digital Health Governance. 2018. www.
adb.org/sites/default/files/publication/401976/sdwp-051-transforming-
health-systems.pdf Accessed September 15, 2020.
51. Godinho MA, Borda A, Kostkova P, Molnar A, Liaw S-T. Serious Games’
for unboxing Global Digital Health policymaking. BMJ Simul Technol
Enhanc Learn 2020; 6: 255–6.
52. Godinho MA, Ashraf MM, Narasimhan P, Liaw S-T. Community health
alliances as social enterprises that digitally engage citizens and integrate
services: A case study in Southwestern Sydney (protocol). Digit Health
2020; 6: 205520762093011.
53. Godinho MA, Jonnagaddala J, Gudi N, Islam R, Narasimhan P, Liaw S-
T. mHealth for integrated people-centred health services in the Western
Pacific: a systematic review. Int J Med Inform 2020; 142: 104259.[pub-
lished Online First: Epub Date].
54. Logullo P, MacCarthy A, Kirtley S, Collins GS. Reporting guideline check-
lists are not quality evaluation forms: they are guidance for writing.
Health Sci Rep 2020; 3 (2): e165. doi: 10.1002/hsr2.165.