International Journal for Quality in Health Care, 2020, 32(10), 708–720

doi:10.1093/intqhc/mzaa128
Advance Access Publication Date: 15 October 2020
Systematic Review

Systematic Review

The effect of electronic audits and feedback in

primary care and factors that contribute to their
effectiveness: a systematic review

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

STEVE VAN DEN BULCK1 , DAVID SPITAELS1 , BERT VAES1 ,
GEERT GODERIS1 , ROSELLA HERMENS1,2 and
PATRIK VANKRUNKELSVEN1
1
Academic Center for General Practice, Department of Public Health and Primary Care, KU Leuven, Kapucijnenvoer
33, blok J, 3000, Leuven, Belgium, and 2 Scientific Institute for Quality of Healthcare (IQ Healthcare), Radboud
Institute for Health Science (RIHS), Radboud University Medical Center, Radboud University Nijmegen, PO Box 9101,
Nijmegen, 6500, HB, The Netherlands
Address reprint requests to: Steve Van den Bulck, Academic Center for General Practice, Department of Public Health and
Primary Care, KU Leuven, Kapucijnenvoer 33 Blok J, Leuven, 3000, Belgium, Tel: 32 16 37 90 14; Fax: 32 16 33 74 80;
E-mail: steve.vandenbulck@kuleuven.be
Received 17 April 2020; Editorial Decision 28 September 2020; Revised 21 September 2020; Accepted 6 October 2020

Abstract
Purpose: The aim of this systematic review was (i) to assess whether electronic audit and feed-
back (A&F) is effective in primary care and (ii) to evaluate important features concerning content
and delivery of the feedback in primary care, including the use of benchmarks, the frequency of
feedback, the cognitive load of feedback and the evidence-based aspects of the feedback.
Data sources: The MEDLINE, Embase, CINAHL and CENTRAL databases were searched for articles
published since 2010 by replicating the search strategy used in the last Cochrane review on A&F.
Study selection: Two independent reviewers assessed the records for their eligibility, performed
the data extraction and evaluated the risk of bias. Our search resulted in 8744 records, including the
140 randomized controlled trials (RCTs) from the last Cochrane Review. The full texts of 431 articles
were assessed to determine their eligibility. Finally, 29 articles were included.
Data extraction: Two independent reviewers extracted standard data, data on the effectiveness and
outcomes of the interventions, data on the kind of electronic feedback (static versus interactive) and
data on the aforementioned feedback features.
Results of data synthesis: Twenty-two studies (76%) showed that electronic A&F was effective.
All interventions targeting medication safety, preventive medicine, cholesterol management and
depression showed an effect. Approximately 70% of the included studies used benchmarks and
high-quality evidence in the content of the feedback. In almost half of the studies, the cognitive
load of feedback was not reported. Due to high heterogeneity in the results, no meta-analysis was
performed.
Conclusion: This systematic review included 29 articles examining electronic A&F interventions
in primary care, and 76% of the interventions were effective. Our findings suggest electronic
A&F is effective in primary care for different conditions such as medication safety and preventive
medicine. Some of the benefits of electronic A&F include its scalability and the potential to be cost
effective. The use of benchmarks as comparators and feedback based on high-quality evidence
are widely used and important features of electronic feedback in primary care. However, other

© The Author(s) 2020. Published by Oxford University Press on behalf of International Society for Quality in Health Care. All rights reserved.
For permissions, please e-mail: journals.permissions@oup.com 708
E-A&F: a systematic review • Systematic Review 709

important features such as the cognitive load of feedback and the frequency of feedback pro-
vision are poorly described in the design of many electronic A&F intervention, indicating that
a better description or implementation of these features is needed. Developing a framework or
methodology for automated A&F interventions in primary care could be useful for future research.

Key words: quality of healthcare, clinical audit, medical records systems, computerized, primary health care, systematic review

Introduction Our inclusion and exclusion criteria and our search strategy
were based on the Cochrane review on A&F [3]. Our methods
Audit and feedback (A&F) is a well-known healthcare intervention
adhere to the Preferred Reporting Items for Systematic Reviews and
to improve the quality of care that can be defined as ‘any summary,
Meta-Analyses (PRISMA) guidelines [26] (see Appendix 1).
which was delivered to healthcare providers of their clinical perfor-
mance over a specific period in time’ [1]. A&F has been proven to

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

be effective, but there is no gold standard available for the design
and implementation of an A&F intervention [2, 3]. To better under-
stand the development and effectiveness of future A&F interventions
certain design elements, hypotheses and suggestions for improvement
have been published [4–6]. However, the importance of most of these
in the design of an A&F intervention needs to be investigated [5–7].
In addition to the many studies examining why and when A&F is
more effective, research is being published on creating tools to facil-
itate feedback, especially via an electronic medium [8–11]. With the
evolution in health information technology, electronic A&F based
on data stored in electronic health records (EHRs) offers a promising
evolution in A&F interventions [12–15]. Large data repositories and
EHR-extractable quality indicators are already available and could
be useful for this purpose [16–22].
A previous systematic review investigated the effectiveness of
electronic A&F interventions in primary and secondary care [23].
However, the authors only included interventions where electronic
A&F was provided with the help of ‘interactive computer interfaces’;
such interventions not only give feedback but also allow interaction
with the feedback to assist in the clinical decision process [23]. For
the purpose of our literature review, this definition is too narrow and
excludes less comprehensive and rather static forms of electronically
delivered feedback that might be effective in a primary care setting.
In addition, little is known about electronic A&F in primary care and
more specifically how electronic A&F interventions can be optimized
in primary care. Previous work indicated that feedback is a promis-
ing tool for quality assessment in primary care but that more research
is needed, especially concerning electronic feedback [24]. Moreover,
observational research indicated that general physicians (GPs) prefer
brief feedback interventions and reports with comparisons and best
practice guidelines for quality assessments [25].
The aim of this systematic review was therefore (i) to assess
whether electronic A&F is effective for improving health provider
performance and healthcare outcomes in primary care and (ii) to eval-
uate important features concerning content and delivery of electronic
feedback in primary care that could optimize the design of future
electronic A&F interventions, namely the evidence-based aspect,
the cognitive load, the frequency and the use of comparators, as
proposed in previous research [25].
Methods
Background
The protocol of this systematic review is described in detail in
PROSPERO: https://www.crd.york.ac.uk/prospero/display_record.
php?ID=CRD42018089069.
Inclusion and exclusion criteria
Inclusion criteria: participants, interventions, comparators,
outcomes and design
The included participants were primary healthcare providers who
were responsible for patient care. The interventions studied had to
involve electronic A&F alone or as a core element of a multifaceted
intervention in primary care. Electronic A&F was defined as ‘any
summary, which was delivered electronically, of clinical performance
of healthcare over a specified period of time.’ A broad definition was
used to investigate both static (without the ability to interact with
the feedback) and interactive (with the ability to interact with the
feedback) types of electronic feedback. Electronic A&F (alone or as
a core element of a multifaceted intervention) was compared with
usual care or different types of A&F. The latter comparison is used to
evaluate potential differences between the aspects of the content and
delivery of the feedback, including the evidence-based aspect, the cog-
nitive load, the frequency and the use of benchmarks. The outcome
of interest was the effect of the intervention on improving healthcare
provider performance and healthcare outcomes. Finally, the included
studies were randomized controlled trials (RCTs; see Appendix 2).
Exclusion criteria
Studies in which real-time feedback was provided during procedu-
ral skills were excluded as well as studies that examined feedback on
performance with simulated patient interactions or studies in which
the term feedback would be best classified as ‘facilitated relay’ of
patient-specific clinical information [3]. RCTs that were not con-
ducted in a primary care setting were also excluded. Studies without
full-text availability were excluded. Studies were also excluded if they
lacked clarity as to whether feedback was delivered electronically
(see Appendix 2).
Searches
We searched the following databases: MEDLINE (Ovid) was
searched on 25 November 2018, Embase (Elsevier) was searched on
25 November 2018, CINAHL (EBSCO) was searched on 31 Octo-
ber 2018 for trials published since 2010 until the end of October
2018, and the Cochrane Central Register of Controlled Trials (CEN-
TRAL) was searched on 14 February 2019 for studies published since
2010 until the beginning of February 2019 (searched 14 February
2019). The CENTRAL database was searched again at a later date
due to technical issues. We searched for trials published since 1 Jan-
uary 2010 to ensure that there was some overlap with the results from
the Cochrane review from 2012 and to avoid articles being missed.
The search strings are available in Appendix 3.
710
The 140 RCTs included in the Cochrane review were also added
to our search results [3].
Data collection and synthesis
Selection of studies
After removing duplicates, the titles and abstracts were indepen-
dently screened by two review authors (S.V.D.B. and D.S.) and were
classified as inclusion, doubt or exclusion. The full texts of all arti-
cles that were classified as doubt and inclusion were obtained. Two
review authors, both medical doctors with experience in conduct-
ing A&F interventions (S.V.D.B. and D.S.), independently read all
full manuscripts and reapplied the inclusion criteria. If there was
still no consensus or if doubt remained after reading the full text,
a third review author, the director of Cochrane Belgium (P.V.), was
consulted to give his opinion.
Data extraction and risk of bias assessment
Two independent reviewers (S.V.D.B. and D.S.) extracted the data
and assessed the risk of bias. A data extraction sheet and a list
to assess the risk of bias were tailored based on tools provided by
Cochrane [27–29]. A separate data extraction file was made for
dichotomous and continuous data. Some features of feedback, based
on previous observational research and on hypotheses regarding con-
tent and delivery, were also incorporated on our data extraction sheet
[3–6, 25]. These features were feedback frequency, evidence-based
aspect of the feedback (feedback based on evidence-based guidelines:
yes, no or unclear), the use of benchmarks as comparisons in the
feedback (yes, no or unclear) and the cognitive load of the feedback
(feedback with a low cognitive load: yes, no or unclear). Interventions
with feedback consisting of many graphs and/or text were consid-
ered as having a high cognitive load, while interventions with few
graphs and no unnecessary in-depth elements or text were considered
as having a low cognitive load.
Disagreements were resolved by discussion. If no consensus was
reached, another reviewer (P.V.) was consulted. For each article,
standard data were extracted (see Appendices 4 and 5 for the
data extraction sheets for continuous and dichotomous outcomes,
respectively).
For our risk of bias summary, blinding of participants and per-
sonnel (performance bias) was not considered a key domain since the
nature of an A&F intervention makes blinding difficult. The risk of
performance bias was therefore not used to calculate the summarized
risk of bias of the different studies. However, all of the other forms
of bias were considered key domains, and if one of them had a high
or an unclear risk of bias, the summary was considered as having a
high or an uncertain risk of bias, respectively.
Data synthesis
Because no meta-analysis was carried out, the results were described
narratively. The hypotheses concerning content and delivery of the
feedback were described for each included study, and the type of
electronic delivered feedback (static versus interactive) was evaluated.
Feedback was defined as static if it was not possible to interact with
the electronic feedback and defined as interactive if it was possible to
interact with the electronic feedback. The effect of each intervention
on improving health provider performance and healthcare outcomes
was evaluated. Interventions were considered effective if they had
statistically significant results (P < 0.05).
Van den Bulck et al.
Results
Searches
A total of 12 054 records were identified through database screen-
ing. The 140 articles from the Cochrane review were also included,
which resulted in a list of 8744 records after removing the duplicates
(see Figure 1).
Data collection and synthesis
Selection of studies
In total, 8313 records were excluded, mostly because there was no
(electronic) A&F intervention or because they were not conducted in
a primary care setting. A total of 431 full-text articles were assessed
for eligibility, and 402 articles were excluded (see Figure 1). The
Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022
total number of studies included through database searching was 23
[30–52], and an additional 6 [53–58] articles published before 2010
were included from the Cochrane review published in 2012.
Data extraction and risk of bias assessment
The data extraction indicated that 12 articles (41%) had continu-
ous outcomes, while 17 articles (59%) had dichotomous outcome
measures. There was high heterogeneity in the outcome measures of
the trials, and a wide range of clinical conditions was targeted by the
interventions. Most trials used a cluster RCT design, although 5 stud-
ies (17%) used an RCT design. Most interventions included general
physicians, but there were 2 trials (7%) including dentists and 1 trial
(3.5%) including pharmacists. Patients were often the unit of analysis
(19 studies, 65.5%), but some studies used the providers (7 studies,
24%) or the distribution/prescriptions of medication (2 studies, 7%)
as the unit of analysis. Finally, one study (3.5%) analyzed both data
at the patient and provider levels (see Tables 1 and 2).
There was a high risk of performance bias in 17 of the included
studies (59%), while the risk of selection and detection bias was
minimal. The risk of both attrition and reporting bias was high in
6 different studies (21%). To summarize, 4 studies (14%) had a
low overall risk of bias, 12 studies (41%) had a high risk of bias
and 13 studies (45%) had an unclear risk of bias (See Figure 2 and
Appendix 6).
Data synthesis
Twenty-two studies (76%) showed a statistically significant effect of
the intervention, of which 3 studies (10.5%) only had a partial effect
(on only one of multiple co-primary outcomes). Of the 8 studies with
interactive electronic feedback, 5 showed an effect of the interven-
tion, while 17 studies out of 21 with static electronic feedback were
effective. The data on the different features of electronic feedback
showed 17 studies (58.5%) where feedback was provided more than
once. In 19 studies (65.5%), the feedback was evidence-based and
20 studies used benchmarks as a comparison in the feedback (69%).
Finally, the cognitive load of the feedback was low in 12 studies
(41%).
In addition, all electronic A&F interventions targeting medication
safety, preventive medicine, cholesterol management and depression
were effective. Four studies incorporated all important features of
feedback (evidence-based aspect, the cognitive load, the frequency
and the use of benchmarks), and three of them showed an effect.
Of these Elouafkaoui et al. included all general dentist practices
in Scotland and used an A&F intervention based on routinely col-
lected electronic data. In this particular study, subgroup analyses
indicated that evidence-based feedback with a low cognitive load
E-A&F: a systematic review • Systematic Review
Figure 1 PRISMA flow chart.
showed the greatest impact while the use of benchmarks gave no
significant difference [37]. However, Hayashino et al. used the aver-
age of the 10% best performers as benchmark according to the
Achievable Benchmark of Care method to improve the quality of
care for diabetes [41]. Finally, Gerber et al. nearly halved the inap-
propriate prescribing of broad-spectrum antibiotics to children [45].
(See Tables 1, 2 and 3)
Discussion
Principal findings and comparison with previous work
This systematic review identified 29 articles describing an electronic
A&F intervention in primary care. Overall, 22 studies (76%) showed
an effect of the intervention on different outcome measures. All inter-
ventions targeting medication safety, preventive medicine, choles-
terol management and depression were effective. Eight (27.5%) of
the electronic feedback interventions were interactive, while 21 stud-
ies (72.5%) used static electronic feedback. Approximately 70% of
the included studies used benchmarks and high-quality evidence in
the content of the feedback. There was high heterogeneity in the
results, making a meta-analysis unreliable.
Our findings suggest that electronic A&F is effective for improv-
ing the quality of care for different conditions in primary care such
711
Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022
as medication safety and preventive medicine. For these conditions,
the A&F interventions mostly targeted processes of care instead of
outcomes. This could be a potential explanation for their effective-
ness because process measures are directly actionable while outcome
measures are affected by different factors that are beyond the control
of health providers [22, 48, 59]. Thus, electronic A&F interventions
targeting processes of care may be more effective, as the health pro-
fessionals receiving the feedback also have the ability to implement
improvements. Furthermore, our review indicated some of the ben-
efits of electronic A&F in primary care: (i) A&F can be deployed
(inter)nationally, drastically increasing the number of patients for
whom quality of care can be improved [32, 33, 37], (ii) A&F can
be cost effective [30, 52] and (iii) A&F can target many different
conditions and procedures [32, 46, 47, 50].
However, our review also confirms that there is insufficient
research on implementation to further the field and build on existing
knowledge [60]. Previous work showed that feedback is best when it
is provided more than once, and our findings indicate that this was
only the case in 58.5% of the included studies. However, the use
of benchmarks and evidence-based feedback in approximately 70%
of the electronic feedback interventions shows a promising trend in
the implementation of these 2 hypotheses on the content and deliv-
ery of feedback. After all, observational research has shown that
 712

Table 1 Studies with continuous outcomes

Type of
Study ID (first Study targeted Targeted health
author, Year) design behavior Primary outcomes Type of interventions compared Clinical condition professional

M. Bahrami, CRCT Compliance The proportion of patients whose Group 1 received a copy of SIGN Management of Dentist
2003 with the treatment complied with the Guideline and an opportunity to unerupted and
SIGN guideline attend a postgraduate education impacted third
guideline course. Group 2 received A&F. molar teeth
Group 3 received a CAL package.
Group 4 received A&F and CAL
P. Elouafkaoui, CRCT Prescribing Total number of antibiotic items Control (no A&F) versus (i) indi- Infections Dentist
2016 antibiotics dispensed per 100 NHS treatment vidualized graphical A&F or (ii)
claims individualized graphical A&F plus
a written behavior-change message
J. S. Gerber, CRCT Prescribing (i) Change in broad-spectrum antibi- One 1-hour on-site clinician edu- Acute respiratory GP
2013 antibiotics otic prescribing and (ii) change cation session followed by 1 year tract infections
in antibiotic prescribing for viral of personalized, quarterly A&F of
infections prescribing for bacterial and viral
acute respiratory tract infections
versus usual practice
Y. Hayashino, CRCT Improving The rate of patient drop-out. Usual care versus a disease manage- Diabetes care GP
2015 the techni- Despite the process of diabetes ment system of monitoring and
cal quality care was evaluated providing feedback on the qual-
of dia- ity of diabetes care, reminders
betes for regular visits and lifestyle
care modifications
L. G. RCT Prescribing Prescribed DDD of antibiotics to Quarterly updated personalized Infections GP
Hemkens, antibiotics any patient per 100 consultations prescription feedback versus physi-
2017 cians in the control group who
received no material
T.A. Holt, CRCT Prescribing Proportion of patients eligible for Feedback (reminders) versus usual Atrial fibrillation GP
2017 OACs OACs who were prescribed an care
OAC at the end of the intervention
period
Van den Bulck et al.

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

Table 1 (Continued)

Type of
Study ID (first Study targeted Targeted health
author, Year) design behavior Primary outcomes Type of interventions compared Clinical condition professional

N. H. McAl- CRCT Hypertension Percentage of patients with a dias- Usual care versus computer- Hypertension GP
ister, manage- tolic pressure of 90 mm Hg or less generated feedback
1986 ment at last visit
D. R. Murphy, CRCT Reduce Time to documented follow-up for Electronic triggers applied on Colon, prostate and Physicians, physi-
2015 delays in colon cancer, prostate cancer and EHR data repositories versus no lung cancer cians assistants,
diagnostic lung cancer electronic triggers nurse practitioners
evaluation
for three
types of
cancer
M. S. Patel, CRCT Prescribing The change in the percentage New statin prescription rates for Hyperlipidemia Primary care
2018 statins of patients eligible for statin usual care with primary care providers
prescription providers receiving an active
choice intervention with and with-
E-A&F: a systematic review • Systematic Review

out peer comparison performance

feedback
Sarafi, A. RCT Prescribing The monthly total sum of drug Providing performance feedback by Prescribing parental GP
Nejad, 2016 behavior usage calculated as the sum of traditional postal letters or mobile steroids
of PSs DDD for PSs short text messages compared to
usual care
L. P. Svetkey, CRCT Hypertension The change in systolic blood Physician intervention versus control Hypertension Internal medicine or
2009 manage- pressure from baseline to 6 months and/or patient intervention ver- family practice and
ment sus control. Physician intervention patients
included internet-based training,
self-monitoring, and quarterly
feedback reports. Patient inter-
vention included 20 weekly group
sessions followed by 12-monthly
telephone counseling contacts
J. Trietsch, CRCT Test order- The volumes of tests ordered and Groups in both trial arms were Anemia, dyslipi- GP
2017 ing and drugs prescribed per practice, per exposed to the same intervention, demia, prostate
pre- 1000 patients, per 6 months but on different clinical topics. and rheumatic
scribing Each LQIC in one arm served complaints, UTI,
behavior as an unmatched control for the chlamydia, dia-
in primary LQICs in the other arm. Core ele- betes type II,
care ments of the intervention are audit stomach and
and comparative feedback on test perimenopausal
ordering and prescribing volumes, complaints, thyroid
dissemination of guidelines and dysfunction
peer review in quality improve-
ment collaboratives moderated by
local opinion leaders

CAL, computer-aided learning; CRCT, cluster randomized controlled trial; DDD, defined daily dose; LQIC, local quality improvement collaboratives; NHS, National Health Service; OAC, oral anticoagulant; PS, parenteral
713

steroid; SIGN, Scottish Intercollegiate Guidelines Network; UTI, urinary tract infection.

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

Table 2 Studies with dichotomous outcomes
714

Type of Targeted
Study ID (first targeted Clinical health
author, year) Study design behavior Primary outcomes Type of interventions compared condition professional

O. P. Almeida, CRCT The care for A composite measure of clinically The intervention consisted of a Depression GP
2012 patients with significant depression (Patient practice audit with personal-
depression Health Questionnaire score ≥ 10) ized automated audit feedback,
and self-harm or self-harm behavior (suicide printed educational material, and
behavior thoughts or attempt during the 6-monthly educational newslet-
previous 12 months) ters.Control physicians completed
a practice audit without individ-
ualized feedback and 6-monthly
newsletters describing the progress
of the study
A. J. Avery, CRCT Medication The proportions of patients at 6 Computer-generated simple Medication GP
2012 errors months after the intervention who feedback (control) versus a safety
had had any of three clinically pharmacist-led information tech-
important errors nology intervention, composed of
feedback, educational outreach,
and dedicated support
B. Bonevski, RCT Preventive Smoking and benzodiazepine use Intervention received a comput- Preventive GP
1999 medicine sensitivity, specificity, and over- erized feedback system; control medicine
all accuracy and whether blood group was given usual care
pressure and cholesterol screening
levels were obtained
C. A. Estrada, CRCT Improving ‘Acceptable control’ and ‘optimal A multi-component intervention Diabetes Primary care
2011 diabetes control’ of diabetes including Web-based CME, per- physicians
control formance feedback and quality
improvement tools versus usual
care
T. L. Guldberg, CRCT Quality of type Prescriptions for type 2 dia- To receive or not to receive elec- Type 2 GP
2011 2 diabetes betes, measuring of HbA1c tronic feedback on quality of diabetes
care and cholesterol and visits to care
ophthalmologists
B. Guthrie, CRCT Safety of Proportion of patients included in Three arms: ‘usual care’ (= emailed Safety of GP
2016 prescribing one or more of the defined six educational material with sup- prescribing
individual secondary outcomes port); usual care plus feedback on
(denominator) who receive any practice’s high risk prescribing;
high risk prescription (numerator) usual care plus the same feedback
incorporating a behavioral change
component
(i) Full feedback intervention,
W. Y. Lim, CRCT Prescribing The percentage of prescriptions with Errors in Primary care
(ii) partial feedback intervention or
2018 medication at least one error (error versus no prescribing prescribers
(iii) usual care as control
error)
J. A. Linder, CRCT Prescribing The primary outcome was the The ARI Quality Dashboard, an Acute res- Primary care
2010 antibiotic intent-to-intervene antibiotic pre- EHR-based feedback system versus piratory physicians
scribing rate for acute respiratory usual care infections
infection visits
Van den Bulck et al.

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

Table 2 (Continued)

Type of Targeted
Study ID (first targeted Clinical health
author, year) Study design behavior Primary outcomes Type of interventions compared condition professional

J. W. Mold, RCT Preventive The number of practices implement- Intervention practices received per- Preventive Clinicians
2008 medicine ing one or more evidence-based formance feedback, academic medicine
processes and the total number of detailing, a practice facilitator,
processes implemented and computer support to feedback
and benchmarking (= control)
G. Ogedegbe, CRCT Blood Pressure The rate of BP control at 12 months, Intervention patients received edu- Hypertension GP
2014 control defined as mean BP < 140/90 mm cation, home BP monitoring and
Hg (or mean BP < 130/80 mm Hg lifestyle counseling. Intervention
for those with diabetes mellitus or physicians attended hypertension
kidney disease) case rounds and received feedback
on their patients’ home BP read-
E-A&F: a systematic review • Systematic Review

ings and chart audits. Patients and

physicians at the usual care sites
received patient education mate-
rial and hypertension treatment
guidelines, respectively
S. Ornstein, CRCT CRC screening Proportion of active patients up to A quality improvement interven- Colorectal Primary care
2010 date with CRC screening and hav- tion combining EHR based A&F, cancer physicians
ing screening recommended within academic detailing and participa-
past year among those not up to tory planning, and ‘best-practice’
date dissemination on CRC screening
versus usual care
G. A. Pape, CRCT Cholesterol Proportion of participants in each The intervention included remote Cholesterol Family practice
2011 management arm achieving a target LDL level physician–pharmacist team-based management and inter-
in diabetes of 100 mg/dL or lower care focused on cholesterol man- in diabetes nal medicine
mellitus agement in DM versus control. All mellitus physicians
clinicians in the study had access
to automated DM-related point-
of-care prompts, a Web-based
registry, and performance feedback
with benchmarking
(i) The proportion of patients who
D. Peiris, 2015 CRCT Cardiovascular The intervention arm consisted of a Cardiovascular GP
received appropriate screening
disease risk computer-guided QI intervention disease risk
of CVD risk factors.
management comprising point-of care electronic management
(ii) The proportion of patients
decision support, A&F tools, and
defined as being at high CVD
clinical workforce training versus
risk, receiving recommended
usual care
medication prescriptions.
715

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

 716

Table 2 (Continued)

Type of Targeted
Study ID (first targeted Clinical health
author, year) Study design behavior Primary outcomes Type of interventions compared condition professional

I. Urbiztondo, CRCT Prescribing The change in the proportion of Intervention (evidence-based online respiratory GP
2017 antibiotics patients treated with antibiotics for feedback) versus control (no expo- tract
respiratory tract infection sure to the evidence-based online infections
feedback)
D. Vinereanu, CRCT Use of oral The change in the proportion Intervention consisting of two com- Atrial Health care
2017 anticoagulant of patients treated with oral ponents (education and regular fibrillation providers
medication anticoagulants monitoring and feedback) versus
in atrial usual care
fibrillation
W. C. Wad- CRCT Smoking Changes in clinician referrals in both Comparing the impact of 6 Smoking Clinicians
land, cessation intervention and control groups quarterly feedback reports (inter- cessation
2007 vention) with that of general
reminders (control)
N. Winslade, RCT Provision of The number of hypertension/asthma Pharmacy-specific performance feed- Asthma and Pharmacist
2016 professional services billed per pharmacy and back reports versus no feedback hypertension
services and percentage of dispensing to nonad- reports
the quality herent patients over the 12 months
of patients’ postintervention
medication
use

ACE-I, angiotensin-converting enzyme inhibitor; ARI, acute respiratory infection; BP, blood pressure; CME, continuing medical education; CRC, colorectal cancer; CRCT, cluster randomized controlled trial; CVD,
cardiovascular disease; DM, diabetes mellitus; LDL, low-density lipoprotein; NSAID, nonsteroidal anti-inflammatory drug.
Van den Bulck et al.

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

E-A&F: a systematic review • Systematic Review
Figure 2 Risk of bias summary.
GPs prefer brief feedback interventions and reports with compar-
isons and best practice guidelines [25]. In addition, according to
717
the recent Clinical Performance Feedback Intervention Theory, the
use of benchmarks can improve feedback by comparing and moti-
vating feedback recipients [7]. Finally, only 41% of the included
studies used feedback with a low cognitive load, making this hypoth-
esis the least common to be implemented among the 4 hypotheses
investigated herein regarding the content and delivery of feedback.
Although electronic A&F was studied extensively in primary care,
a meta-analysis to pool the results and produce generalizable data
was not feasible. This emphasizes the difficulties in designing com-
plex healthcare interventions and indicates the need for a framework
and a well-defined research agenda when developing electronic A&F
trials so that interventions can be reproduced and compared [60–62].
Designing a methodology for developing generalizable automated
A&F interventions in primary care could be useful since automated
Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022
quality assessment based on EHRs offers promising prospects if the
challenges are answered [15].
Large data repositories, such as those of the Dutch NIVEL, the
British Royal College of General Practitioners (RGCP) Research
and Surveillance Centre (RSC) network, and the Belgian INTEGO
database, have already been available for many years in primary care
[17, 63, 64]. Using the facilities of these institutes in a well-designed
trial with a standardized methodology could address some of the
problems in evaluating the effectiveness and features of electronic
A&F interventions. In this respect, recent research indicates the need
for an evolution from a two-arm trial of A&F versus control to head-
to-head trials of various A&F variants to measure small differences
in the effectiveness of different A&F features [65]. Such trials need
to be sufficiently powered, requiring large sample sizes that could be
provided by these large primary care data repositories [65]. However,
further research is necessary to develop a methodology for automated
and EHR-based A&F interventions in primary care. Designing and
using a standardized methodology to create automated A&F inter-
ventions based on EHR data could enable comparisons of future
electronic A&F interventions and enable investigations of different
features of interventions, thus advancing the field of A&F research.
Strengths and limitations
To our knowledge, this is the first systematic review that investigated
electronic A&F only in primary care. One of the strengths of this
review is that our search was identical to that of the last Cochrane
review. By replicating the search strings of the Cochrane review and
screening abstracts and full texts based on our inclusion and exclu-
sion criteria, this review had a solid basis. Furthermore, a broad
definition of electronic feedback was used, thus reducing the risk of
missing relevant articles.
Our review also has several limitations. Because our results
showed high heterogeneity, meta-analysis was not feasible, and no
generalizable data could be produced. Furthermore, for the calcula-
tion of our risk of bias summary, every form of bias was considered
as a key domain, except for performance bias, which may have
produced too severe an overall risk of bias evaluation.
Conclusion
This systematic review included 29 articles that examined electronic
A&F interventions in primary care, and 76% of the interventions
were found to be effective. Approximately 75% of the studies pro-
vided electronic feedback without the ability to interact with it.
Despite that the design of the electronic A&F interventions varied
widely, approximately 70% of the included studies used benchmarks
 718

Table 3 Feedback features

Effect of
First author Year Unit of analysis Frequency of the feedback Evidence-based feedback Benchmarks Low cognitive load Electronic feedback intervention

1 O. P. Almeida 2012 Patients Unclear Unclear Yes Unclear Static Yes

2 A. J. Avery 2012 Patients Less than monthly Unclear Unclear Unclear Static Yes
3 B. Bonevski 1999 Patients Unclear Yes Yes Unclear Interactive Yes
4 C. A. Estrada 2011 Patients Less than monthly Unclear Yes Unclear Interactive No
5 T. L. Guldberg 2011 Patients Unclear Yes Yes Yes Interactive Yes
6 B. Guthrie 2016 Patients Less than monthly Yes Yes No Static Yes
7 W. Y. Lim 2018 Number of prescriptions Less than monthly Unclear Yes Yes Static Yes
8 J. A. Linder 2010 Providers Monthly Yes Yes Yes Interactive No
9 J. W. Mold 2008 Providers Unclear Unclear Yes Unclear Static Yes
10 G. Ogedegbe 2014 Patient Less than monthly Yes Unclear Unclear Static No
11 S. Ornstein 2010 Patients Unclear No Unclear Unclear Static Yes
12 G. A. Pape 2011 Patients Unclear Yes Yes Unclear Interactive Yes
13 D. Peiris 2015 Patient-level data analysis Unclear Yes Yes Yes Interactive Yes (partially)
14 I. Urbiztondo 2017 Individual data at patient and GP level Weekly Yes Unclear Unclear Static Yes
15 D. Vinereanu 2017 Patients Unclear Unclear Unclear Unclear Static Yes
16 W. C. Wadland 2007 Providers Less than monthly Unclear Yes Yes Static Yes
17 N. Winslade 2016 Number of dispensings Once Yes Yes No Static Yes (partially)
18 M. Bahrami 2003 Patients Less than monthly Yes Unclear Unclear Static No
19 P. Elouafkaoui, 2016 Providers Less than monthly Yes Yes Yes Static Yes
20 J. S. Gerber 2013 Patients Less than monthly Yes Yes Yes Static Yes
21 Y. Hayashino 2015 Patients Monthly Yes Yes Yes Static Yes
22 L. G. Hemkens 2017 Providers Less than monthly Yes Yes No Static No
23 T.A. Holt 2017 Patients Unclear Yes No Yes Interactive No
24 N. H. McAlister 1986 Patients Unclear Yes Yes Yes Static Yes
25 D. R. Murphy 2015 Patients Monthly Unclear No Unclear Static Yes
26 M. S. Patel 2018 Patients Monthly Yes Yes Unclear Interactive Yes
27 S. A. Nejad 2016 Providers Less than monthly Unclear Unclear Yes Static Yes
28 L. P. Svetkey 2009 Patients Less than monthly Yes Yes Unclear Static Yes (partially)
29 J. Trietsch 2017 Providers Unclear Yes Yes Yes Static No
Van den Bulck et al.

Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022

E-A&F: a systematic review • Systematic Review
and high-quality evidence in the content and delivery of the feed-
back. In almost half of the studies, the cognitive load of feedback
was not reported. Our findings suggest electronic A&F is effective in
primary care for different conditions such as medication safety and
preventive medicine. Some of the benefits of electronic A&F include
its scalability and the potential to be cost effective. The use of bench-
marks as comparators and feedback based on high-quality evidence
are widely used features of electronic feedback in primary care. How-
ever, other important features such as the cognitive load of feedback
and the frequency of feedback provision are poorly described in the
design of many electronic A&F intervention, indicating that a better
description or implementation of these features is needed. Develop-
ing a framework or methodology for automated A&F interventions
in primary care could be useful for future research.
Acknowledgements
S.V.D.B., B.V., G.G., R.H. and P.V. contributed to the design and conceptual-
ization of the study.
S.V.D.B. performed the search.
S.V.D.B., D.S. and P.V. performed the screening, data extraction and risk of
bias assessment.
S.V.D.B., D.S., B.V., G.G., R.H. and P.V. reviewed and edited the manuscript.
The authors would like to thank Dr Anne-Catherine Vanhove for her assistance
with the search and Jacques Adriaansen for his assistance with the tables.
Funding
There was no funding for this systematic review.
Ethics and other permissions
Ethical approval was not required.
Data availability statement
The data extraction sheets containing all the articles in this review
are available as supplemental material. No new data were generated
in support of this review.
Appendices
Appendix 1: PRISMA checklist
Appendix 2: Inclusion and exclusion criteria
Appendix 3: Search strings and results
Appendix 4: Data extraction sheet continuous outcomes
Appendix 5: Data extraction sheet dichotomous outcomes
Appendix 6: Risk of bias evaluation
References
1. Jamtvedt G, Young JM, Kristoffersen DT et al. Audit and feedback: effects
on professional practice and health care outcomes. Cochrane Database Syst
Rev 2006; (2: CD000259. PMID: 12917891.
2. Flottorp SA, Jamtvedt G, Gibis B et al. Using audit and feedback to health
professionals to improve the quality and safety of health care. World
Health Organization, on behalf of the European Observatory on Health
Systems and Policies 2010.
3. Ivers N, Jamtvedt G, Flottorp S et al. Audit and feedback: effects on
professional practice and healthcare outcomes. The Cochrane Databa of
Systematic Rev 2012; 6: Cd000259.
4. Colquhoun H, Michie S, Sales A et al. Reporting and design elements
of audit and feedback interventions: a secondary review. BMJ Qual
Saf 2017 Jan; 26(1): 54–60
719
5. Brehaut JC, Colquhoun HL, Eva KW et al. Practice feedback interventions:
15 suggestions for optimizing effectiveness. Ann Intern Med 2016; 164:
435–41.
6. Colquhoun HL, Carroll K, Eva KW et al. Advancing the literature on
designing audit and feedback interventions: identifying theory-informed
hypotheses. Implementation Science 2017; 12: 117.
7. Brown B, Gude WT, Blakeman T et al. Clinical performance feedback
intervention theory (CP-FIT): a new theory for designing, implementing,
and evaluating feedback in health care based on a systematic review and
meta-synthesis of qualitative research. Implementation Science: IS 2019;
14: 40.
8. Mould DR, Upton RN, Wojciechowski J Dashboard systems: implement-
ing pharmacometrics from bench to bedside. Aaps J 2014; 16: 925–37.
9. Waitman LR, Phillips IE, McCoy AB et al. Adopting real-time surveil-
lance dashboards as a component of an enterprisewide medication safety
Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022
strategy. Jt Comm J Qual Patient Saf 2011; 37: 326–32.
10. Khairat SS, Dukkipati A, Lauria HA et al. The impact of visualization dash-
boards on quality of care and clinician satisfaction: integrative literature
review. JMIR Hum Factors 2018; 5: e22.
11. Karami M, Langarizadeh M, Fatehi M. Evaluation of effective dashboards:
key concepts and criteria. Open Med Inform J 2017; 11: 52–57.
12. Patel S, Rajkomar A, Harrison JD et al. Next-generation audit and feed-
back for inpatient quality improvement using electronic health record
data: A cluster randomised controlled trial. BMJ Qual Saf 2018; 27:
691–99.
13. Gulliford MC, Prevost AT, Charlton J et al. Effectiveness and safety
of electronically delivered prescribing feedback and decision support
on antibiotic use for respiratory illness in primary care: REDUCE
cluster randomised trial. BMJ (Clinical Research Ed) 2019; 364:
l236.
14. Hsu Y-HE, Yang Y-T. Solve the problems for better health—from quality
indicators to system audits. Int J Quality Heal Care 2018; 30: 577.
15. Roth CP, Lim YW, Pevnick JM et al. The challenge of measuring quality
of care from the electronic health record. Am J Med Quality: The Official
J Am College of Med Quality 2009; 24: 385–94.
16. Bartholomeeusen S, Kim C-Y, Mertens R et al. The denominator in general
practice, a new approach from the Intego database. Fam Pract 2005; 22:
442–47.
17. Schweikardt C, Verheij RA, Donker GA et al. The historical development
of the Dutch Sentinel General Practice Network from a paper-based into a
digital primary care monitoring system. J Public Health (Bangkok) 2016;
24: 545–62.
18. UK: https://www.cprd.com/home/. accessed [August 18 2019].
19. Md. Emdadul Hoque D, Ruseckaite R, Lorgelly P et al. Cross-sectional
study of characteristics of clinical registries in Australia: a resource for
clinicians and policy makers. Int J Quality Heal Care 2018; 30: 192–99.
20. Smets M, Smeets M, Van den Bulck S et al. Defining quality indicators
for heart failure in general practice. Acta Cardiol 2019; 74: 291–298,
10.1080/00015385.2018.1492659
21. Van den Bulck SA, Vankrunkelsven P, Goderis G et al. Development of
quality indicators for type 2 diabetes, extractable from the electronic health
record of the general physician. A rand-modified Delphi method. Prim
Care Diabetes 2020 Feb; 14(1): 75–84.
22. Van den Bulck SA, Vankrunkelsven P, Goderis G et al. Developing quality
indicators for chronic kidney disease in primary care, extractable from the
electronic medical record. A rand-modified Delphi method. BMC Nephrol
2020; 21: 161.
23. Tuti T, Nzinga J, Njoroge M et al. A systematic review of electronic
audit and feedback: intervention effectiveness and use of behaviour change
theory. Implementation Science 2017; 12: 61.
24. Guldberg TL, Lauritzen T, Kristensen JK et al. The effect of feedback to
general practitioners on quality of care for people with type 2 diabetes. A
systematic review of the literature. BMC Fam Pract 2009; 10: 30.
25. Sebo P, Maisonneuve H, Fournier JP et al. General practitioners’ views
and preferences about quality improvement feedback in preventive care: a
cross-sectional study in Switzerland and France. Implementation Science
2017; 12: 95.
720
26. Moher D, Liberati A, Tetzlaff J et al. Preferred reporting items for system-
atic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;
6: e1000097.
27. Higgins JPT, Altman DG, Gøtzsche PC et al. The Cochrane Collaboration’s
tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.
28. EPOC C. Cochrane Effective Practice and Organisation of Care Review
Group. Data Collection Checklist. 2002.
29. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic
Reviews of Interventions. Chichester (UK): John Wiley & Sons, 2011.
30. Patel MS, Kurtzman GW, Kannan S et al. Effect of an automated patient
dashboard using active choice and peer comparison performance feedback
to physicians on statin prescribing: the PRESCRIBE cluster randomized
clinical trial. JAMA Network Open 2018; 1: e180818.
31. Lim WY, Hss AS, Ng LM et al. The impact of a prescription review and
prescriber feedback system on prescribing practices in primary care clinics:
a cluster randomised trial. BMC Fam Pract 2018; 19: N.PAG-N.PAG.
32. Vinereanu D, Lopes Renato D, Bahit MC et al. A multifaceted interven-
tion to improve treatment with oral anticoagulants in atrial fibrillation
(IMPACT-AF): an international, cluster-randomised trial. Lancet (London,
England) 2017; 390North American Edition:1737-46.
33. Urbiztondo I, Bjerrum L, Caballero L et al. Decreasing inappropriate use
of antibiotics in primary care in four countries in South America—cluster
randomized controlled trial. Antibiotics 2017; 6: 4.
34. Trietsch J, van Steenkiste B, Grol R et al. Effect of audit and feedback
with peer review on general practitioners’ prescribing and test ordering
performance: a cluster-randomized controlled trial. BMC Fam Pract 2017;
18: 53.
35. Holt TA, Dalton A, Marshall T et al. Automated software system to
promote anticoagulation and reduce stroke risk: cluster-randomized con-
trolled trial. Stroke 2017; 48: 787–90.
36. Hemkens LG, Saccilotto R, Reyes SL et al. Personalized prescription feed-
back using routinely collected data to reduce antibiotic use in primary care
a randomized clinical trial. JAMA Intern Med 2017; 177: 176–83.
37. Elouafkaoui P, Young L, Newlands R et al. An audit and feedback inter-
vention for reducing antibiotic prescribing in general dental practice:
the RAPiD cluster randomised controlled trial. PLoS Med 2017; 13(8):
e1002115.
38. Winslade N, Eguale T, Tamblyn R. Optimising the changing role of the
community pharmacist: a randomised trial of the impact of audit and
feedback. BMJ Open 2016; 6: 5.
39. Sarafi NA, Farrokhi NM, Haghdoost A et al. The effect of registry-based
performance feedback via short text messages and traditional postal letters
on prescribing parenteral steroids by general practitioners-A randomized
controlled trial. In: Int J Med Informatics 2016; 87: 36–43.
40. Murphy D, Wu L, Thomas E et al. Electronic trigger-based intervention
to reduce delays in diagnostic evaluation for cancer: a cluster randomized
controlled trial. J Clinical Oncol 2015; 33(31): 3560–3567.
41. Hayashino Y, Suzuki H, Yamazaki K et al. A cluster randomized trial on
the effect of a multifaceted intervention improved the technical quality of
diabetes care by primary care physicians: the Japan Diabetes Outcome
Intervention Trial-2 (J-DOIT2). Diabetic Med 2016; 33: 599–608.
42. Guthrie B, Kavanagh K, Robertson C et al. Data feedback and behavioural
change intervention to improve primary care prescribing safety (EFIPPS):
multicentre, three arm, cluster randomised controlled trial. BMJ (Online)
2016 Aug 18; 354; i4079.
43. Peiris D, Usherwood T, Panaretto K et al. Effect of a computer-guided,
quality improvement program for cardiovascular disease risk management
in primary health care: the treatment of cardiovascular risk using electronic
decision support cluster-randomized trial. Circulation Cardiova Quality
Outcomes 2015 Jan; 8(1): 87–95.
44. Ogedegbe G, Tobin JN, Fernandez S et al. Counseling African Americans
to Control Hypertension: cluster-randomized clinical trial main effects.
Circulation 2014; 129: 2044–51.
45. Gerber JS, Prasad PA, Fiks AG et al. Effect of an outpatient antimicro-
bial stewardship intervention on broad-spectrum antibiotic prescribing
by primary care pediatricians: a randomized trial. JAMA 2013; 309:
2345–52.
Van den Bulck et al.
46. Almeida O, Pirkis J, Kerse N et al. A randomized trial to reduce the preva-
lence of depression and self-harm behavior in older primary care patients.
Ann Fam Med 2012 Jul-Aug; 10(4): 347–56.
47. Pape G, Hunt J, Butler K et al. Team-based care approach to cholesterol
management in diabetes mellitus: two-year cluster randomized controlled
trial. Arch Intern Med 2011 Sep 12; 171(16): 1480–6.
48. Guldberg T, Vedsted P, Kristensen J et al. Improved quality of Type 2
diabetes care following electronic feedback of treatment status to general
practitioners: a cluster randomized controlled trial. Diabetic Med 2011;
28: 325–332.
49. Estrada CA, Safford MM, Salanitro AH et al. A web-based diabetes inter-
vention for physician: a cluster-randomized effectiveness trial. Int J Quality
Heal Care 2011; 23: 682–89.
50. Ornstein S, Nemeth LS, Jenkins RG et al. Colorectal cancer screening
in primary care: translating research into practice. Med Care 2010; 48:
Downloaded from https://academic.oup.com/intqhc/article/32/10/708/5924355 by guest on 23 August 2022
900–06.
51. Linder J, Schnipper J, Tsurikova R et al. Electronic health record feedback
to improve antibiotic prescribing for acute respiratory infections. Am J
Manag Care 2010 Dec; 16(12 Suppl HIT): e311–9.
52. Avery AJ, Rodgers S, Cantrill JA et al. A pharmacist-led information tech-
nology intervention for medication errors (PINCER): a multicentre, cluster
randomised, controlled trial and cost-effectiveness analysis. The Lancet
2012; 379: 1310–19.
53. Svetkey LP, Pollak KI, Yancy WS Jr. et al. Hypertension improvement
project: randomized trial of quality improvement for physicians and
lifestyle modification for patients. Hypertension 2009; 54: 1226–33.
54. Mold JW, Aspy CA, Nagykaldi Z. Implementation of evidence-based
preventive services delivery processes in primary care: an Oklahoma physi-
cians resource/research network (OKPRN) study. J Am Board Fam Med
2008; 21: 334–44.
55. Wadland WC, Holtrop JS, Weismantel D et al. Practice-based referrals to a
tobacco cessation quit line: assessing the impact of comparative feedback
vs general reminders. Ann Fam Med 2007; 5: 135–42.
56. Bahrami M, Deery C, Clarkson JE et al. Effectiveness of strategies to
disseminate and implement clinical guidelines for the management of
impacted and unerupted third molars in primary dental care, a cluster
randomised controlled trial. Br Dent J 2004; 197: 691–96. discussion 88
57. Bonevski B, Sanson-Fisher RW, Campbell E et al. Randomized controlled
trial of a computer strategy to increase general practitioner preventive care.
Prev Med 1999; 29: 478–86.
58. McAlister NH, Covvey HD, Tong C et al. Randomised controlled trial of
computer assisted management of hypertension in primary care. British
Med J (Clinical Research Ed) 1986; 293: 670–74.
59. Bilimoria KY. Facilitating quality improvement: pushing the pendulum
back toward process measures. JAMA 2015; 314: 1333–34.
60. Ivers NM, Sales A, Colquhoun H et al. No more ‘business as usual’ with
audit and feedback interventions: towards an agenda for a reinvigorated
intervention. Implementation Science: IS 2014; 9: 14.
61. Campbell M, Fitzpatrick R, Haines A et al. Framework for design and eval-
uation of complex interventions to improve health. BMJ (Clinical Research
Ed) 2000; 321: 694.
62. Hibbert P, Saeed F, Taylor N et al. Can benchmarking Australian hospitals
for quality identify and improve high and low performers? Disseminat-
ing research findings for hospitals. Int J Quality Health Care 2020; 32:
84–88.
63. de Lusignan S, Correa A, Smith GE et al. RCGP Research and Surveil-
lance Centre: 50 years’ surveillance of influenza, infections, and respiratory
conditions. The British J General Practice: The J Royal College General
Practitioners 2017; 67: 440–41.
64. Truyers C, Goderis G, Dewitte H et al. The Intego database: background,
methods and basic results of a Flemish general practice-based continuous
morbidity registration project. BMC Med Inform Decis Mak 2014; 14:
48.
65. Grimshaw JM, Ivers N, Linklater S et al. Reinvigorating stagnant sci-
ence: implementation laboratories and a meta-laboratory to efficiently
advance the science of audit and feedback. BMJ Quality Safety 2019; 28:
416.