Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes A Systematic Review Amit X. Garg, MD) Neil KJ. Adhikari, MD. Heather McDonald M, Patricia Rosas-Arellano, MD, PhD PT Devereaux, MD Joseph Justina Sa B. Brian Haynes, MD, PAD (OMPUTERIZED CLINICAL DECL- sion support systems (CDSSs) are information sys tems designed to improve clinical decision making. Characteris- tics of individual pati toa computerized knowledge ba soliware algorithms generate pat specific recommendations, Practition- crs, health care staf, or patients can manually enter patient characteristics into the compu tively, electronic medical records can bbe queried for retrieval of patient char- acteristics. Computer-generated re ‘ommendations are delivered to the cli- nician through the electronic medical record, by pager, or through printouts placed in a patient's paper chart. Such, systems have been developed for a myriad of clinical issues, including di- agnosis of chest pain, reatment of in- fertility, and timely administration of immunizations, These systems pro- vide several modes of decision sup- port, including alerts of critical val- sare matched and 1 system; alterna- See also pp 1197 and 1261. (©2005 American Medical Assoc ton, Al rights reserved. (Reprinted) JANA, sth 9 Context Developers of health care software have attributed improvements in pa- tient care to these applications. As with any health care intervention, such claims re- quite confirmation in clinical tials. Objectives To review controlled tials assessing the effects of computerized clinical decision support systems (CDSSs) and to identify study characteristics predicting. benefit. Data Sources. We updated our earlier reviews by searching the MEDLINE, EMBASE, ochang bay Inspec, and 5 databases and consulting reference st trough Sep tember 2004. Authors of 64 primary studies confirmed data or provided addtional information. Study Selection We included randomized and nonrandomized controlled trials that evaluated the effect of a CDSS compared with care provided without a CDSS on prac~ titioner performance or patient outcomes, Data Extraction Teams of 2 reviewers independently abstracted data on methods, setting, CDSS and patient characteristics, and outcomes. Data Synthesis One hundred studies met our inclusion criteria. The number and methodologic quality of studies improved over time, The CDSS improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome, Including 4 (40%) of 10 diagnostic systems, 16 (76%) of 21 reminder systems, 23, (62%) of 37 disease management systems, and 19 (66%) of 29 drug-dosing or prescribing systems. Fifty-two trials assessed 1 or more patient outcomes, of which 7 trials (13%) reported improvements. Improved practitioner performance was associated with CDSSs that automatically prompted users compared with requiring Users to activate the system (success in 73% of trials vs 47%; P=.02) and studies in which the authors also developed the CDSS software compared with studies in which the authors were not the developers (74% success vs 28%; respectively, P=.001), Conclusions Many CDSSs improve practitioner performance. To date, the effects on patient outcomes remain understudied and, when studied, inconsistent, JAMA, 20052931222-1238 ww jamacom | Author Aflaons:Ovsion of Nephciog (Drs Garg Scenes Cente and nerdearmental Dison of Ct 2nd Rosae-Arlano) and Departmen of Epidemic” cla (Or Ahila,Ponulton Heath Scenes Hes fy and satisis (Dr Cag). Unerty of West. plo Sk Cre Beene) ae acl of ed fin Ontaro, Lond: Departments of Cael pie- Gre (Ms San), Unvest fered, Toonla Onto. mology and Bostatistes (Ors Garg. Adnan, Conesponding Author: R Bran Payner MD, nO, Devers ardHaynesard Ms MeDenakd andived!- Cini Epenoiog and Bost, Fact of Beaty fe (Drs Devereaux and Hayes) MeMasterUnve~ Sdenes Mister Unventy, 1200 Mans Ream Sty, Hamton, Orta; Department of Cea Care 2C108, MeMacter Unversty, Bamiton, Ontano IMedine, Suneybock and Women's Colege Heath Canada LN 375 (bhaynesOmemast a sol 293, No. 101238 Downloaded From: by a Queen Mary, University of London User on 11/20/2017DECISION SUPPORT, PRACTITIONER PERFORMANCE, AND PATIENT OUTCOMES ues, reminders of overdue preventive Stues Eligible for Review ized controled tia, and cohort studies heli tasks, advice for drug prescrib- We included English-language ran- (complet strategies available from the ing, ciiques ofexisting healdreareor-domizedandnontandomizedtralawah authors). Palrs of reviewers indepes ders, and suggestions for various nc-acontemporaneouscontrol group that denulyevahnted the eligi fall std- tive care sues Compared patient care witha CDSS to ies identified in our search, Disages Toutine cate withouta CDSSandevali- tenis were resolved by third reviewer ated clinteal performance (ie, mes- _orby consensus. Pul- ext articles we sureofprocessofear) orapatient oul tetieved any viewer considered a Come. We stipulated thatthe CDSS had tation potentially relevant. Suppleme to provide patient-specific advice hat tary methods of finding studies in- tras reviewed by a health care practi- chuleda review ofarile reference ists, tioner before any clinical action: Stud- articles citing included studies as Uisted ies were excluded ifthe system (1) was _ inthe Scence Citation Index, PubMed tsed solely by medical students, @2) related articles feature, informatics con- only provided sursmatiesof patient in- ference proceedings, information pro formation, (3) provided feedback on vided by primary study authors, and groups of patients without individual other recent reviews" Where daa from tseessment, (4) only provided com- tral were distributed in more than 1 piteraided instruction, or (3) wasused publication, we cited the principal pub- for image analysis. studies assessing, heaton CDSS diagnostic performance against 2 defined gold standard were not in-_ Data Abstraction Cludedin this review unleascinial use Palrs of reviewers independently ab- othe diagnostic CDSS was also com- _stracted the following data from all pated with routine care. Based on these studies meeting eligi criteria: study Criteria, werreevaluatedallstudies rom setting, study methods, CDSS charac- comes, Disagreements were resolved by As with any health care innovation, CDSSs should be rigorously evaluated. before widespread dissemination into clinical practice. Various stages in this assessment process have been previ- ously described. Iterative qualitative and {quantitative assessment begin early in the software development cycle."* When, preliminary testing suggests that a CDSS improves clinical care or patient out ‘comes, confirmatory controlled tials are ‘warranted. We previously reviewed con- trolled trials of computer-aided quality assurance’ and CDSSs published up to 1992! and 1998. This fields rapidly evolving because of technological ad- vances, increasing aceess to computer systems in clinical practice, and grow- ing concern about the processand qual- ity of medical care, We therefore up- dated previous reviews to provide a ‘cumulative summary of controled trials taluating heellctivenessof CDSSson Finding Relevant Studies Siatenecres bpemenee we practitioner performance and patient We haye previously described ourmeth- attempted to contact primary authors outcomes. ods for finding relevant studies until of all included studies to confirm data March 1998, For this update, we ex- and provide missing dat METHODS amined citations from MEDLINE, All studies were scored for method- Research Questions EMBASE, Evidence-Haced Reviews dar ological qualty ona 10 point sale con The primary questions of this review. tabases (Cochrane Database of System- sisting of 5 potential sources of bias, were (1) Do CDSSs improve practi- atic Reviews, ACP Journal Club, Data- which we have described elsewhere.’ In Loner performance oF patient oul- base of Abstracts of Reviews of Effects, brief, we considered the method of al- ‘comes? and (2) Which CDSSandstudy- and Cochrane Central Register of Con- location to study groups (random, 2, vs level actorsare associated withelfective trolled Trials), and Inspec biblie- quasi-random, 1, vs selected coneur~ CDSSs? A priori, we hypothesized that graphic databases from 1098 through rent controls, 0), the unit of the alloca- studies reporting better outcomes September 2004. All citations were tion (a cluster such as a practice, 2, vs would assess CDSSs that automati- downloaded into Reference Manager, physician, 1, vs patient, 0), the pre cally prompted users (vs requiring the version 10.0 (Thomson ISI Research- ence of baseline dillerences between the user to actively initiate the system), Soft, Philadelphia, Pa). An experienced groups that were potentially linked to were built into an electronic medical librarian developed the search strate- study outcomes (of particular impor- record or computer order entry sys- gles using sensitive terms for identify tance for observational studies; no ba tem (vs a stand-alone system), pro- ing linical studies of CDSSs. We pilot- _ line differences present or appropriate vided reminders (vs information ondis- tested search strategies and modified statistical adjustments made for differ- ease management, drug dosing, or them to ensure that they identified ences, 2, vs baseline differences present diagnosis), were tested using less rig- known eligible articles. The final strat- and no statistical adjustments made, 1, corous study methods, were studied by egies used the terms computer-assisted ys baseline characteristics not reported, their software developers (vs by evalu- decision making, computer-assisted diag- 0), the objectivity of the outcome (ob- ators not involved in the CDSS de- nosis, computer-assisted therapy, deci- jective outcomes or subjective out- sign), described pilot testing, and de- sion support systems, reminder systems, comes with blinded assessment, 2, vs scribed user training, hhospital information systems, random- subjective outcomes with noblinding but 4224 JAMA Mass 9, 2005Vol 293, No. 10 Reprinted) (©2005 American Medical Association. All ights reserved. Downloaded From: by a Queen Mary, University of London User on 11/20/2017clearly defined assessment criteria, 1, vs subjective outcomes with no blinding and poorly defined, 0), and the com- pleteness of follow-up for the appropri ate unit of analysis (>90%, 2, vs 80 to 90%, 1, vs <80% oF not deseribed, 0) The unit of allocation was included be- cause of the possibility of group con- lamination in tials in which interven lions were applied to clinicians even though individual patients were allo- cated to the intervention and contral ‘groups. Contamination bias would lead tounderestimating the effect of CDSS, The studies substantially differed in the type and number of outcomes as- sessed. Inaddition, the majority ofstd- ies did not define a single outcome for statistical testing. We aimed to effi- cently summarize the benelits of CDSSs and to identily CDSS and study charac- leristics that predicted success. For a given study we abstracted all reported practitioner performance and patient health outcomes. Situations where the CDSS worsened outcomes were rare Thus, for each study we defined the ef- fects of CDSSs in terms of success, de- Fined as an improvernent in atleast 50% ‘of outcomes measured, each ata 2-sided significance level less than .05, Statistical Analysis Reviewer agreement on study eligibil- lty was quantified using the Cohen i." Study and CDSS characteristics predict- ing success were analyzed and inte preted with the study as the unit of analysis. Data we using descriptive summary measures, inelud- ing proportions for categorical vari- ables and mean (standard deviation) for continuous variables, Univariable and multivariable logistic regression mod- cls, adjusted for study methodological quality, were used to investigate asso- ations between the outcomes ofinter- fest and study-specific covariates de- lined in our a priori hypotheses. All analyses were carried out using the SAS statistical package, version 8.2 (SAS In- stitute Inc, Cary, NC). We interpreted P=.05 as indicating statistical signif- ‘cance; ll P values are 2-sided. When re porting results from individual studies (©2005 American Medical Assoc 1, All rights reserved. DECISION SUPPORT, PRACTITIONER PERFORMANCE, AND PATIENT OUTCOMES weccited the measures of association and P values reported in the studies, RESULTS Finding and Selecting Studies From 3907 screened citations, we re- twieved 226 full-text articles, and 100 trials met our criteria for review. The chance-corrected agreement between 2 independent reviewers for article in- clusion was good (=0.81; 95% con- fidence interval [CI], 0.73-0.88). Description of Studies The 100 trials examined more than 3820 practitioners or practices (me- dian, 42; range, 2-300 [when re- ported]) caring for more than 82895 patients (median, 488; range, 19- 12980 [when reported] from 1973 0 2004." The number of eligible trials imereased with time: lin 1970-1974, 4 in 1075-1079, 10 in 1980-1984, 13 in 1085-1989, 20 in 1990-1904, 26 in 1095-1099, and 26 in 2000-Septem- ber 2004. Of these 100 trials, most were conducted in the United States (60%), followed by the United Kingdom (14%), Canada (5%), Australia (4%), Italy (2%),and Austria, France, Germany, Is- rael, Norway, and Switzerland (1% each). Sixty-nine percent of trials de- scribed funding from the public sec- torand 16% from the private sector. De- velopers of CDSS software were also study authors in 72% of trials. Ninety- seven trials described the effect of CDSS fon at least 1 measure of health care practitioner performance. Fifl tials assessed at least 1 pati come. We successfully contacted au- thors of 91 trials, and authors of 64 tials provided adelitional information oF con- firmed the accuracy of abstracted data.* ‘Methodological Quality ‘Assessment Trial methodological rigor increased With time—36% of trials before the year 2000 were cluster randomized, com- pared with 67% after this time (P=.01), mtn nT aa See e 47.49, 51 52,36 60-65 67,6871, 73-75, 0,8, 298 fi, 16, 2-18, Downloaded From: by a Queen Mary, University of London User on 11/20/2017 (fall trials, 88% were randomized. OF the randomized trials, 40% were clus- ter randomized and 40% used a cluster as the unit of analysis or adjusted for clustering in the analysis. Twenty-four randomized trials and 1 cohort study ported a power calculation for a pre- specifed difference between groups on specific outcome, Fifteen ofthese trials (60%) calculated sample size based on 4 practitioner performance outcome, 9 (Go%) based on a patient outcome, and 1 (4%) based on the cost of prescribed medications. Only 2 studies examined patient outcomes without measuring practitioner performance, Of the 88 ran- domized trials, 52% described an ap- propriate method of generating ran- dom numbers and 28% reported allocation concealment. On the 10- point methods seal, the mean score was 7 (SD, 1.7) and the range was 2 to 10. Description of Users and CDSSs The 100 tials had the following char- acteristics: 92% of trials enrolled physi- cians as primary users, 48% enrolled training health care practitioners (in- terns and residents) as users, 34% dk seribed pilot testing with users prior to implementation, 42% described ser in structional training at the tine of mpl mentation, 76% took place in academic centers, and 33% were inpatient-hased. In 47% of studies, the CDSS was part of aan electronic medical record of com- puter order entry system, Most of thes ‘were early generation systems lacking the full functionality of eurrent systems. In 15% of studies, the CDSS had a graphi- cal user interface, Feedback from the CDSS occurred at the time of patient care in 88% of studies; in 60% the user was automatically prompted to use the sys- tem (ys the user atively initiating the sys- tem), and in 91% the CDSS suggested. new orders (vs critiquing existing or- ders). Expert physician opinion or elini- cal practice guidelines usually formed the knowledge base for the CDSS. The process of data entry into the CDSS was clear in 80% of wials, some of which used more than 1 method. Exist- ing personnel most often entered data (attending or training physician, 389%; (Reprinted) JAMA, Mish 9, 2005 Vo 295, No, 10 1235