(Coomarasamy et al, 2004)
EBM daily ward teaching rounds (Coomarasamy et al, 2004)
EBM journal club based on queries from the ward or

Received February 18, 2007; revised June 22 and September 2, 2007; accepted October 1, 2007. The authors are af\ufb01liated with the De- partment of Psychiatry at the University of Colorado Health Sciences Center. Address correspondence to Robert E. Feinstein, M.D., Se- nior Associate Dean of Education, University of Colorado Denver School of Medicine, Mail Stop F523, 13001 East 17th Place, Room E1330, Aurora, CO 80045; Robert.Feinstein@ucdenver.edu (e-mail).

Psychiatry Evidence-Based Medicine (EBM) Project began to investigate whether formal educational interven- tions could help residents develop a positive attitude to- ward EBM, acquire EBM knowledge and skills, and fa- cilitate the daily use of EBM with patients in a psychiatric outpatient residency-training site. We developed our cur- riculum and teaching approach by reviewing the world lit- erature, three published EBM curricula (1\u20133), and addi- tional EBM curricula presented at American Association of Directors of Psychiatric Residency Training (AADPRT) meetings. In December 2006, we found no published com- prehensive EBM curricula from any specialties reporting effectiveness data.

Two common approaches to EBM education found in the literature include the use of problem/case-basedlearn- ing methods (4\u20139) and adult learning theory (10). These approaches emphasize active learning to link knowledge and skill with clinical practice. In 1995, a systematic review of 102 trials (8) revealed that didactic approaches had little effect on physicians\u2019 clinical practice. A recent 2004 meta- analysis (9) reviewing 18 studies using standalone EBM teaching methods demonstrated that these methods im- proved EBMknowledge, but failed to change attitudes to- ward EBM or foster clinical use of EBM with patients. Table 1 reviews empirically validated, clinically integrated teaching methods (9, 11\u201314), utilized by nonpsychiatric colleagues, fostering clinical use of EBM.

lative effectiveness of a curriculum with educational inter- ventions. The goal is to signi\ufb01cantly increase resident knowledge, skills, and clinical use of EBM with psychiatric outpatients. The project includes curriculum development, implementation of four educational interventions, devel- opment of two attitude questionnaires, development and validation of a psychiatry EBMknowledge exam, and cur- riculum evaluation using a pre/post-intervention design. This article describes the curriculum, study instruments, and preliminary evidence about the program\u2019s effective- ness using data from 37 psychiatry residents who volun- tarily enrolled in the study.

The resident curriculum is progressive over 3 years (postgraduate years 1\u20133) and follows a traditional EBM approach of teaching the 6As (Assess a patient, Ask

a clinical question, Acquire the information, Appraise the information, Apply the information with a patient, Assess the outcome with the patient) (3\u20135). Residents learn and practice assessing a patient, asking an EBM question using the PICO-QQ format (Population, Interventions, Com- parison group, Outcome, Question type, Quality of the study), acquiring the information using online library searching of textbooks and the world literature, appraising the literature using EBM mathematics, applying the liter- ature with a patient, and assessing patient outcomes. All courses are highly interactive and use computers, mini-lec- tures, problem/case-based learning, and resident teaching.

The PGY-1 course consists of three weekly, 2-hour ses- sions, designed to introduce EBM and create interest in the 3As (Assess a patient, Ask a question, Acquire the information), while avoiding EBM mathematics. The con- tent addresses the question, \u201cWhat is EBM and why use it?\u201d as well as assessing, asking, and acquiring articles from online searches.

The PGY-2 course meets weekly and consecutively for 6 hours. Two sessions are devoted to practicing the 3As. Residents choose one randomized controlled trial to eval- uate. Three sessions focus on critical appraisal skills and an introduction to EBM mathematics (1\u20136). The last ses- sion covers outcomes and application of information with patients, with consideration of patient values and prefer- ences.

The PGY-3 course consists of 25 75-minute classes. The \ufb01rst 15 sessions solidify use of the 6As and EBM mathe- matics. Faculty members model an ideal case for two ses- sions. In 13 sessions, residents present their own cases to practice and teach all 6As. With faculty support, residents lead critical appraisals of therapy, use EBM mathematics, and learn to use outcome measures. The remaining 10 ses- sions use the 6As with practice guidelines, systematic re- views, and a wider variety of outcome measures. Practice guidelines are critically appraised, using the AGREE (15) assessment. Meta-analyses/systematic reviews are critically appraised using guidelines described elsewhere (4\u20136, 16).

Use of EBM in daily psychiatric care is strongly empha- sized in the PGY-3 outpatient year. Midway into the PGY- 3 course, we add four educational interventions designed to facilitate residents\u2019 clinical use of EBM.

dex case.\u201d In the subsequent hours, the resident discusses with faculty any knowledge gaps in using the 6As. Together they set individualized learning goals. By the end of this session, the resident will have selected an outcome mea- sure, from a rating scale book (17) or CD, which will be used with the index patient. The resident is also introduced to the EBM Reminder Survey and completes the \ufb01rst sur- vey detailing usage of EBM practices during the session.

index patient, the resident explains, negotiates, and begins use of an outcome measure with the patient. Using the EBM Reminder Survey, the resident details EBM activi- ties during this and all subsequent patient visits.

index patient visit, the resident chooses four additional cases. A staff member from medical records attaches the EBM Reminder Survey to the front of the additional charts for the next \ufb01ve visits of each patient. After each patient visit, the resident completes a survey. Over many months, each resident can complete a total of 25 surveys.

residents begin working with their index cases, four EBM attendings, who precept all PGY-3 residents, begin using a 1-minute preceptor \u201cmicroskills\u201d (14, 18, 19) approach, which is supportive of EBM. The attending tries to elicit a commitment from the resident regarding a case formu- lation and treatment plan, probe the resident\u2019s thinking and evidence supporting all decisions, teach something new about EBM, reinforce what was done correctly, and correct mistakes.

A critical component of curriculum implementation in- volves availability of faculty knowledgeable in EBM. With only one expert EBM teacher, we formed a four-member EBM faculty self-teaching group, designed to help all fac- ulty learn EBM and prepare to co-teach EBM courses. We also scored exams and supervised in the outpatient clinic using microskills. These attendings all work in the outpa- tient department, a major teaching site for all residents. Close resident-faculty working relationships may have in- \ufb02uenced the 100% willingness of 37 eligible residents to enroll, with only one resident opting out of the clinical component of the study. Senior psychiatric research and clinical faculty volunteered to take, and completed, an

Based on the literature and prior experience teaching EBM to residents, we assumed that residents would have little prior exposure to EBM. We hypothesized that atti- tudes toward EBM and EBMknowledge, as measured by the Colorado Psychiatry Evidenced-Based Medicine Ex- amination (CP-EBM Exam), would increase progressively with this curriculum. After six sessions, we expected resi- dents would do more searching using PICO-QQ. After 25 sessions, we hypothesized that R3 scores on the CP-EBM Exam would substantially increase. Furthermore, we hy- pothesized that application of EBM with patients would not signi\ufb01cantly increase until PGY-3 residents had a 3- hour teaching session, saw patients, completed the EBM Reminder Survey, and received feedback from attendings using the 1-minute preceptor. To evaluate these hypothe- ses, we conducted a prospective pre/post-intervention study examining changes in resident attitudes/beliefs, knowledge, and clinical use of EBM at multiple time points.

Three instruments were prospectively designed to assess program impact: the EBM Attitudes and Beliefs Survey; the EBM Reminder Survey, designed to prompt and track resident use of EBM with patients; and an EBM knowl- edge exam.

The EBM Attitudes and Beliefs Survey is a 72-item questionnaire assessing prior computer use and experi- ence, prior experience with EBM, perceived importance of using EBM, perceived con\ufb01dence in EBM skills, and per- ceived barriers or facilitators to EBM use. The attitude questionnaire is administered before and after each course and after clinical teaching interventions.

The EBM Reminder Survey is placed on the front of patient charts, prompting residents to use EBM during clinical care. Nine questions ask residents to detail their use, or nonuse, of EBM practices after each of 25 visits. Investigators are blinded to which residents complete the surveys and any identifying patient information, except for the initial index patient visit.

The Colorado Psychiatry Evidenced-Based Medicine Examination (CP-EBM Exam) is a 14-question exam as- sessing EBM knowledge and skills. Based on the Fresno Test of Competence in Evidence-Based Medicine (20), the psychiatry version and scoring rubric were developed in

consultation with Dr. Ramous. The 14 open-ended ques- tions attempt to reproduce the 6As clinical thinking used during a patient encounter. The exam is administered to residents with an open time frame of 60\u2013120 minutes.

To validate the exam and scoring rubric, at random we chose 8 out of 20 exams previously completed by PGY-4 or PGY-5 child fellows not enrolled in the study but re- ceiving EBM instruction fromthe \ufb01rst author. Initially, rat- ers independently scored the same exam and compared results on each of 14 items. We discussed all items on which our scorings disagreed. After three rounds, using three different exams, we agreed on all answers. Interrater reliability was obtained using our four investigators, scor- ing two rounds of \ufb01ve additional exams. Raters were blinded to resident and resident year. One common exam was embedded in each group of \ufb01ve exams for each rater, and raters were blinded to the common exams. Reliability was estimated using average interrater correlations on the total score and by the intraclass correlation. The interrater reliability for the two rounds of \ufb01ve tests was 0.95, and the intraclass correlation was 0.93. Given these high reliabili- ties, we were comfortable having a single investigator score resident exams. Preliminary EBM scores are reported by number of sessions, while pre/post-improvement in scores is reported in the Results section.

To validate the exam, psychiatry EBM experts were identi\ufb01ed using the AADPRT list of e-mail addresses. All experts identi\ufb01ed themselves as EBM teachers in their re- spective psychiatry residency training programs. Five ex- perts volunteered to complete the exams via the honor sys- tem following the same test conditions offered to residents. Two investigators scored the expert exams. With 226 as a perfect score, four experts scored in the range of 211\u2013221, and one expert scored 178.

results of our attitudes and beliefs survey consist of data from four resident groups, totaling 37 residents in all 4 years. Table 2 details the number for each year, the number of residents who took and completed the attitudes and be- liefs survey at different points in time, and the numbers from our sample that were available for our preliminary analysis.