The Mini-CEX

John Norcini, Ph.D.

The Mini-CEX

Overview

Why the mini-CEX Mini-CEX: Description and study results Faculty preparation Feedback to trainees Implementation strategies

Why: Clinical Skills are Important

Interview and PE are the primary source of diagnostic information

Good communication skills

Compared to lab studies and imaging

Data gathering mistakes are the most common cause of diagnostic errors

Improve health status of patients Increase the efficiency of care Decrease the likelihood of malpractice suits

Why: Clinical Skills are Deficient

Several studies document deficiencies

Mangione (1997) and ascultatory skills in students/residents Ramsey (1999) and history-taking/preventive health screening among primary care doctors Reilly (2003) and missed physical findings among residents

Detection through trainee assessment has historically been limited

Why: Traditional CEX has Flaws

One examiner observes a trainee interact with an unfamiliar (in)patient Trainee does a complete Hx/PE, presents findings, management plan, written record Examiner rates along several dimensions Takes about two hours 82% of trainees undergo a CEX in their first year

Why: Traditional CEX has Flaws

The trainee is evaluated with only one patient

Physician performance varies considerably from patient to patient

One third of the mice used in the experiment were cured by the test drug; One third of the test population were unaffected by the drug and remained in a moribund condition; The third mouse got away.
Erwin Neter

Why: Traditional CEX has Flaws

The trainee is evaluated by only one examiner

Examiners differ in stringency

You get 15 Democrats in a room and you get 20 opinions. Senator Patrick Leahy

Why: Traditional CEX has Flaws

Most real physicianpatient encounters are short and focused

"Reality is merely an illusion, albeit a very persistent one."

Albert Einstein

The task is artificial

Why: Formative Assessment and Feedback are Critical

Critical to learning and have a significant influence on achievement

General education (Hattie, 1999)

Meta-analysis of 12 meta-analyses Feedback is among the largest influences on achievement (ES=.79) Feedback alone effective is effective in 71% of studies

Medical education (Veloski et al., 2006)

Why: Formative Assessment and Feedback are Lacking

There is a lack of assessment/feedback in workplace

Medical students

Structured observation done for only 7-23% of students (Kassebaum & Eaglen, 1999) Only 28% of IM clerkships include formative assessment strategy (Kogan & Hauer, 2006) 82% were observed only once (Day et al., 1990) 80% observed never or infrequently (Isaacson et al., 1995)

Postgraduate trainees

Mini-CEX

Purpose

Description

Focuses on formative assessment of clinical skills Responds to the assessment problems of the traditional CEX Responds to the educational problems of the workplace

Requires observation and feedback

Examiner observes a trainee with a patient in any setting Trainee performs a focused task Examiner rates along several dimensions Takes 15-20 minutes Multiple encounters are expected

Intended to be short and routine

Data Gathered in a Field Trial

Participants

Demographics

421 trainees, 316 examiners, 21 programs (1228 encounters)

Patient complexity Patient sex Type of visit

Ratings

Setting

First or return

Interviewing PE Professionalism Clinical judgment Counseling Org./Efficiency Overall competence

1-3 Unsatisfactory 4-6 Satisfactory 7-9 Superior

Focus

Ambulatory, inpatient, ED, other Data gather, diagnosis, RX, counseling

Scale

A Broad Range of Patients and Settings Were Captured

Patients

Patients

Well patient visit Presenting complaints

Other common problems

Abdominal pain, chest pain, cough, dizziness, fever, headache, low back pain, shortness of breath, weight gain

Seizure, substance abuse, depression, dementia, rash CHF, hypertension, diabetes Sepsis, myocardial infarction

Multiple problems

Common internal medicine problems

Acute problems

Arthritis, asthma, COPD, CHF, CAD, diabetes, hypertension

Settings

Ambulatory, inpatient, ER

Encounters Were as Long as Anticipated

Time Observing
Time Providing Feedback

15 minutes
5 minutes

Ratings of Overall Competence Were High

500 400 300 200 100 0 Number

2 1

3 4

4 20

5 98

6 310

7 424

8 175

9 65

There Were Differences Among the Components

Interviewing Physical Exam Professionalism Clin. Judgment Counseling Org/Efficiency Mean(SD) 6.6 (1.0) 6.4 (1.1) 7.1 (0.9) 6.6 (1.0) 6.8 (0.9) 6.6 (1.0)

There Was Growth Throughout the Year

1st Qtr 7.4 7.2 7 6.8 6.6 6.4 6.2 6 Hx PE Prof Clin J Coun Org/Eff 2nd Qtr 3rd Qtr 4th Qtr

Components of Competence Were Highly Correlated

Hx Hx --PE .73 Prof .71 ClinJ .75 Coun .69 Org .72 PE --.65 .76 .61 .73 Prof ClinJ Coun Org

--.68 .68 .64

--.67 .78

--.67

---

Multiple Encounters Produce Reasonable Confidence

Encounters 1 2 4 8 12 95%CI +1.5 +1.0 +0.7 +0.5 +0.4 Range Around 4 2.5-5.5 3-5 3.3-4.7 3.5-4.5 3.6-4.4

Examiners Were Satisfied

Rating of Satisfaction: 7.0+1.3 Range 1-9

Field Trial Summary

A range of patient problems, settings, and types of visits were captured Encounters were about as long as anticipated Ratings increased throughout the year The components of competence were highly correlated Multiple encounters produced acceptable confidence intervals Examiners were satisfied

Selected Studies of the Mini-CEX

Feasible for use with undergraduates and correlated with other assessments Kogan, Bellini, Shea: Acad Med , 2002 and 2003 Correlated with an SP exam Boulet, McKinley, Norcini, Whelan: AHSE, 2002 Correlated with an ITE and ratings for PGs Durning, Cation, Markert, Pangaro: Acad Med , 2002 Can differentiate scripted videos Holmboe, Huot, Chung, Norcini, Hawkins: JGIM, 2004

Assessment Challenges

Not many trainees will be considered unsatisfactory

There remains a need for national assessment, perhaps near the end of specialist training

"Everywhere I go I'm asked if I think the university stifles writers. My opinion is that they don't stifle enough of them."
Flannery O'Connor

Assessment Challenges

Another assessment process is needed for unsatisfactory trainees

Traditional measures are appropriate

"The power of accurate observation is frequently called cynicism by those who don't have it."
George B. Shaw

Knowledge test Clinical skills exam (OSCE)

Rules out false negatives Provide diagnostic feedback

Assessment Challenges

Trainees have some control over who examines them and indirectly over the content of the assessment

It is hard to believe that a man is telling the truth when you know that you would lie if you were in his place.
H. L. Mencken

The assessment might be biased in their favor

Assessment Challenges

Standards across programmes will not be equivalent

Results will not be useful for national ranking of trainees

Equal opportunity means everyone will have a fair chance at becoming incompetent.

Laurence J Peter

Video Exercise

A trainee interviews and examines a patient in the clinic. Use the following scale

Rate the following skills

1-3 Unsatisfactory 4-6 Satisfactory 7-9 Superior

Interviewing PE Professionalism Clinical judgment Counseling Org./Efficiency Overall competence

Faculty Preparation

Direct Observation of Competence training (Holmboe, Hawkins, Huot, 2004)

Behavioral observation

Know what to look for Prepare resident and patient Minimize intrusiveness and interference

Performance dimension training

Decide which dimensions of performance are important

Faculty Preparation

Direct Observation of Competence training

Frame of reference training

Improve accuracy and discrimination Reduce stringency differences

Practice Didactic mini-lectures Small group and videotape evaluation exercises Practice with standardized residents and patients

Workshop

Faculty Preparation

Study of the DOC model (RCT)

Faculty who underwent training

Thought the workshop was excellent Felt more comfortable performing direct observation Were more stringent than control group faculty

Feedback

Trainees are rarely observed in patient encounters Limits the opportunity for evaluation and feedback

When observed feedback is sometimes poor

Mini-CEX requires observation and offers the possibility for educational feedback

The belief that all genuine education comes about through experience does not mean that all experiences are genuinely or equally educative. John Dewey

Video Exercise

A trainee interviews and examines a patient in the clinic. Use the following scale

Rate the following skills

1-3 Unsatisfactory 4-6 Satisfactory 7-9 Superior

Interviewing PE Professionalism Clinical judgment Counseling Org./Efficiency Overall competence

What feedback would you give the trainee?

Feedback

Factors affecting the impact of feedback

Characteristics of the feedback Perceived need for change Technique Creating an action plan Mentoring

Feedback: Characteristics

Response to positive feedback (Fidler et al, 1999)

83% of the 255 physicians contemplated a change 66% reported initiated a change for at least one aspect of practice Physicians who contemplated or initiated changes had lower mean ratings than did physicians who reported no change

Feedback: Characteristics

Response to negative feedback

Extremely negative feedback can lead recipients to abandon their goals (Kluger et al, 1996)

Feedback: Characteristics

Response to positive and negative feedback is influenced by perceptions

Accuracy (Sargent et al, 2005) Credibility (Albright et al, 1995) Usefulness (Brett et al, 2001)

Feedback: Need for Change

No perceived need to change when (Johnson et al, 1999)

Feedback was generally favourable (e.g. above the midpoint of the rating scale) Unfavourable feedback was consistent with a low self-evaluation

Feedback: Technique

Technique influences impact (Hewson et al, 1998)

Establish an appropriate interpersonal climate Use an appropriate location Elicit the learner's thoughts and feelings Reflect on observed behaviors Be nonjudgmental Be specific Offer the right amount of feedback Offer suggestions for improvement

Feedback: Action Plan

Creation of an action plan leads to change

Feedback alone does not cause change, it is the goals that people set in response to feedback (Locke et al, 1990) Trainees receiving negative feedback were more likely to set goals (Brutus et al, 1999)

Feedback: Mentoring

Mentoring increases the likelihood of change following feedback

Broad management literature (Luthans et al., 2003; Walker et al, 1999)

Feedback: Summary

Plan for good feedback (Holmboe et al., 2004)

Provide an assessment of strengths and weaknesses Enable learner reaction Encourage self-assessment Develop an action plan

Implementation Strategies

Efficient administration strategies

Interns first visit of the day in clinic Pre-rounds Discharge rounds

Faculty preparation must be ongoing and can be incorporated into routine meetings

Standing activities (e.g. noon conference), curriculum committee, competency committee

Summary: Mini-CEX

Addresses some of the needs of workplacebased education

Focuses on clinical skills Overcomes some the assessment problems Provides formative assessment and feedback Assessment and feedback

Faculty preparation is critical

Implementation strategies influence feasibility

Mini-CEX: A method for assessment and feedback in training

1:

Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003 Mar 18;138(6):476-81. PMID: 12639081 [PubMed - indexed for MEDLINE] Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003 Aug;78(8):826-30. Kogan JR, Bellini LM, Shea JA. Implementation of the mini-CEX to evaluate medical students' clinical skills. Acad Med. 2002 Nov;77(11):1156-7. PMID: 12431932 [PubMed - indexed for MEDLINE]

7:

2:

Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med. 2002 Sep;77(9):900-4. PMID: 12228088 [PubMed - indexed for MEDLINE]

8:

3:

Boulet JR, McKinley DW, Norcini JJ, Whelan GP. Assessing the comparability of standardized patient and physiciaan evaluations clinical skills. Adv Health Sci Educ Theory Pract. 2002;7(2):85-97. PMID: 12075142 [PubMed - indexed for MEDLINE]

4:

Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. J Gen Intern Med. 2004 May;19(5 Pt 2):558-61.
Hauer KE. Enhancing feedback to students using the mini-CEX (Clinical Evaluation Exercise). Acad Med. 2000 May;75(5):524. No abstract available. PMID: 10824798 [PubMed - indexed for MEDLINE]

9: Norcini JJ, Blank LL, Arnold GK, Kimball HR. Examiner Differences in the Mini-Cex. Adv Health Sci Educ Theory Pract. 1996;2(1):27-33. PMID: 12386412 [PubMed - as supplied by publisher] 10: Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995 Nov 15;123(10):795-9. PMID: 7574198 [PubMed - indexed for MEDLINE

5:

6:

Wragg A, Wade W, Fuller G, Cowan G, Mills P. Assessing the performance of specialist registrars. Clin Med. 2003 Mar-Apr;3(2):131-4. PMID: 12737369 [PubMed - indexed for MEDLINE]