PROSEDUR

OSCE, MINI-CEX DAN PORTOFOLIO

Susanto Nugroho

Laboratorium Ilmu Kesehatan Anak

Fakultas Kedokteran Universitas Brawijaya
Sir William Osler (1849-1919)
Father of Modern Medicine
“He who studies medicine without
books sails an uncharted sea, but he
who studies medicine without
patients does not go to sea at all”

“No teaching without the patient for

a text and the best teaching is often
that taught by the patient himself”
…....in Clinical Teaching/Training
Patient is very IMPORTANT !!!

How should we know if students

competence?
How should we know if
students have
good performance?

“CLINICAL ASSESSMENT”
OBJECTIVE
After this lecture, all participants will be able
to describe and apply these clinical
assessments following in medical
education/training:
• Objective Structured Clinical
Examination (OSCE)
• Mini-Clinical Evaluation Exercise (Mini-
CEX)
• Portfolio
Clinical
Assessment
Why should we
assess the medical
students in clinics ?
Why to assess the medical
students ?
• Evidence of competence/performance &
inform progression (Tomorrow’s doctors -
GMC, 2003)
• To drive learning (van der Vleuten, 2000)
• To improve trainee confidence (van der
Vleuten, 2000)
• Public confidence:
- Scepticism of profession to self-
regulate (Smith, 1998)
- Better measures of quality of practice
(Scally, 1998)
Clinical
Assessment
What should we
assess the medical
students in clinics ?
Clinical
Assessment
COMPETENCE
~ “able to do”
PERFORMANCE
~ “actually does”
Clinical
Assessment
How should we
assess the medical
students in clinics ?
How to assess the clinical
• OSCE ~ Objective structure clinical examination
competency?
• OSLER ~ Objective structured long case
examination record
OSATS
•...... ~ Objective
undertaken structures
”outside the assessment
real” clinical environmentof
technical
...... skills
have many aspects of realism of workplace

How to assess the clinical

• Mini-CEX ~ Mini-clinical evaluation exercise
performance?
• DOPS ~ Direct observation of procedural skill
• CbD ~ Case-based discussion
• Mini-PAT ~ Mini peer assessment tool
...... undertaken ”on the real” patient & workplace
...... have many aspects of realism of workplace
 Assessment ~ Miller’s Pyramid
(Miller, 1990)
Mini-CEX, DOPS, CbD,
Mini-PAT, Portfolio

Performance Does
OSCE, short case,
long case

Competence Shows how MEQ, EMQ, PMPs,

SAQ, SEQ

Knows how
Knowledge MCQ, Essay, Oral

Knows

A framework for assessing clinical competence and performance

 Assessment ~ Miller’s Pyramid
(Miller, 1990)

Performance Does OSCE

Competence Shows how

Knows how
Knowledge
Knows

A framework for assessing clinical competence and performance

Latar Belakang
UU Praktek Kedokteran  Konsil Kedokteran
Indonesia (KKI)  SKDI & Standar Pendidikan
Profesi Dokter pada tahun 2006
(Direvisi tahun 2012)

KBK
Kemampuan
Institusi Berbeda

Kualitas
STANDARISASI
Berbeda
 Standar Pendidikan Dokter

INPUT PROSES OUTPUT OUTCOME

Kualitas
Kualitas Kualitas Kualitas
lulusan
penerimaan pendidikan & profesionalisme
pembelajaran

ASESSMENT

OSCE
Affandi, 2008
Pengertian OSCE

Objective: semua peserta diuji dengan

O ujian yang sama

Structured: penilaian di setiap stasion

S terstruktur & yang diujikan adalah
ketrampilan klinik tertentu (anamnesis,
PF, prosedur tindakan, dll)
Clinical Examination: penilaian
CE terhadap kemampuan ketrampilan klinik
(bukan pengetahuan) & mahasiswa
harus mendemonstrasikan
Tujuan OSCE
Menilai kompetensi dan ketrampilan klinis
mahasiswa secara objektif dan terstruktur.

Komponen dalam Pelaksanaan

OSCE
• Blue print & soal • Peserta/kandidat
ujian • Manekin & peralatan
• Penguji ujian
• Pasien standar • Sarana & prasarana
(PS) • Supporting team/staf
• Pelatih PS
Rancangan OSCE
Station
1
Station Station
12 2

Station Station
11 3

Station Station
10 4

Station Station
9 5

Station Station
8 6
Station
7
Rancangan OSCE
Heteroanamnesis
ibu dengan anak
sakit

Melakukan
Pemeriksaan
Prosedur
Abdomen
Aseptik

Blue Print
Pemeriksaan
Bedah Minor
Refleks

Anamnesis
Pemeriksaan
Penyakit
Ginekologis
Kronis
Blue Print OSCE
• Kategori kompetensi: kemampuan
anamnesis, pemeriksaan fisik, penunjang &
interpretasinya, prosedur tindakan, edukasi
& profesionalisme sesuai standar
kompetensi
• Sistem organ/divisi ~ sesuai dengan yang
akan diujikan
• Kasus: untuk memberikan situasi klinik
yang diharapkan
• Distribusi & proporsi pencapaian
kompetensi: perilaku profesional harus
 Sistem Endokrin &
Muskuloskeletal

Muskuloskeletal

Muskuloskeletal

Sistem Kulit &

Sistem Kulit &

Hematopoietk

Hematopoietk

jaringan Ikat

jaringan Ikat

Metabolik
Minimal

Sistem

Sistem

Sistem

Sistem

Sistem
Katagori Kompetensi

Kasus

1. Anamnesis 1 +

2. Pemeriksaan fisik 1 + +
3. Melakukan
tes/prosedur
3 + + +
klinik/interpretasi
data
4. Menentukan
diagnosis atau 5 + + + + +
diagnosis banding
5. Penatalaksanaan:
a. Non 1 +
Farmakoterapi
b. Farmakoterapi 4 + + + +
Template Stasion OSCE
 Standard Setting OSCE

Absolute Methods Compromise Methods

• Anggoff (modified) • The Hofstee

• Ebel Method
• Borderline Group
Method
• Borderline
Regression
Method
Borderline Regression Method
• Metode standar setting yang sering digunakan
pada OSCE
• Penilaian meliputi: “Actual Mark” dan “Global
Rating”
• Actual Mark: deskripsi skor (0 s/d 3) di daftar
tilik (rubrik) harus jelas agar penguji tepat
dalam memberikan skor
• Global Rating: persepsi penguji terhadap
“overall performance” (meminimalisasi
subyektivitas penilaian)
1 = tidak lulus
2 = borderline (minimally competence)
Penilaian (Skor) yang
Dimasukkan
Total nilai :
Actual Mark : ..............

Global Rating :
1. tidak lulus 2. borderline
3. lulus 4. outstanding
Penentuan “Minimum Passing
Level ” (MPL) NO GR AM
1 1 10
2 3 20
30 3 2 14
25 4 2 18
5 3 22
20
6 4 28
15 7 4 30

10 8 3 26
9 2 16
5
10 3 24

1 2 3 4
 Global Actual 30

Rating Marck
1 4 24
2 1 12 25
3 1 9
4 2 15
5 2 20
6 3 22 20 ≥14
7
8
2
3
16
17
Lulus
Actual Marck

9 1 10 15
Actual Marck
10 2 12
Linear (Actual Marck)
11 3 14
12 4 22
14
10
13 4 21
14 3 16
15 1 8
16 4 26 5

17 3 20
18 2 11
19 3 16 0
20 1 8 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
Global Rating
Kelebihan OSCE
1. Valid
2. Reliabel
3. Setting klinik yang nyata dan menarik
4. Berbagai ketrampilan dengan variasi dapat
diujikan dalam waktu yang relatif singkat
5. Setting standar kompetensi tertentu dapat
ditentukan
6. Obyektif: variasi pasien dan penguji dikurangi
7. Format OSCE bersifat fleksibel
8. Pengamatan langsung pada setiap peserta
9. Terstruktur dan terencana
10.Feasibel
Kekurangan OSCE
1. Kompartementalisasi
2. “High cost”
3. “High human resources”:
- Pasien standar (PS)
- Pelatih PS
- Penguji yang terlatih: perlu pelatihan
4. “High time consuming”: untuk persiapan,
pelaksanaan dan evaluasi
5. Perlu organisasi dan koordinasi yang baik
 Assessment ~ Miller’s Pyramid
(Miller, 1990)
MINI-CEX

Performance Does

Competence Shows how

Knows how
Knowledge
Knows

A framework for assessing clinical competence and performance

Features of Mini-CEX
Method to assess clinical performance of students in
workplace – “DOES” level (Miller’s pyramid)
Key features – real patients, clinical conditions, work
settings, clinical tasks and constrains
Relies on – multiple encounters, assessors, settings,
occasions and judgements
Involves – clinicians to score, short focused patient-trainee
interaction, and clinician offering ± 5 minutes of
developmental feedback to trainee to improve quality of
clinical skills
Advantages of Mini-CEX
Can assess the clinical skills – based on standard of
competencies
Can assess the clinical performance – clinical skills,
student – patient interaction (attitudes & behaviour)
Direct observation & immediate feedback – not only
helps student to identify strength & weakness, but can helps
to improve skills
Need few minutes – 10-15 minutes of observation & 5-10
minutes of feedback
Mini-CEX Forms

Student’s information

Assessment

Feedback & action plan

Assessor ID
Competencies assessed &
descriptors
Step 1. Preparation
Step 2. Observation &
Assessment
Step 3. Verbal & Written
Feedback
The Validity of Mini-CEX
The Reliability of Mini-CEX

The mini-CEX is a reliable tool for performance

assessment, and is acceptable to and well received
by both learners and supervisors.
Nair, et al., 2008
Changes in reliability as a function of
the number of encounters

Students should be observed at

least four times by different
assessors to get a reliable
assessment of competence.

Norcini & Burch, 2007

The Reliability of Mini-CEX
Mini-CEX is a good example of workplace-
based assessment method that fulfils three
requirements for facilitating learning:
1. The course content, expected competencies &
assessment practices are aligned
2. Feedback is provided either during or
immediately after the assessment
3. The assessment is used to direct learning
towards desired outcomes

Norcini & Burch, 2007

The Educational Impacts of
Mini-CEX

• Knowledge acquisition and enhancement of

giving feedback when the faculty members
used the tool.
• Providing effective feedback should be
conducted to increase the impact of the mini-
Liao, et al., 2013
CEX as a formative assessment.
Conclusions
Mini-CEX ~ a valid, reliable & feasible
Validity – it is important instrument for the direct observation
of trainee’s clinical performance
Reliability
• more encounters & assessors - more reliable
• expected competency, feedback & used to direct learning
Educational impacts
• can correct the weakness & mature professionally
• can monitor progress & identify educational needs
• reassures student’s satisfactory performance
• increases the interaction student & teacher
 Assessment ~ Miller’s Pyramid
Portfolio

Performance Does

Competence Shows how

Knows how
Knowledge
Knows

A framework for assessing clinical competence and performance

 Portfolio

EXAMINATION
toward broader methods of
assessment
• To encourage closer links between assessment &
learning using assessment & feedbackby 
learning improvement
• To enhance the assessment of areas that are
difficult to assess by traditional methods: attitudes,
personal attributes, reflection & professionalism
Haldane, 2014
Definition of Portfolio

A purposeful collection of work

(Stecher, 2001)

A collection of papers and other

forms of evidence that learning has
taken place
(Davis et al.,
A collection2001)
of student work that exhibits the
student’s efforts, progress and achievements in
one or more areas
(Gisselle & Martin-Kniep, 2000)
Why Use Portfolio ?
1. Portfolios’ contribution to assessment
2. Focus on personal attributes
3. Enhances interactions between students &
teachers
4. Stimulates the use of reflective strategies
5. Expands understanding of professional
competence
 Models of Portfolio
Portfolio Description Advantages Disadvantages
Shopping Contains anything which Very inclusive Difficult to assess.
trolley has been produced or No analysis of
used during the learning contents
proces
Toast rack “Toast” for each period of Corresponds with Each item is
learning the curriculum discrete & does not
Can be marked provide overall
Includes assessment of
reflection learning
No overall
reflection
Cake mix Integration of the parts Global Individual
“Mixing” is reflection on assessment components may
the not be clear
analytical components
Spinal Series of competency Each
Portfolio Contents, Alignment &
Assessment
• Portfolio model should be easily aligned with
the curriculum ~ “toast rack” or “spinal
column”
• Portfolio should provide opportunity to
demonstrate learning in many different ways
& should be a holistic record of learning
• The appropriate model depends on its
purpose  will be used in assessment
process
• Structure should be decided based on the
format the evidence is required
Haldane, 2014
Five Steps in Portfolio
Assessment
Step Evidence
Process
I. Documentation of experience by the learner
Step Reflection
II. Commentary by the learner on experiences
and learning that has resulted
Step Evaluation
III. Studying the evidence by examiners
Step Defending the evidence
IV. A dialogue between learner and examiner
Step Assessment decision
V. Formative and summative
Davis & Ponnamperuma, 2006
Learning Outcome
A three-circle classification model of
learning outcome
A. Inner circle ~ “what the doctor is able to
do”
(1) Clinical skills
(2) Practical procedures
(3) Investigating a patient
(4) Patient management
(5) Health promotion & disease prevention
(6) Communication skills
(7) Information handling & retrieval Harden et al, 1999
B. Middle circle ~ “how the doctor
approach the task”
(8) Understanding of basic, clinical & social
sciences
(9) Appropriate attitudes, ethical understanding
& legal responsibilities
(10) Appropriate decision making, clinical
reasoning
C. Outer circle&~judgement
“doctor as a professional”
(11) The role of the doctor within the health
service
(12) Attitude for personal development

Harden et al, 1999

Steps in Developing Portfolio
1. Defining the purpose
2. Determining competences to be assessed
3. Selection of portfolio material
4. Developing a marking system
5. Selection and training of examiners
6. Planning the examination process
7. Student orientation
8. Developing guidelines for decisions
9. Establishing reliability and validity evidence
10. Designing evaluation procedures

AMEE Medical Education Guide, 2001

1. Defining the purpose
The Purposes of Portfolio
1. For the trainee:
• to record the training experience, education
supervision, professional development
plans, workshops attended, reflective entries
and assessment reports
• to identify deficiencies
• to plan for necessary remediation
2. For supervisors:
• to assess overall training & work with the
trainee
• to correct deficiencies
Joint Committee on Specialist Training, 2011
Expected Achievement of
Competencies
Expected achievement in the 6 core
competencies:
1. Patient care
2. Medical knowledge
3. Practice based learning & improvement
4. Interpersonal & communication skills
5. Professionalism
6. System based practice
2. Determining competences to be
assessed

Joint Committee on Specialist Training, 2011

Expected frequency of assessments
Other areas of curriculum & assessment
3. Selection of portfolio material
To assess: patient management skills, e.g.
patient education
• Written outline of a patient education
programme in community
• A video of individual patient education
session with patient discussing one topic
To assess: attitudes, ethical & legal
understanding & responsibility
• The elective report ~ the student shows
ethical understanding of issues inherent in the
elective
• Case discussion on ethics ~ the students will
provide evidence of ethical judgement & moral
Record the cases
Reflective entries
DOPS for
Medical
Training
Record of Procedures
Name of procedure: Intraosseous needle insertion

Name of procedure: Percutaneous central line insertion

Practice Based Learning Assessment
Teaching of Communication Skill Evaluation
4. Developing a marking system
• Students’ work is judged by criteria which will
specify the level of their academic
achievements and will determine their progress
towards state standards.
• The portfolio material should direct the
examiner to consider student progress
according to the outcome specification & should
enable the examiner to identify strengths &
weaknesses.
• If one competence is assessed, highly specific
criteria could be employed; if multiple
competences, general standards should be
developed.
5. Selection & training of examiners
• According to its purpose of portfolio  the
appropriate examiners, include: staffs,
teachers in the basic sciences & laboratory-
based diciplines, clinicians, faculty who
indicate special interest in education &
student development.
• Another selection issue: seniority of
examiners
• A key point of the success of programme:
the training of faculty examiners &
maintaining them  should be preserved &
reinforced
6. Planning
the
examination
process
7. Student orientation
• Students must be informed at the beginning
of the course about the portfolio
examination, the guidelines & criteria for
judging performance clearly.
• Students can use their on-going work for
selection of material (if they demonstrate
good progress in their achievements & have
confidence in ability to pass the portfolio)
• The more information given to students the
more positive they become towards the
portfolio.
8. Developing guidelines for
decisions
Flow diagram for the decision-making process
9. Establishing reliability & validity
evidence
• It is important to determine what will
constitute good reliable evidence & plan the
examination, e.g two independent
examiners, one examiners, independent
rating, consensus or both, minimum desired
reliability or generalisability co-efficient.
• Define desired correlations or absolute inter-
rater agreement and set the minimum
standards for tolerance of misclassification
error.
• Triangulation of portfolio results with other
forms of assessment will increase the
10. Designing evaluation
procedures
Feedback: “student” & “examiners” opinion on the
portfolio’s strength & weakness  changes &
improvements
• Questionnaires
• Focus group discussion
• Individual interview
• Request for written comments
Portfolio’s Assessment Features
1. Formative & summative
2. Qualitative & quantitative
3. Personalised
4. Standarised
5. Authentic
Thank You for Your Attention