Tools in Measuring Program and Learning Outcomes

ROJoson OCIL on Medical Education
OCIL = Online Collaborative and Interactive

Learning
Tools in Measuring Program and Learning Outcomes

of Medical Students
Tools in Measuring Program and Learning Outcomes of Medical

Students – A Guide in Scoring and Grading (Part 3)
Subtitle: Tools in Measuring Knowledge, Skill and Attitude of Medical
Students - A Guide in Scoring and Grading (Part 3)
Learning
Tools in Measuring Program and Learning Outcomes of
Medical Students
Students – A Guide in Designing a Master Blueprint (Part 1)
Students – A Guide in Designing the Tools (Part 2)
Recommended Prior Study:

• Assessment and Evaluation of Medical Students – ROJoson TPORs
• Tools in Measuring Program and Learning Outcomes of Medical Students – A
Guide in Designing a Master Blueprint (Part 1); A Guide in Designing the Tools
(Part 2)
Learning

Learning objectives:
At the end of the session, the student must be able to:

• Define scoring.
• Define grading.
• Define points.
• Define raw scores.
• Give examples of translated scores from raw scores.
Learning


• Differentiate norm-referenced and criterion-referenced system of
grading.
• Discuss the different methods on how to determine the pass or cut
score.
• Discuss the limitations of scoring and grading systems in a medical
school.
Learning


• Explain why medical schools should go for a criterion-referenced
systems of scoring and grading.
• Discuss the limitations of scoring and grading systems in medical
schools.
• Discuss the usefulness of item analysis.
Learning
Students – A Guide in Designing a Master Blueprint (Part 1)
ROJoson recommended framework
for the design and development of tools to measure the global
accomplishment of the student learning and performance
Elements to Tools to Use Performance Assessment
Measure to Measure Indicators and
Evaluation
Program Educational
Objectives
Program Outcomes
Course-level Learning
Outcomes
Session-level Learning
Objectives
Learning
Students – A Guide in Designing the Tools (Part 2)
GENERAL FORMAT OF A BLUEPRINT FOR EDUCATIONAL MEASURING TOOLS WITH
SEVERAL LEARNING OBJECTIVES AND OUTCOMES
Contents for Plans
Assessment Time Weight Number of Types of Instructions
allotment (relative questions questions and notes
(overall) importance
of contents)
LO1
LO2
LO3
LO4
LO5
*LO – Learning Objectives / Outcomes
Learning
Assumptions on activities that have been undertaken:

• The tools have been formulated. These consist of 3 categories:
TEST-BASED TOOL, OBSERVATION-BASED TOOL, and PROJECT-BASED
TOOL (per ROJoson recommendation for medical schools).
• What will be done now are: PLANS ON HOW TO SCORE and HOW TO
GRADE.
Learning
SCORING AND GRADING MANAGEMENT SYSTEMS
2 distinct but interrelated and should be integrated systems:
Scoring Management System
Grading Management System
TEST-BASED OBSERVATION- PROJECT-
TOOLS BASED TOOLS BASED TOOLS
(e.g. (e.g. OSCE) (e.g. Case
Written/Oral) Study)
Scoring Mgt
System
Grading Mgt
System
Learning
SCORING AND GRADING MANAGEMENT SYSTEMS
2 distinct but interrelated and should be integrated systems:
Scoring Management System
Grading Management System
TEST-BASED OBSERVATION- PROJECT-
TOOLS BASED TOOLS BASED TOOLS
(e.g. (e.g. OSCE) (e.g. Case
Written/Oral) Study)
Scoring Mgt
System Scoring and grading systems will be dependent
on the type of educational measuring tools to be
Grading Mgt used. They may differ.
System
Learning
SCORING vs GRADING MANAGEMENT SYSTEM
Scoring Management System refers to a set of policies, processes and

procedures on scoring of educational measuring tools.
Scoring refers to the process of assigning numerical units to represent
points that students will get and/or have gotten in educational
measuring tools such as test-based, observation-based and project-
based tools.
Learning
Grading Management System refers to a set of policies, processes and

procedures on grading of educational measuring tools.
Grading refers to the process of applying standardized measurements
for varying levels of achievements in a learning activity or course of
study using the scores the students will get and/or have gotten in
educational measuring tools such as test-based, observation-based
and project-based tools.
Learning
In terms of implementation of actual processes, there is a proper

sequence:
• Scoring precedes grading.
• Assign scores and get scores before giving grades.
However, in the formulation of the plan for scoring and grading, there
is already integration of the two processes – how to score and how to
grade are already placed in the plan.
Learning
POINTS, MARKS, SCORES, RATINGS, GRADES etc.

Some words of clarifications on use of terms which can be used
interchangeably and which oftentimes cause confusion
POINTS = MARKS
SCORES MAY BE THE SAME AS POINTS BUT NOT ALL THE TIME.
TOTAL POINTS = SCORES
RATINGS USED IN BROADER SENSE THAN GRADES.
GRADES UTILIZE THE SCORES FOR DECISION-MAKING.
SCORES ARE CONVERTED TO GRADES.
Learning
POINTS
Points are numeric units which represent relative value or weight of
importance determined by a teacher on specific items in educational
measuring tools such as tests, observation technique and project-
based technique.
The total point gotten by the students in each educational measuring

tool is the sum of all points assigned to each item of assessment.
The total point gotten by the students is the sum of all points on
correct answers and sum of all numeric values in a rating scale.
Learning
POINTS on a test with a numeric value on each question or item

Question Points assigned Student 1 Student 2
or item Points gotten Points gotten
(correct answers) (correct answers)
1 3 3 3
2 2 2 0
3 2 2 0
4 3 3 3
5 5 5 0
Total Points = 15 Total = 15 (100%) Total = 6 (40%)
Total points = Score (which may be the GRADE or not, if score will be standardized)
Learning
POINTS on an assessment tool with rating scale with assigned values
on each item (given an example of a student performance)
Key:
LP1 LP2 LP3 LP4 LP5 1 – Very poor
Items / 1 2 3 4 5 2 – Poor
Criteria Assigned points
3 – Average
4 – Good
1 ❸ 5 – Excellent
2 ❹
There is an
3 ❺ accompanying
Total points gotten by student = 12 descriptor of
criteria of level of
performance.
LP – Level of Performance with Descriptor
Learning
SCORES
SCORES are numeric units which represent the total number of points
that students will get and/or have gotten in educational measuring
tools such as tests, observation technique and project-based
technique.
SCORES = TOTAL POINTS

POINTS in specific or groups of items or questions – “scores” not
usually used, but “points.”
Learning
POINTS on a test with a numeric value on each question or item
Question Points assigned Student 1 Student 2
or item Points gotten Points gotten
(correct answers) (correct answers)
1 3 3 3
NOT SCORE
2 2 2 0
but
3 2 individual 2 0
points for
4 3 3 3
each item
5 5 5 0
Total = 15 Total = 15 (100%) Total = 6 (40%)
If above is the whole test, the total point of 15 of Student 1 is also

the score; total points of 6 of Student 2 is the score.
Learning
SCORES
RAW SCORES – scores without any sort of adjustment or

transformation such as simple number of questions answered
correctly
TRANSLATED SCORES – the result of some transformation(s) applied to
the raw scores – adjusted and converted to a standardized grading
system
The score at which a student passes or fails is called the pass or cut

score.
Learning
Translated Scores – Example
Raw Scores Range of Scores Grade Point Letter Grade

(examples) System System
96 95-100 1 A
92 90 -94 1.5 B
86 85-89 2 C
83 80-84 2.5 D
76 75-79 3 E
70 70-74 4 R
60 Below 70 5 F
Learning

Raw Scores Norm- Range of Grade Point Letter Grade
(examples) referenced Scores System System
scores
96 Raw scores 95-100 1 A
92 adjusted to 90 -94 1.5 B
the mean –
86 0.5 to 1 SD 85-89 2 C
83 as the pass 80-84 2.5 D
score based
76 on norm- 75-79 3 E
reference
70 system 70-74 4 R
60 Below 70 5 F
Learning

Raw Scores Criterion- Range of Grade Point Letter Grade
(examples) reference Scores System System
scores
96 Raw 95-100 1 A
92 scores 90 -94 1.5 B
adjusted
86 to 85-89 2 C
criterion-
83 referenced 80-84 2.5 D
76 system 75-79 3 E
70 70-74 4 R
60 Below 70 5 F
Learning
GRADES
Grades can be assigned as letters (usually A through F), as a range (for

example, 1 to 5), as a percentage, or as a number out of a possible
total (often out of 100).
Grades in medical schools are usually in numeric units.

May be using raw scores or translated scores.
May be using norm-referenced or criterion-referenced systems for the
passing grades.
Learning
GRADES
GRADES CAN BE BASED ON RAW SCORES.
For example, if 100 points is the total score, and a student got 100
points or total score, the student gets a grade of 100%. Another
student got 90 points or total score, then this student gets a grade of
90%.
Learning
GRADES
GRADES can be based on TRANSLATED SCORES.
If scores within
95 -100 points or score = Grade of 1
90 – 94 points or score = Grade of 1.5
85 – 89 points or score = Grade of 2
For example, a student got 93 total points or score, the student gets a
grade of 1.5. Another student got 89 points or total score, then this
student gets a grade of 2.
Learning
GRADES
GRADES CAN BE BASED ON:
Norm-referenced vs Criterion-referenced Systems
Norm-referenced means referencing how the score of a student

compares to other students. (grading on a curve using mean and
standard deviations)
Criterion-referenced means referencing how the score compares to a
criterion such as a cut score or a body of knowledge. (grading using a
set of defined standards)
Learning
Determining the Pass or Cut Scores
After constructing the educational measuring tools (test-based tools,

observation-based tools, and project-based tools),
the next step is to determine how to grade and most importantly, how
to determine the pass or cut scores.
Let us now FOCUS on how to determine the pass or cut scores.

The score at which a student passes or fails is called the cut score or
pass score or passing score.
Minimum Pass Level (MPL) is another common term.
Learning
ROJoson’s Recommended Ways:

Some Preliminaries
• ROJoson’s categorization of educational measuring tools for medical
students consist of TEST-BASED, OBSERVATION-BASED, AND
PROJECT-BASED TOOLS.
• TEST-BASED Tools consist of SELECTIVE OBJECTIVE TESTS and ORAL /
ESSAY TESTS.
Learning

Some Preliminaries
• Scoring and grading systems will be dependent on the types of
educational measuring tools.
• The SELECTIVE OBJECTIVE TESTS will be managed differently from
the ORAL / ESSAY TESTS, OBSERVATION-BASED and PROJECT-BASED
Tools.
• The ORAL / ESSAY Tools, OBSERVATION-BASED and PROJECT-BASED
Tools will be managed similarly.
Learning

Some Preliminaries
Policies and Procedures on Scoring and Grading System
• The medical school should come out with policies and procedures
on its scoring and grading systems.
• The students should be informed of these policies and procedures
so that there will be NO misunderstanding.
Learning

• The passing score for an educational measuring tool should be set in

accordance with the purpose of the exam (formative, summative,
knowledge, skills, attitude, etc.)
• The passing score set for an educational measuring tool must be
defensible, acceptable, and credible (not arbitrary).
Learning

• The process of determining an effective and valid cut score for an

educational measuring tool is referred to as standard setting.
• This requires a properly selected panel of expert judges and a
suitable standard setting method, which best fits the purpose of the
educational measuring tool.
Learning
There are a number of methods for determining the pass or cut scores,
some of which are mentioned below:
• Using norm-referenced system (or grading on a curve): the teacher
computes for the mean and standard deviations and then decides
whether the pass or cut score (or minimum pass level) is 0.5 or 1
deviations below the mean. Some would just use the mean as the
pass or cut score. Critique: Designed to only pass the “best of the
best”; relative standard as students of medicine need to be judged
on their mastery of their professional content.
Learning

There are a number of methods for determining the pass or cut scores,
some of which are mentioned below:
• Standard setting using criterion-referenced system: process of
evaluating (and grading) the learning of students against a set of
pre-specified qualities or criteria, without reference to the
achievement of others. The pre-specified qualities or criteria are
what students have to do during assessment in order to
demonstrate that they have achieved the learning outcomes.
Learning

There are a number of methods for standard setting using criterion-

referenced system.
• Institution’s policy on absolute pass score: commonly, 70% or 75%
(just make sure that the tools are properly constructed and valid with
clear criteria)
Learning
There are a number of methods for standard setting, some of which

are mentioned below:
• Institution’s policy on absolute pass score: commonly, 70% or 75%
• If flexibility is allowed, the teacher or panel of teachers can decide
on the pass or cut score based on importance of content, level of
expectations on the students’ learning and difficulty or ease of
exam. Thus, some tests may have a pass or cut score of 65%; some,
even 80%. This arrangement on flexibility will be defensible,
acceptable, and credible.
Learning
There are a number of methods for standard setting, some of which

are mentioned below:
FOR OBJECTIVE WRITTEN TESTS (MULTIPLE CHOICE QUESTIONS):
• Nedelsky Method
• Angoff Method
• Ebel Method
*Among the three, Angoff Method is most commonly used (and will be
described below).
Learning

The Angoff method relies on subject-matter experts (SMEs) who

examine the content of each test question (item) and then predict how
many minimally-qualified candidates would answer the item correctly.
The average of the judges’ predictions for a test question becomes
its predicted difficulty. The sum of the predicted difficulty values for
each item averaged across the judges and items on a test is the
recommended Angoff cut score (or minimum pass level – MPL).
MPL = Sum of MPL of all items /no. of items x 100
Learning
The Angoff method

The experts assess each question and determine the percentage of 100
minimally competent students who will be able to answer the item
correctly.
The MPL of the test is computed as follows:
MPL = Sum of MPL of all items /no. of items x 100
Learning

The Angoff method

Questions Points (one point MPL by experts (% will answer
each) correctly)
1 1 0.6
2 1 0.7
3 1 0.9
↓ 1 1.0
100 1 0.8
100 Points Sum (example = 65.5)
MPL for whole test = (65.5 / 100) x 100 = 65.5
Learning
Criterion-referenced System - Rubrics for ORAL / ESSAY TESTS
Oral / No correct With correct With correct With 100%

Essay answers answers but answers more than complete
Questions whatsoever (0 less than 50% 50% but less than correct
point) complete (1 100% complete (2 answers (3
point) points) points)
Q1
Q2
Q3
Q4
Q5
Learning
Criterion-referenced System – Rubrics for OBSERVATION and
PROJECT-BASED TOOLS
Items / Level of Level of Level of Level of Level of
Criteria performance performance performance performance performance
with with with with with
DESCRIPTOR DESCRIPTOR DESCRIPTOR DESCRIPTOR DESCRIPTOR
and and and and and
Points or Score Points or Score Points or Score Points or Score Points or Score
Assigned Assigned Assigned Assigned Assigned
1
2
3
4
5
Learning
Examples of Scoring and Grading Sheets

being used by different institutions
Learning
Letter Grade System
Grade Point Letter Grade

Equivalence Equivalence % Equivalence
4.0 A 92–100%
3.5 B+ 87–91%
a
3.0 B 83–86%
2.5 C+ 79–82%
2.0 C 75–78%
1.0 D 70–74%
0.0 F below 70%
Learning
Grade Point System
Grade Point Equivalence Equivalence Description
1.00 96–100% Excellent
1.25 94–95% Very Good
1.50 92–93% Very Good
1.75 89–91% Good
2.00 87–88% Good
2.25 84–86% Good
2.50 82–83% Fair
2.75 79–81% Fair
3.00 75–78% Pass
5.00 Below 75% Failure
INC Incomplete
Learning
Latin Honors Grade Point Equivalence

Equivalence Range
Summa Cum Laude 1.00–1.50 92% and above
Magna Cum Laude 1.51–1.85 88.5% to 91.9%
Cum Laude 1.86–2.10 86% to 88.4%
Grade Point Equivalence

Latin Honors Range
Summa Cum Laude 3.87–4.00
Magna Cum Laude 3.70–3.86
Cum Laude 3.50–3.69
Learning
Philippine Physician Licensure Examination – Sample Complete Examination Report
Subjects Ratings
1 Biochemistry 84
2 Anatomy and Histology 86
3 Microbiology 85
4 Physiology 89
5 Legal Medicine, Ethics, and Medical Jurisprudence 88
6 Pathology 81
7 Pharmacology and Therapeutics 82
8
Surgery and Ophthalmology, Otorhinolaryngology and Rhinology 85
9 Medicine 82
10 Obstetrics and Gynecology 82
11 Pediatrics and Nutrition 93
12 Preventive Medicine and Public Health 81
GENERAL AVERAGE RATING 84.83 (Ʃ/12)
REMARKS PASSED
To pass the examination, an examinee must obtain a general average rating of seventy five
percent (75%) without a grade lower than fifty percent (50%) in any subject.
Learning
Philippine Physician Licensure Examination – Sample Complete Examination Report
Subjects Ratings
1 Biochemistry 84
2 Anatomy and Histology 86
3 Microbiology 85
4 Physiology Note the use of the word 89
5 Legal Medicine, Ethics, and Medical
“rating,“ sameJurisprudence
as “grade”. 88
6 Pathology 81
7
General
Pharmacology and Therapeutics
average rating = general 82
8 average
Surgery and Ophthalmology, grade

Otorhinolaryngology and Rhinology 85
9 Medicine 82
10 Obstetrics and Gynecology 82
11 Pediatrics and Nutrition 93
12 Preventive Medicine and Public Health 81
GENERAL AVERAGE RATING 84.83 (Ʃ/12)
REMARKS PASSED
To pass the examination, an examinee must obtain a general average rating of seventy five
percent (75%) without a grade lower than fifty percent (50%) in any subject.
Learning
Educational Measuring Tools for Medical Students’ Learning and

Performance - Scoring and Grading
ROJoson Thoughts, Perceptions, Opinions and Recommendations
There is a need to measure the learning and performance of medical

students after the course of study and thereafter, to decide whether or
not to certify them as graduates who are qualified to practice
medicine in the Philippines (that is, after passing the Philippines
Physician Licensure Exam).
Certifying the students to graduate means they have acquired the

needed competencies to practice as basic-generalist physicians in the
Philippines.
Learning

Certifying the students to have acquired the needed competencies to

practice as basic-generalist physicians in the Philippines means they
have mastered the professional contents of the medical curriculum.
Professional contents cover the cognitive knowledge, skills and
attitude domains required of basic-generalist physicians, not just
passing written tests.
Learning

The educational measuring tools for medical students’ learning and

performance are not perfect and probably will never be perfect,
whatever they are.
Despite this imperfection, they should still be administered because

there is a need to measure, to evaluate and to determine who are
eligible to be basic-generalist physicians.
Learning

However, the tools should be CREDIBLE and VALID.
They should not ARBITRARY and not using the norm-referenced

scoring and grading (using mean or average as a passing score, so
called grade on a curve).
Criterion-referenced scoring and grading must be used.

Learning

Even the criterion-referenced scoring and grading are not perfect. But
at least, students are assessed and evaluated against a set of criteria or
standards on mastery of the professional contents of the medical
curriculum.
Learning

The standard setting processes utilized for such purposes generate

absolute standards, in contrast to relative standards where students
are judged against each other as is being done in norm-referenced
system.
These standards are considered absolute, because they are expressed
in terms of how much content the students need to know.
Learning

However, to repeat, there is no “gold standard” for setting passing

scores, in the standard setting in the criterion-referenced system.
Usually the choice of the standard setting method is based on the
available resources and the practical realities of the educational
environment. As such, it is critical to document all procedures used in
establishing the passing standard, especially in a language the school is
willing to make public.
Learning

Every effort should be exerted to use one of the standard setting

approaches described in the literature so as to make the passing
scores defensible and meaningful.
Also important is the continual improvement of the scoring and

grading systems. There should be evaluation and review of the
systems at planned intervals.
Learning

One specific activity recommended in the continual evaluation and

review of the scoring and grading systems is the so-called ITEM
ANALYSIS.
Item analysis is a process which examines student responses to
individual test items (questions) in order to assess the quality of those
items and of the test as a whole.
Learning

Item analysis typically focuses on four major pieces of information:

item difficulty, item discrimination, item distractors and response
frequency.
Learning

Item analysis is especially valuable in improving items which will be

used again in later tests, but it can also be used to eliminate
ambiguous or misleading items in a single test administration.
In addition, item analysis is valuable for increasing instructors’ skills in
test construction and identifying specific areas of course content which
need greater emphasis or clarity.
Learning

Last words:
An exam standard is ultimately just a recommendation.
The final evaluation of a student’s performance lies with educators
and educational institutions.
No exam is perfect: an educator will always know more about a
student than any exam ever could.
When considering a student’s performance on a summative exam,
educators should take into account the body of evidence that student
has built up during the duration of the academic career.
Learning


• Define scoring.
• Define grading.
• Define points.
• Define raw scores.
• Give examples of translated scores from raw scores.
Learning


• Differentiate norm-referenced and criterion-referenced system of
grading.
• Discuss the different methods on how to determine the pass or cut
score.
• Discuss the limitations of scoring and grading systems in a medical
school.
Learning


• Explain why medical schools should go for a criterion-referenced
systems of scoring and grading.
• Discuss the limitations of scoring and grading systems in medical
schools.
• Discuss the usefulness of item analysis.
Learning
Tools in Measuring Program and Learning Outcomes
of Medical Students

Subtitle: Tools in Measuring Knowledge, Skill and Attitude of Medical
Students - A Guide in Scoring and Grading (Part 3)

Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tools in Measuring Program and Learning Outcomes - A Guide in Scoring and Grading (Part 3)

Uploaded by

Copyright:

Available Formats

ROJoson OCIL on Medical Education

OCIL = Online Collaborative and Interactive

Tools in Measuring Program and Learning Outcomes of Medical

Recommended Prior Study:

Tools in Measuring Program and Learning Outcomes of Medical

At the end of the session, the student must be able to:

Tools in Measuring Program and Learning Outcomes of Medical

At the end of the session, the student must be able to:

Tools in Measuring Program and Learning Outcomes of Medical

At the end of the session, the student must be able to:

Assumptions on activities that have been undertaken:

Scoring Management System refers to a set of policies, processes and

SCORING vs GRADING MANAGEMENT SYSTEM

Grading Management System refers to a set of policies, processes and

SCORING vs GRADING MANAGEMENT SYSTEM

In terms of implementation of actual processes, there is a proper

POINTS, MARKS, SCORES, RATINGS, GRADES etc.

The total point gotten by the students in each educational measuring

POINTS on a test with a numeric value on each question or item

SCORES = TOTAL POINTS

If above is the whole test, the total point of 15 of Student 1 is also

RAW SCORES – scores without any sort of adjustment or

The score at which a student passes or fails is called the pass or cut

Translated Scores – Example

Raw Scores Range of Scores Grade Point Letter Grade

Translated Scores – Example

Translated Scores – Example

Grades can be assigned as letters (usually A through F), as a range (for

Grades in medical schools are usually in numeric units.

Norm-referenced means referencing how the score of a student

After constructing the educational measuring tools (test-based tools,

Let us now FOCUS on how to determine the pass or cut scores.

Determining the Pass or Cut Scores

ROJoson’s Recommended Ways:

ROJoson’s Recommended Ways:

Determining the Pass or Cut Scores

ROJoson’s Recommended Ways:

Determining the Pass or Cut Scores

• The passing score for an educational measuring tool should be set in

Determining the Pass or Cut Scores

• The process of determining an effective and valid cut score for an

Determining the Pass or Cut Scores

Determining the Pass or Cut Scores

There are a number of methods for standard setting using criterion-

There are a number of methods for standard setting, some of which

There are a number of methods for standard setting, some of which

Determining the Pass or Cut Scores

The Angoff method relies on subject-matter experts (SMEs) who

The Angoff method

Determining the Pass or Cut Scores

The Angoff method

Oral / No correct With correct With correct With 100%

Examples of Scoring and Grading Sheets

Letter Grade System

Grade Point Letter Grade

Latin Honors Grade Point Equivalence

Grade Point Equivalence

Educational Measuring Tools for Medical Students’ Learning and

There is a need to measure the learning and performance of medical

Certifying the students to graduate means they have acquired the

Educational Measuring Tools for Medical Students’ Learning and

Certifying the students to have acquired the needed competencies to

Educational Measuring Tools for Medical Students’ Learning and