Measurement and

Quantitative Methods
Session 2

Marieliz (M) Maines

Research Associate, IPA
INTRODUCTION
PRE-TEST

Pre-test

https://tinyurl.com/scmeasure-pre
OUTLINE

Outline
1. Conceptual principles
a. Measurement
b. Link to TOC
c. What makes a good measurement?

2. Sources of data
a. Response process

3. Minimizing error and bias

a. Measurement error
Measurement and Quantitative methods

Measurement
Why is it valuable to understand measurement?
How do you measure access to justice?
CONCEPTUAL PRINCIPLES

What is measurement?
The main concept being investigated. A construct is sometimes
Construct abstract and can have multiple definitions.
(E.g., learning, health)

A way to measure a given construct.

Indicator (E.g., test scores for education, blood pressure for health)

Decisions about collection method

Data collection (E.g., Point-in-Time Count)

What we use to measure our indicators.

Data
CONCEPTUAL PRINCIPLES

Linking to the Theory of Change

Inputs/Activities Outputs Outcomes Goal

Indicator Indicator Indicator Indicator

Data collection Data collection Data collection Data collection

Assumptions Assumptions Assumptions

Indicator Indicator Indicator

Data collection Data collection Data collection

CONCEPTUAL PRINCIPLES

Defining and operationalizing TOC concepts

NEED ASSESSMENT: Girl drop-out rate is GREATER than Boy drop-out rate

Inputs/Activities Outputs Intermediary Outcome Outcomes Goal

Gender sensitive Teachers are more More girl-friendly Increase in girls
Gender training attitude, activities, gender-sensitive school attendance and
and curriculum and supportive environment school completion

# Schools that % girls enthusiastic % of school-aged

Proportion of Attitudes towards implemented about going to girls attending
teachers trained girls in school gender-sensitive school school
practice % girls who find % of school-aged
school a positive girls completing
learning experience school
% parents trust
school to look after
girls

Training invite, School survey, School enrolment

School survey Direct observation
Attendance sheet Household survey data & test scores
CONCEPTUAL PRINCIPLES

What makes a good measure?

Construct
How well does the indicator map
Validity to the construct we are trying to
measure?
Indicator

Is the indicator measured in a way

that is consistent and precise?
Data collection Reliability
Would we get the same data if we
measured several times?

Data
 Discuss:
Is this a good measurement for the elimination of
Violence against Children:

Number of reported abuse cases for children

(Source: Administrative Data by PNP)
 Brainstorm:
How do we measure whether we are improving the
quality in case disposition ?

How do we measure whether we are improving the

efficiency in case disposition ?
CONCEPTUAL PRINCIPLES

How do you measure access to justice?

CONCEPTUAL PRINCIPLES

Validity

Validity (a.k.a. accuracy or unbiasedness)

LOW HIGH

“Biased” “Unbiased”
CONCEPTUAL PRINCIPLES

Reliability

LOW

“Noisy”
Reliability
(a.k.a. precision)

HIGH

“Precise”
CONCEPTUAL PRINCIPLES

SCENARIO: A surveyor doesn’t follow

the exact phrase of the question.
SURVEYOR: “You don’t feel secure in this
www. MENTI.com neighborhood, right?”

6994 9548 RESPONDENT: “Well, sometimes yes, sometimes no.”

SURVEYOR: “So, it’s more of a yes, right?”
RESPONDENT: “Mmm…well”
SURVEYOR: “Ok, thanks.”
*coded “YES”

Is this introducing noise or bias?

CONCEPTUAL PRINCIPLES

Validity
(a.k.a. accuracy or unbiasedness)
LOW HIGH

“Biased” “Unbiased”

LOW

“Noisy”
Reliability
(a.k.a. precision)

HIGH

“Precise”
CONCEPTUAL PRINCIPLES

Validity
(a.k.a. accuracy or unbiasedness)
LOW HIGH

“Biased” “Unbiased”

LOW

“Noisy”
Reliability
(a.k.a. precision)

HIGH

“Precise”
CONCEPTUAL PRINCIPLES

Validity
(a.k.a. accuracy or unbiasedness)
LOW HIGH

“Biased” “Unbiased”

LOW

“Noisy”
Reliability
(a.k.a. precision)

HIGH

“Precise”
CONCEPTUAL PRINCIPLES

Which is worse?
Validity
(a.k.a. accuracy or unbiasedness)

www. MENTI.com LOW HIGH

“Biased” “Unbiased”
6994 9548
LOW

Reliability “Noisy”
(a.k.a. precision)

HIGH
“Precise”
Measurement and Quantitative methods

Sources of data
SOURCES OF DATA

Where can we get data?

Primary data Secondary data
Collected for specific study From existing sources
● Surveys ● Administrative data
● Focus groups ● Sensor data
● Observation ● Web traffic data
● Games ● Existing survey data
● Audits ● School records
SOURCES OF DATA
Pros and cons of Administrative data
● Costs
● Reduce participant burden
● Near universal coverage
● Accuracy
● Long-term availability
● Access
● Occasionally format of data
● No say in data collection and
processing
● Various biases
SOURCES OF DATA

Response process

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

How many times did you

consume rice this month?
a. 0
b. 1
c. 2-3
d. 4-6
e. 7-12
?
f. 13+
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

How many times did you

consume rice this month?
a. 0
b. 1
c. 2-3
d. 4-6
e. 7-12
f. 13+
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

How many times did you

consume rice this month?
a. 0
b. 1
c. 2-3
d. 4-6
e. 7-12
f. 13+
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

How many times did you a. 0

consume rice this month? b. 1
a. 0 c. 2-3
b. 1 d. 4-6
c. 2-3 e. 7-12
d. 4-6 f. 13+
e. 7-12
f. 13+
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

User experience in eCourt 2.0

Clarity of procedures

__________________________
__________________________
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

User experience in eCourt 2.0 My internet connection

is slow. Should this
Clarity of procedures affect my rating for this
question? Should I
focus just on the
instructions?

__________________________
__________________________
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

User experience in eCourt 2.0

At the start, the
procedures are
Clarity of procedures overwhelming…
But now - after some
time - I’m used to it.

__________________________
__________________________
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

User experience in eCourt 2.0 Procedures for eCourt

Vs procedures for current

Clarity of procedures way of retrieving cases

Vs procedures for baking

Chocolate Banana Cake

__________________________
__________________________
SOURCES OF DATA

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

User experience in eCourt 2.0

Clarity of procedures

__________________________
__________________________
Measurement and Quantitative methods

Minimizing error and bias

MINIMIZING ERROR & BIAS

What makes a good measure?

Construct

Validity error
Indicator

Data collection Measurement error

Data
MINIMIZING ERROR & BIAS

What makes a good measure?

Construct

Indicator

Data collection Good measurement

Data Good data

MINIMIZING ERROR & BIAS

Understanding measurement error

Measurement Error – occurs when the response provided by a
respondent differs from the real or true value.

Errors may be random or systematic

1. Random Error – caused by factors that randomly affect
measurement of the variable across the sample. e.g., mood, fatigue

2. Systematic Error – not determined by chance but is an inaccuracy

inherent in the whole sample → Bias
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

There are key things to keep in mind at each stage

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

Respondents need to understand your question to answer it.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Incomprehensible/unanswerable question

Example:
Q. How many times did you consume rice this month?
● Times → What is “time”? Is it a meal, a serving?
● Consume → Does “consuming” only mean eating?
● Rice → Does this include rice noodles, rice milk, other forms of rice?
● This month → If the survey was done during the 2nd day of the month, does that
only include the past 2 days? Or should it be a full calendar month?

Look at each word.

Brainstorm alternate (unintended) meanings.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Incomprehensible/unanswerable question

Example:
Q. How many times did you consume rice this month?
In the past 7 days, how many days have you eaten a bowl of rice?
A: ___ (from 0 to 7)

Look at each word.

Brainstorm alternate (unintended) meanings.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Incomprehensible/unanswerable question

Examples:
Q. Are you digitally empowered by e-Court v 2.0?

Q. What is the depreciation rate of your vehicle?

Q. Have you purchased household furniture in the last week?

Use vignettes for complex concepts.

Avoid jargons.
Provide definitions.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Context of the survey

Examples:

We are doing a survey on the ICT infrastructure of the judicial system

Q. How would you rate the clarity of procedures of e-Court v 2.0?
vs.
We are doing a survey on user-experience with e-Court v 2.0
Q. How would you rate the clarity of procedures of e-Court v 2.0?

Prevent unintended inferences about definitions

introduced through introductions, visuals, question order, imagery.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Negatives

Examples:

Q. Has there no difficulties accessing the system? Yes/No

Q. How many times did you not generate requested reports on time?

- Questions that include negatives can be confusing to the respondent and lead to
misinterpretations.
- Having a negative might throw some people off.

Avoid using negatives whenever possible.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Double-barreled

Example:

Q. Should the SC improve on the slow processing of transactions through e-Court v2?

Avoid double-barreled questions.

Separate into multiple questions.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Presumptions

Example:

Q. Which university did you attend?

Use filters and skip patterns.

Include “Not applicable” option when relevant.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Presumptions

Example:

Q. Which university did you attend?

Q1.1 Did you attend university?

● Yes (proceed to Q1.2)
● No (proceed to Q2)
Q1.2 Which university did you attend?

Use filters and skip patterns.

Include “Not applicable” option when relevant.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

Respondents need to understand your question to answer it.

● Be clear and concise
● Check translations
● Pilot
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

Recall is imperfect.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Recall bias

Example:

Q. How much was your total screen time in the past 4 weeks?

Ask respondents to record information in real-time

(e.g., diaries, log books, save receipts).
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Recall or Salience

Example:

Q. How many times did you buy groceries in the last 6 months?
vs.
Q. Did you buy a TV in the last 6 months?

Know the likely direction of bias.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Time perception

Examples:

Q. Did you purchase a computer or other electronic (worth over Php 10,000)
in the past 12 months?

Ask, “since the last time I visited you, have you…?”

Use specific dates.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

Recall is imperfect.
● Use diaries or other records
● Consider tradeoffs between asking about recent or typical behaviors
● Give well-defined boundaries for your questions
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

You can influence answers.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Anchoring bias

Example:

Q. Under the CTG, judgment should be promulgated within 90 days.

How many days does promulgation in your court usually take?

Avoid adding anchors to your questions.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Framing

Example:

Q. Hitting your child to discipline them is illegal in your country. Have you ever hit your
child to discipline them?
vs.
Q. Many people think that physically disciplining their child is an effective way to
teach them how to behave. Have you ever hit your child to discipline them?

Try to ask questions neutrally.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Priming

Example:

Q1. Did you have to pay to file a case?

Q2. How much did you spend on legal fees?
Q3. On top of legal fees, are there other expenses that you incurred?
Q4. How do you think can the public have better access to justice?

Consider question order carefully.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

You can influence answers.

● Phrase questions carefully to avoid influencing answers
● Use neutral language
● Avoid anchors or priming, which will make certain answers more salient
● Consider how previous questions will influence answers
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

Consider whether respondents want to report the answer, able to pick the correct
answer, and whether the right people are answering or getting asked at all.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Social desirability bias

Examples:

Q. How many times have you taken illegal drugs?

Q. Did you cheat - in any way - on the 2021 national exam?

- Tendency of respondents to answer questions in a manner that is seen as

favorable, i.e., emphasize strengths, hide flaws, or avoid stigma

Ask indirectly.
Ensure privacy/anonymity.
Make sensitive questions less specific.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Reporting bias

Example:

Application for scholarship

Q. What is your child’s school attendance rate?

- If a certain level of school attendance is required to qualify for a scholarship then

respondents may overestimate the amount their children attended school.

Use proxy measures or direct observation.

Make use of other sources of information.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Completeness of answers

Example:
Q. What is your highest level of education?
● Some primary education
● Graduated primary education
● Some secondary education
● Graduated secondary education
● Some tertiary
● Graduated tertiary education

Make sure response categories are exhaustive.

Include “Don’t know”, “Refuse to response”, and “Other (specify)” where relevant.
MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Overlapping answer choices

Example:

Q. How many days did you go to the office last week?

● Did not work onsite
● 1-3 days
● 3-4 days
● 5 days or more

Make sure that categories are mutually exclusive.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

MEASUREMENT ERROR: Construction of answer choices

Example:
Q. What would you like for dinner, we have…
● Chicken with rice
● Steak with potatoes
● Macaroni and cheese
● Trout with greens
● Salmon with rice
● Beans with rice
● Etc.

Consider randomizing response order.

MINIMIZING ERROR & BIAS

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an

of the question information estimation answer

Consider whether respondents want to report the answer, able to pick the correct
answer, and whether the right people are answering or getting asked at all.
● Understand incentives and sensitivities
○ Consider social desirability, surveyor characteristics
○ Ensure privacy
○ Choose questions/variables less susceptible to bias
○ Consider context in which data are collected
● Write good answer options
○ Exhaustive
○ Mutually exclusive
○ Not overwhelming
MINIMIZING ERROR & BIAS

Many things can cause measurement error

1. Comprehension 2. Retrieval of 3. Judgment and 4. Reporting an
of the question information estimation answer

- Translation & - Memory - Anchoring - Incentives

interpretation - Vividness / salience - Framing / priming - Sensitivities
– Back-translation – Social desirability
– Local dialects – Surveyor
- Ambiguity characteristics
- Jargon - Response options
- Double negatives – Exhaustive
– Mutually exclusive
- How to record is clear
MINIMIZING ERROR & BIAS

Q: We have learned about the various

potential errors and biases in every stage
of the response process. It is not difficult
www. MENTI.com to create a good measure now than
before because I actively participated in
6167 4878 the discussion, correct?
a) Yes
b) Of course
c) Maybe

What measurement error/s is/are present

in this question?
SOURCES OF DATA

www. MENTI.com
6167 4878 What source/s of data do
you expect to be the most
important for your project?
MINIMIZING ERROR & BIAS

What about admin data?

Two questions to ask:

1. Why were these data collected?

a. Were there incentives or opportunities to misreport information?
b. Can you choose variables that are less susceptible to bias?

2. Which individuals are included in the data and which are excluded,
and why?
a. What steps have to occur before appearing in the data?
b. Does the intervention affect reporting of outcomes?
MINIMIZING ERROR & BIAS

What about admin data?

CHALLENGE: Accessing administrative data

Example:

Regulations may limit access to identified data

MOU (Memorandum of Understanding)

Follow data security requirements, etc.
MINIMIZING ERROR & BIAS

What about admin data?

CHALLENGE: Unusable format

Example:

● Handwritten records
● PDF files
● Scanned copies saved as images

Digitize records.
MINIMIZING ERROR & BIAS

What about admin data?

CHALLENGE: Matching to other dataset

Example:

Researcher want to match student responses to a self-reported survey with the

national exam data

Incorporate unique IDs.

In the absence of this, use personal identifying information, e.g., names, address, DOB.
MINIMIZING ERROR & BIAS

What about admin data?

CHALLENGE: Reporting bias

Example:

School overreport attendance to meet policy requirements.

Identify the context in which data were collected.

Were there incentives to misreport information?
MINIMIZING ERROR & BIAS

What about admin data?

CHALLENGE: Selection bias

Example:

Researcher obtains school records from 600 primary schools in the Philippines to
make an assessment on primary education in the Philippines. Some school records
are handwritten while others are digitized. To avoid additional work, she opts to only
include records that were digitized.

Identify data universe: which individuals are included/excluded?

MINIMIZING ERROR & BIAS

What about admin data?

CHALLENGE: Differential response

Example:

An intervention includes video cameras in the classroom to deter teacher tardiness.

At endline, it appears that the number of reported tardiness cases increased in the
treatment group, when in reality it is because the measurement (camera footage) is
more precise.

Ensure that the ability to measure an outcome is not correlated to treatment assignment.
MINIMIZING ERROR & BIAS

SCENARIO: The Department of Health is

trying to combat cervical cancer. They decide
to run a small scale randomized controlled trial
(RCT). The treatment group receives an
www. MENTI.com information campaign encouraging women to
get screened. The control group receives
6167 4878 nothing.

What issue is present when using hospital

records to evaluate the intervention?

A) Recall Bias
B) Reporting Bias
C) Differential Response
D) There are no issues
 Discussion:
SRD

How does the SRD collect data?

 Discussion:
MISO

What data will be collected by the e-Court 2.0?

What should be collected by the e-Court 2.0?
 Discussion:
OCA/CMO

What indicators are most relevant to measure the

success of the unified platform?
CONCLUDING THOUGHTS

Concluding thoughts
● To collect “good” data requires a lot of effort and thought
● Theory of change guides measurement
○ Indicators need to be valid and reliable
● The information received from any survey instrument is as good as the words
used to create it
○ Devote sufficient time to survey design
○ Pay careful attention to the wording of the questions (and options)
○ Try to eliminate systematic biases that can arise
● Take advantage of administrative data
○ Understand the context of how the data was collected
○ Think through incentives and sources of bias
POST-TEST

Post-test

https://tinyurl.com/scmeasure-post
GROUP ACTIVITY

Questionnaire Appraisal
INSTRUCTIONS:
Groups will be provided a workbook for Activity 2.

1. Using the Checklist for for questionnaire appraisal, work through the
Questionnaire - eCourts v1 Satisfaction Survey. Assess block by block and
identify what parts of the questionnaires should be improved (focus specifically
about the kind of measurement errors the questions might be prone to). Consider
also the target respondents of each questionnaire.
2. Appraise the instrument as a whole. Note both the good points and things for
improvement.
3. Pick 2-3 questions or 1 section from each survey and reframe them/it using
approaches discussed in the the 'Measurement and Quantitative Data Collection'
session.
GROUP ACTIVITY

Questionnaire Appraisal
CONTEXT:

The objective of the survey is to learn more about how judicial reforms
(particularly the eCourts system version 1) affect case management in the
courts. It aims to get the insights on efficiency and performance of the
judges and clerk of courts. It also hopes to better understand the needs
and to identify new solutions to improving the target respondents' efficiency
in the environment in which they currently work.
Maraming salamat po!