Measurement,

Validity & Reliability

 Review
◼What do we know about social science research so far?
✔ Use scientific methods to investigate relationships between variables.
✔ Theory gives us a way to filter information and organize our thoughts or ideas to try
to explain predictable patterns.
✔ Develop research questions and hypotheses based on our existing theory (i.e.,
deductive) or based on data or observations (i.e., inductive) to construct
explanations.

A Good Hypothesis is
✔Simple
✔Specific
✔Formed a priori

2
Measurement

◼ Theories specify how concepts relate to one

another, but how do we measure construct
(variables)?

Operationalization

◼ Process of defining the operations or steps to

take that will measure a construct (variable)
to make a concept an empirical reality.
 Operationalization

Construct Operationalization
(VARIABLE) Income + Education + Occupational Prestige
Socioeconomic Status
# of hrs watching TV per weekday
Television Exposure
GPA
Academic Achievement
# of times using condom / # of times had sex
Safe Sex
Measurement is not perfect

◼ ALL data collection is incomplete to some

extent

Measurement is always incomplete and inaccurate .

It is nearly impossible to measure, count, observe, or
describe a construct completely.

◼ Goal: Create best measures possible to

approximate the real thing.
If measurement is not perfect,
why is it important?

◼ Your data and ability to

make claims (i.e.,causal
inference) is only as good
as your measures.

◼ “Garbage in, garbage

out”
How do we define measurement?

◼ Measurement = assignment of categories, ranking or values to

observations according to specific rules

◼ Measurement Assumptions
▪ We generally assume (hope?) that the metric or categorical assignments we
are making actually reflect meaningful differences in the properties of the
observations.
▪ When we measure, we are acting on an expectation that the measurements
(assignments) we are making are "meaningful" in some way.

◼ Determining the relevant measures (that is, ones that accurately

captures the appropriate differences in the observations so
classified) is not easy.
Meaningful Assignments

◼ Gender?
◼ Zipcode of Residence ?
◼ Left, Right Handedness, Ambidexterity?

Not all assignments or classification categories are

meaningful in all circumstances, or may be relevant for
quite all purposes.
 Variable→Operationalize→Measure

Income+

VARIABLE Operationalize Measure (Questions)

Socioeconomic Income+ Education + Occup.
Status Prestige
TV Exposure # of hrs watching TV per
weekday
Academic GPA
Achievement
Safe Sex # of times using
condom / # of times had
sex
Levels of Measurement

◼ Nominal Level Measurement

◼ Ordinal Level Measurement
◼ Interval Level Measurement
◼ Ratio Level Measurement
Nominal Level Measurement

◼ Numbers used as ways to identify or name categories

◼ Numbers do not indicate degrees of a variable but

simple groupings of variables, no order

◼ The attributes that we are assigning value to must be

mutually exclusive and exhaustive

◼ Examples: race, political party, type of media, gender,

experimental versus control group
Ordinal Level of Measurement

◼ Uses rank order to determine differences

◼ Measures whether items are “greater than” or “less

than” other items

◼ “How often do you watch Real Housewives?”

▪ Never, Rarely, Sometimes, Often

◼ Identifies relations of measured items to each other,

but not how much difference between levels
▪ What is this distance between never and rarely?
Interval Level Measurement
◼ Allows meaningful arithmetic (e.g., sums, differences) to
be applied

◼ Measures whether items are “greater than” or “less than”

other items, with equal intervals between numbers

◼ No known absolute zero point

◼ Scales are often treated this way

◼ Strongly Agree -------- Strongly Disagree
◼ 7--------------------------------------1
Ratio Level Measurement

◼ Distances between measured items are

identified using a quantitative scale that
includes an “absolute zero”
▪ Absolute Zero = when property measured is
completely absent

◼ Allows meaningful arithmetic to be applied

◼ Examples: age, Hours watched television, # of

websites visited (0,1,2,3, …)
Let’s Review: Measurement Levels
Continuous Data

Ratio Absolute zero point

Interval Distance between points is

meaningful

Ordinal Attributes can be ordered

Nominal Objects sorted into categories

according to an attribute

Categorical Data
Let’s Review: Measurement Levels
 Interval vs. Ratio
INTERVAL RATIO

DEFINED AS equidistant points Equidistant

VARIABLES between each of points & AN
THAT. . . the scale ABSOLUTE
elements
ZERO as a
starting point

EXAMPLES Scales, Age, # of

Temperature (F) Students, SAT
scores,
What about measurement
error?
Two types of error in measures:

1. Random or nonsystematic error (“noise”, “static”)

▪ Nonsystematic random fluctuation is measurement

unreliability.

2. Non-random or systematic errors (“bias”)

▪ Issue of validity.

18
Good Measures are 1) Reliable

◼ Reliability
▪ The internal consistency of a measure
▪ Test yields consistent scores when the variable
being measured is not changing

◼ Reliability is a test of the stability of a measure

▪ If a measure is unstable, it cannot produce an
accurate report of the data.
▪ Always measures the same thing every time.
Good Measures are 2) Valid

◼ Validity
▪ Degree to which a measure actually measures
what it claimed.
Reliability & Validity

◼ Valid measures are

Reliable typically.

BUT

◼ A reliable measure
is not necessarily a
valid one.
How do I choose my measures?
◼ Use measures that have been used before and whose reliability
and validity have been established in other contexts.

◼ What if I have to develop my own measures?

▪ Conduct a pretest in which you use the measure with a small sample
and check its reliability.

▪ Ask yourself if it makes sense? Do people understand your question

the way you do?

▪ Use multiple measures to measure the same construct.

Measurement & (Mis)Measurement
Race
What is wrong with
this graphic?
Race in America
1. Racial categories developed as a social system
to “other” people as not white, 1 in 7 people
identify as other race.
2. Social Construct, Political!
3. Culture Bound & Time Bound
4. Real Consequences on life outcomes
5. Measurement of Race has changed for social
and political reasons
“There is no uniform practice regarding the use of race as a study variable
and little to no expectation that authors examine racism as a cause of residual
health inequities among racial groups.”-
Census Measures Fun Facts
Until 1860 - Slave was a category to increase political power of slaveholding states,
but slaves were less than a full human. (enslaved Africans counted as ⅗ of a person)

American Indian added in 1860 - when Cali, Minnesota and Oregon wanted to
increase their seats in congress.

1850 - 1870 : New Mexican Pueblo community counted as “non-white”, “indian”,

“copper”. Size of population fluctuated between 44K (1860) , 26K, (1870) and 66K
(1880)

1860 - Chinese Americans in Cali only, 1870 nationally

1930 - Filipino , Mexican (first and last time included)

1940 - Mexicans classified as white, and 1930 census data was reclassified

2000 - grew from 3 to 16 categories and people could identify as more than one

https://www.sciencenews.org/article/census-2020-race-history
1 in 7 people chose “Some Other Race”.
When people measure race. . .

What do they think they are measuring

vs.
What are they measuring?
When people measure race. . .

What do they think they are measuring vs.

What are they measuring?
● Genetics
● Biology ( eugenics)
● Culture
● *the influence of being identified in certain racial group (nominal
variable)
● Racism, bias (interpersonal, internalized, structural)
● Geography
● Acculturation
● “Weathering effect” consequence of long term exposure to social and
environment stressors that result in chronic inflammation and subsequent
disease
proposed changes
Difficult but can’t be ignored
◼ Focusing on race as an individual factor lets us IGNORE solutions to
challenges that are based on systematic and/or structural racism.

◼ Individual understandings remove our chance to advocate for social

and structural change

◼ Race is a immutable factor - you can’t change it but you can blame it!

◼ Pew Research :
https://www.pewresearch.org/interactives/racial-and-ethnic-gaps-in-th
e-u-s-persist-on-key-demographic-indicators/

◼ WE ALWAYS MEASURE RACE, RARELY EXPERIENCED

RACISM, BIAS, ETC . . . . why?
How has race been measured . . .
BRFSS , 2001
Which one or more of the following would you say is your race? [Mark all that apply]

responses1=White 2=Black or African American 3=Asian 4=Native Hawaiian or Other Pacific

Islander 5=American Indian, Alaska Native 6=Other: [specify]__________ 8=No additional
choices 7=DK/NS 9=Refused

BRFSS 2018
questionWhich one or more of the following would you say is your race?

responses10=White 20=Black or African American 30=American Indian or Alaska Native 40=Asian

41=Asian Indian 42=Chinese 43=Filipino 44=Japanese 45=Korean 46=Vietnamese 47=Other
Asian 50=Pacific Islander 51=Native Hawaiian 52=Guamanian or Chamorro 53=Samoan 54=Other
Pacific Islander 60=Other=Multiple responses 88=No additional choices 77=Don’t know/Not Sure
99=Refused

2020 CENSUS: https://2020census.gov/en/about-questions/2020-census-questions-race.html

How to be a better researcher. . . .

◼ Define race during the experimental design, and specify the reason for its use in the study. Such
definitions should be couched within a sociopolitical framework, not a biological one, that explicitly reviews all
relevant social, environmental, and structural factors for which race may serve as a proxy measure. For the
reader, these additional details enable careful interpretation of study results and implications. But for authors, it
engenders critical thinking about racial constructs that prevent the reification of race as a biological entity.

◼ Name racism, identify the form (interpersonal, institutional, or internalized), the mechanism by which it
may be operating, and other intersecting forms of oppression (such as based on sex, sexual orientation,
age, regionality, nationality, religion, or income) that may compound its effects . A
critical race theory framework lends authors a vocabulary for discussing racism and its potential relationship to
the study’s findings. And naming racism explicitly helps authors avoid incorrectly assigning race as a risk factor,
when racism is the risk factor for racially disparate outcomes.

◼ Never offer genetic interpretations of race because such suppositions are not grounded in science. If
race and genetics are being expressed jointly, painstakingly delineate the intended implication.

◼ Solicit community/population input. Use community review boards or form patient panels to ensure the
outcomes of research reflect the priorities of the populations studied.

◼ Identify the stakes. “All policy is health policy,” and all research on racial health inequities has implications for
broader public policy. Inform readers of these potential applications.

◼ Cite the experts, particularly scholars of color whose work forms the basis of the field’s knowledge on racism
and its effects.
Measure “Race”
◼ Socially Assigned Race
◼ Social Environment
◼ Family History/nationality/ lineage
◼ Cultural difference/acculturation
◼ Length of time in US
Measuring Structural Racism
Goal - To develop a valid & reliable measure
of structural racism using publically
available data?
 Measuring Structural Racism
Results: USed publically available
BRFSS data focused on Black/White
differences only. County structural racism
was associated was associated with lower
BMI in whites and higher BMI in blacks.
Conclusion: The results confirm structural
racism as a latent construct and demonstrate
that structural racism can be measured in U.S.
counties using publicly available data with
methods offering a strong conceptual
underpinning and content validity.
Next Week

◼ Topic
▪ Research Design, Ethics & Sampling

Individual Assignment 1: Measuring Ourselves : Due Friday 2/9