GCH 380: Week 10

Validity
Seeking the “Right Answer”
• “Essentially, all models are wrong, but some are
useful” – George E.P. Box
• No study is perfect, but the ones that have a better
design are more useful than those with a bad design
Validity, revisited
• A valid measure is one that measures what it is
intended to measure
• A valid study is one that the findings truly represent
the phenomenon you are trying to study
• We will discuss different domains of validity– Internal and
External
Internal Validity
• Is the extent to which the observed results represent
the ”truth” among our population and are not due to
methodological errors
• After internal validity of a study is assessed, we can
judge its external validity – if there is low internal
validity, no need to think about external validity
External Validity
• Generalizability (applicability of study results to
different groups and setting)
 • Threats compromise our confidence in

Internal saying that a causal relationship exists

between the independent and dependent
variables
Validity • Well-conducted RCTs have high internal
validity

• Threats compromise our confidence in

External stating whether the study's results are

applicable to other groups
• For example, RCTs do not always have high
Validity external validity, dependent on how you
recruit participants
Internal Validity
A low degree of
internal validity,
conclude little or no
evidence of causality

A high degree of
internal validity, we
will consider the
results valid
Internal Validity
• The extent to which a causal relationship is warranted
• A measure of how strong the study is
• The extent to which systematic error (bias) is
minimized during all stages of the study
• Systematic error vs. random error
• Whether the observed results are attributable to the
exposure
• Less confounding = greater internal validity
Factors that affect internal validity
• Study design
• Study population
• Selection bias
• Comparison group
• Retention
• Data collection methods/process
 Review
Studies
Study Design Experimental
Studies

Cohort Studies

Case-Control Studies

Cross-Sectional studies

Ecological Studies
Case Series/Case Report
Ideas, Opinions, Editorials, Anecdotes, Studies
on Non-Human Animals
Study Design
• However, it is not uncommon for researchers to not
correctly specify what type of study they are doing:
• Case-control studies that don’t recruit based on case
status
• RCTs that aren’t done completely randomly
• So CAREFULLY read the study methods to make sure
the researchers have correctly stated their study
design, and read for considerations on internal validity
How Methods can Affect Internal Validity
• Data collection
• A study that uses strong data collection methods will likely
have greater internal validity (objective v. subjective)
• For example, using self-reported data raises a few
questions
• How accurate is self-reported data?
• How may this impact our confidence in a study’s internal
validity?
• What can be done to increase the validity of self-reported data?
Determining Internal Validity
• Sometimes it can be challenging to determine
whether a study has adequate internal validity
• Requires significant critical thinking to assess internal
validity
• Think about each part of the study and its quality
• “Science and an art”
• Once we have determined internal validity, we can
consider the external validity (generalizability)
External Validity
• Generalizability: the extent to which findings from a study can be
applied to a broader or different population
• Another way to say this is external validity
• In order for a study to have external validity it must first have internal
validity
• Study has strong internal validity  how generalizable are findings?
• Study has low internal validity  not even really sure what we found among
our sample, so that is the big issue regardless of external validity
• Studies are usually generalizable to some population, but it can be tricky
to determine what that population is
• This part of research is not cut and dry. Two researchers can have different
opinions about how generalizable a study’s findings are
External Validity
• Usually more about sampling design
• For example
• Convenience sampling generally low external validity
• SRS generally greater external validity
Generalizations Key Considerations
• Are there major differences in the study population
and the population you’d like to generalize to?
• Are those differences likely to result in different
relationship between the exposure and outcome?
• Has this research question been studied in different
populations? Have they found consistent results?
• Is the phenomenon under consideration biologically
or socially?
Threats to Internal Validity
• Confounding
• Selection bias
• Misclassification/information bias
• Maturation
• History
• Differential loss to follow-up
• Repeated testing
• Regression towards the mean
What is a confounder?
• A confounding variable:
• 1– systematically
influences the
independent variable
• 2– influences the
dependent variable
• It is a third variable that is
associated both with the
exposure and the outcome
of interest that can make it
look like exposure and
outcome are related
Confounding
• One of the biggest threats to internal validity in observational
studies– keeps us from knowing whether the exposure is really
associated with the outcome or something else is causing the
relationship
• We cannot measure and control for all potential confounders, so
there is no way to have complete confidence in an observational
study
• Experimental studies are strong because randomization (of a
sufficient sample size) creates groups that are the same on all
characteristics, including confounders
What affects statistical significance?
• Effect size
• Confounding
• Sample size
• Type of analysis
• Adding more variables in a regression model can reduce
statistical power
Population
• If researchers have not chosen the population
correctly, this can also decrease internal validity
• Selection bias: participants who are chosen do not
represent the source population
Comparison Group
• If the study had not chosen the right comparison
group, can affect internal validity
• Different age distributions in the control and treatment
groups
• We can’t measure everything that may be different
between groups, so keep recruiting practices
consistent– all member of a cohort should be
recruited the same way
Selection Bias
• Pre-test differences between groups that exist and
may interact with the independent variable and thus
be “responsible” for the observed outcome
• Subjects have different probability of being selected
according to exposures or outcomes of interest,
creating a biased measure of association (i.e. odds
ratio, relative risk)
Retention
• Some people will be lost to follow-up in any study
(longitudinal cohort studies) – but people who are lost to
follow-up may be different from those who complete the
study
• Loss to follow-up may happen more in one group than
another (this is differential loss to follow-up)
• If there are large losses to follow-up this may affect the
results of the study – but it is difficult to know how will
impact the study
History
• Unanticipated event occurred during experiment
which could affect dependent variable
• Our intervention was effective, because smoking rates
decreased at follow-up compared to baseline
• However, there was also a tobacco tax passed during this
time period…