Types of study designs:

from descriptive studies to

randomized controlled trials

Kirsten Bibbins-Domingo, PhD, MD

Assistant Professor of Medicine and of
Epidemiology and Biostatistics
University of California, San Francisco

Objectives
To understand the difference
between
descriptive and analytic studies
To identify the hierarchy of study
designs, and the strengths and
weakness of each design
To be able to apply different study
designs to the same research
question

Types of Studies
Descriptive Studies
Observational Analytic Studies
Cross Sectional studies
Case Control studies
Cohort studies
Experimental Studies
Randomized controlled trials

Hierarchy of Study Types

Descriptive studies
Getting a lay of the land
Surveys (NHIS, MCBS)
How many men in the U.S. filled Viagra
prescriptions in 2004?
Describing a novel phenomena
Case reports or case series
Viagra-associated serous macular
detachment.
Sildenafil-associated nonarteritic anterior
ischemic optic neuropathy

Descriptive studies
Cannot establish causal relationships
Still play an important role in describing trends and
generating hypotheses about novel associations
The start of HIV/AIDS research
Squamous cell carcinoma in sexual partner of Kaposi
sarcoma patient. Lancet. 1982 Jan 30;1(8266):286.
New outbreak of oral tumors, malignancies and
infectious
diseases strikes young male homosexuals. CDA J. 1982
Mar;10(3):39-42.
AIDS in the "gay" areas of San Francisco. Lancet. 1983
Apr 23;1(8330):923-4

Analytic Studies
Attempt to establish a causal link between a
predictor/risk factor and an outcome.
You are doing an analytic study if you have
any of the following words in your research
question:
greater than, less than, causes, leads to,
compared with, more likely than,
associated
with, related to, similar to, correlated with

Research Question
Is the regular consumption of Red Bull
associated with improved academic
performance among U.S. medical
students?

Rationale
functional drink designed for periods
of mental and physical exertion.
performance, concentration,
memory,
reaction time, vigilance, and
emotional balance
Taurine + glucuronolactone + caffeine

Background
Alford C, Cox H, Wescott R. The effects of red bull energy drink on
human performance and mood. Amino Acids. 2001;21(2):139-50.
Warburton DM, Bersellini E, Sweeney E. An evaluation of a
caffeinated taurine drink on mood, memory and information
processing in healthy volunteers without caffeine abstinence.
Psychopharmacology (Berl). 2001 Nov;158(3):322-8.
Seidl R, Peyrl A, Nicham R, Hauser E. A taurine and caffeinecontaining drink stimulates cognitive performance and well-being.
Amino Acids. 2000;19(3-4):635-42.
Horne JA, Reyner LA. Beneficial effects of an "energy drink" given
to sleepy drivers. Amino Acids. 2001;20(1):83-9.
Kennedy DO, Scholey AB. A glucose-caffeine 'energy drink'
ameliorates subjective and performancedeficits during prolonged
cognitive demand. Appetite. 2004 Jun;42(3):331-3.

Great idea, but how do you get

started.

Interesting, novel, and relevant, but

You only have 25,000 dollars to
start investigating this question.
What is feasible?

Study Design #1
Cross-sectional study of UCSF medical
students taking USMLE Step 2
Questionnaire administered when
registering for USMLE 2
Primary predictor: self-report of >3 cans
Red Bull per week for the previous year
Covariates: Age, sex, undergraduate
university, place of birth
Outcome: Score on USMLE Step 2

Cross-sectional Study:
Descriptive value:
How many UCSF medical students drink Red Bull?
What is the age and sex distribution of UCSF medical
students who drink Red Bull?
Analytic value:
Is there an association between regular Red Bull consumption
and test scores among UCSF med students?
Univariate
Multivariate (controlling for confounders)
Other cross-sectional surveys:
AAMC
California Health Interview Survey (NHIS, CHIS)
National Health and Nutrition Exam Survey (NHANES)

Cross-sectional Study:
Pluses
Prevalence (not incidence)
+ Fast/Inexpensive - no waiting!
+ No loss to follow up
+ Associations can be studied

Measures of association

Cross-sectional study:
minuses
Cannot determine causality

Cross-sectional study:
minuses
Cannot determine causality
ACE inhibitor use and hospitalization
rates among those with heart failure
Heart failure patients with a
documented DNR status and
mortality

Cross-sectional study:
minuses
Cannot determine causality
- Cannot study rare outcomes

What if you are interested in

the rare outcome?

The association between regular Red

Bull consumption and
A perfect score on the USMLE
Step 2
Graduating top 1% of the medical
school
class
Acceptance into a highly selective
residency
ANSWER: A Case-Control study

Study Design #2
A case-control study
Cases: 4th year med students accepted to
residency in highly selective specialty X.
Controls: 4th year med students who
applied but were not accepted.
Predictor: self-reported regular Red Bull
consumption
Additional covariates (age, sex, medical
school,
undergraduate institution)

Case control studies

Investigator works backward
(from outcome to predictor)
Sample chosen on the basis of
outcome (cases), plus
comparison group (controls)

Case-control study structure

Red Bull consumption YES
Red Bull consumption NO
ACTUAL CASES
4th year UCSF students
who matched in highly
selective specialty X
ACTUAL CONTROLS
4th year students who failed
to match in highly selective
specialty X

Case control studies

Determines the strength of the
association between each predictor
variable and the
presence or absence of disease
Cannot yield estimates of
incidence or prevalence of
disease in the population (why?)
Odds Ratio is statistics

Case-control Study: pluses

Rare outcome/Long latent period
Inexpensive and efficient: may
be only feasible option
Establishes association (Odds ratio)
Useful for generating hypotheses
(multiple risk factors can be
explored)

Case-control study-minuses
Causality still difficult to establish
Selection bias (appropriate controls) Caffeine
and Pancreatic cancer in the GI clinic
Recall bias: sampling (retrospective)
Abortion and risk of breast cancer in Sweden
Cannot tell about incidence or prevalence
Studies of diagnostic tests:
Sensitivity, specificity
Positive predictive value, negative predictive
value

Case-control - the house

red
Rely tampons and toxic shock syndrome:
High rates of toxic shock syndrome in menstruating
women
Suspected OCPs or meds for PMS
Cases: 180 women with TSS in 6 geographic areas
Controls: 180 female friends of these patients and 180
females in the same telephone code
Tampon associated with TSS (OR = 29!)
Super absorbency associated with TSS (OR 1.34 per gm
increase in absorbency)
Led to RELY brand tampons being taken off the market.

Where are we?

Preliminary results from our cross-sectional and
case-control study suggest an association
between Red Bull consumption and improved
academic performance among medical students
Whats missing? - strengthening evidence for a
causal link between Red Bull consumption and
academic performance
Use results from our previous studies to apply
for funding for a prospective cohort study!

Study design #3
Prospective cohort study of UCSF medical
students Class of 2009
All entering medical students surveyed
regarding beverage consumption and variety
of other potential covariates
Survey updated annually to record
changes in Red Bull consumption
Outcomes: USMLE Step 1 score, USMLE
Step 2 score, match in first choice residency

Cohort studies
A cohort (follow-up, longitudinal) study is a comparative,
observational study in which subjects are grouped by
their exposure status, i.e., whether or not the subject was
exposed to a suspected risk factor
The subjects, exposed and unexposed to the risk factor,
are followed forward in time to determine if one or more
new outcomes
(diseases) occur
Subjects should not have outcome variable on entry
No new subjects allowed in after initial recruitment
The rates of disease incidence among the exposed and
unexposed groups are determined and compared.

Elements of a cohort study

Selection of sample from population
Measures predictor variables in sample
Follow population for period of time
Measure outcome variable
Famous cohort studies
Framingham
Nurses Health Study
Physicians Health Study
Olmsted County, Minnesota

Prospective cohort study structure

Strengths of cohort studies

Know that predictor variable was
present before outcome variable
occurred (some evidence of
causality)
Directly measureincidence of a
disease outcome
Can study multiple outcomes of a
single exposure (RR is measure of
association)

Weaknesses of cohort
studies
Expensive and inefficient for studying rare
outcomes
HERS vs. WHI
Often need long follow-up period or a very
large population
CARDIA
Loss to follow-up can affect validity of
findings
Framingham

Other types of cohort

studies
Retrospective cohort

Identification of cohort, measurement of
predictor variables, follow-up and
measurement of outcomes have all
occurred in the past

Much less costly than prospective cohorts

Investigator has minimal control over
study
design

Other types of cohort

studies
Nested case-control study
Case-control study embedded in a cohort study
Controls are drawn randomly from study sample
Double cohort
Used to compare two separate cohorts with
different levels of exposure to predictor variable
(e.g., occupational groups)

What type of study is this?

Among individuals with coronary disease, what is the
association between baseline levels of B-type natriuretic
peptide and subsequent risk of heart failure?
Among individuals presenting to heart failure clinic, what
is the association between self-reported symptoms and
risk of hospitalization for heart failure?
Using data from HERS (RCT of HRT in women with coronary
disease):
Determine the risk factors for developing incident heart failure
among women without heart failure at baseline.
Determine whether HRT is associated with mortality among
women with heart failure.
Determine genetic markers for development of heart failure
among black women in HERS

What distinguishes observational

studies from experiments?

Ability to control for confounding

Predictor
Outcome
Confounder
Examples:
sex (men are more likely to drink red bull and men
are
more likely to match in neurosurgery)
Undergraduate institution (students from
northwest school are more likely to drink red bull
and also more likely to score higher on USMLE)

But we measured all of the

potential confounders

In a prospective cohort study you can

(maybe) measure all potential known
confounders, but
You cant control for unanticipated or
unmeasured confounders

Study design # 4
Randomized controlled trial of daily
Red Bull
consumption among entering UCSF
medical
students Class 2009
Randomized to daily consumption
of Red Bull
vs. daily consumption of placebo
Outcomes: USMLE Step 1 score,
USMLE Step

Randomized controlled trials

Investigator controls the predictor
variable (intervention or treatment)
Major advantage over observational
studies is ability to demonstrate causality
Randomization controls unmeasured
confounding
Only for mature research questions

Steps in a randomized
controlled trial

Select participants
high-risk for outcome (high incidence)
Likely to benefit and not be harmed
Likely to adhere
Measure baseline variables
Randomize
Eliminates baseline confounding
Types (simple, stratified, block)

Steps in a randomized
controlled trial
Blinding the intervention
As important as randomization
Eliminates
co intervention
biased outcome ascertainment
biased measurement of outcome
Follow subjects
Adherence to protocol
Lost to follow up
Measure outcome
Clinically important measures
Adverse events

What is Blinding?
Single blind - participants are not
aware of treatment group
Double blind - both participants and
investigators unaware
Triple blind - various meanings
persons who perform tests
outcome adjudicators
safety monitoring group

Why blind?: Co
interventions
Unintended effective interventions
participants use other therapy or
change
behavior
study staff, medical providers,
family or
friends treat participants differently
Nondifferential - decreases power
Differential - causes bias

Why blind?: Biased Outcome

Ascertainment or adjudication

If group assignment is known

participants may report symptoms or outcomes
differently
physicians or investigators may elicit symptoms
or outcomes differently
Study staff or adjudicators may classify similar
events differently in treatment groups
Problematic with soft outcomes
investigator judgement
participant reported symptoms, scales

Analysis of randomized
controlled trial

Analyzed like cohort study with RR

Intention to treat analysis
Most conservative interpretation
Include all persons assigned to
intervention group (including those who
did not get treatment or dropped out)
Subgroup analysis
Groups identified pre-randomization

High Quality Randomized

Trials
Tamper-proof randomization
Blinding of participants, study staff,
lab staff, outcome ascertainment and
adjudication
Adherence to study intervention and
protocol
Complete follow-up