Types of study designs:

from descriptive studies to

randomized controlled trials

Kirsten Bibbins-Domingo, PhD, MD

Assistant Professor of Medicine and of
Epidemiology and Biostatistics
University of California, San Francisco

Objectives

To understand the difference between

descriptive and analytic studies

To identify the hierarchy of study

designs, and the strengths and
weakness of each design

To be able to apply different study

designs to the same research question

Types of Studies

Descriptive Studies

Observational Analytic Studies

Cross Sectional studies

Case Control studies
Cohort studies

Experimental Studies

Randomized controlled trials

Hierarchy of Study Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome

Descriptive studies

Getting a lay of the land

Surveys (NHIS, MCBS)

How many men in the U.S. filled Viagra

prescriptions in 2004?

Describing a novel phenomena

Case reports or case series

Viagra-associated serous macular

detachment.
Sildenafil-associated nonarteritic anterior
ischemic optic neuropathy.

Descriptive studies

Cannot establish causal relationships

Still play an important role in describing trends

and generating hypotheses about novel
associations

The start of HIV/AIDS research

Squamous cell carcinoma in sexual partner of Kaposi

sarcoma patient. Lancet. 1982 Jan 30;1(8266):286.
New outbreak of oral tumors, malignancies and infectious
diseases strikes young male homosexuals. CDA J. 1982
Mar;10(3):39-42.
AIDS in the "gay" areas of San Francisco. Lancet. 1983
Apr 23;1(8330):923-4.

Analytic Studies

Attempt to establish a causal link between

a predictor/risk factor and an outcome.

You are doing an analytic study if you have

any of the following words in your research
question:

greater than, less than, causes, leads to,

compared with, more likely than, associated
with, related to, similar to, correlated with

Hierarchy of Study Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome

Research Question
Is the regular consumption of Red Bull
associated with improved academic
performance among U.S. medical students?

Rationale

functional drink designed for periods of

mental and physical exertion.

performance, concentration, memory,

reaction time, vigilance, and emotional
balance

Taurine + glucuronolactone + caffeine

Background

Alford C, Cox H, Wescott R. The effects of red bull energy drink on

human performance and mood. Amino Acids. 2001;21(2):139-50.

Warburton DM, Bersellini E, Sweeney E. An evaluation of a

caffeinated taurine drink on mood, memory and information
processing in healthy volunteers without caffeine abstinence.
Psychopharmacology (Berl). 2001 Nov;158(3):322-8.

Seidl R, Peyrl A, Nicham R, Hauser E. A taurine and caffeinecontaining drink stimulates cognitive performance and well-being.
Amino Acids. 2000;19(3-4):635-42.

Horne JA, Reyner LA. Beneficial effects of an "energy drink" given

to sleepy drivers. Amino Acids. 2001;20(1):83-9.

Kennedy DO, Scholey AB. A glucose-caffeine 'energy drink'

ameliorates subjective and performancedeficits during prolonged
cognitive demand. Appetite. 2004 Jun;42(3):331-3.

Great idea, but how do you get

started.

Interesting, novel, and relevant, but

You only have 25,000 dollars to start

investigating this question.

What is feasible?

Study Design #1

Cross-sectional study of UCSF medical students

taking USMLE Step 2

Questionnaire administered when registering

for USMLE 2

Primary predictor: self-report of >3 cans Red Bull

per week for the previous year
Covariates: Age, sex, undergraduate university, place
of birth

Outcome: Score on USMLE Step 2

Cross-sectional study: structure

Red Bull consumption

USMLE Score

time

Cross-sectional Study:

Descriptive value:

How many UCSF medical students drink Red Bull?

What is the age and sex distribution of UCSF medical

students who drink Red Bull?
Analytic value:

Is there an association between regular Red Bull

consumption and test scores among UCSF med students?

Univariate
Multivariate (controlling for confounders)

Other cross-sectional surveys:

AAMC

California Health Interview Survey (NHIS, CHIS)

National Health and Nutrition Exam Survey (NHANES)

Cross-sectional Study: Pluses

+

Prevalence (not incidence)

Fast/Inexpensive - no waiting!

No loss to follow up

Associations can be studied

Measures of association
Risk ratio
(relative risk)

Disease
Yes

No

Yes

No

Risk
Factor

A
A+B
C
C+D

Cross-sectional study: minuses

-

Cannot determine causality

Red Bull consumption

USMLE Score

time

Cross-sectional study: minuses

-

Cannot determine causality

ACE inhibitor use and hospitalization rates

among those with heart failure
Heart failure patients with a documented DNR
status and mortality

time

Cross-sectional study: minuses

-

Cannot determine causality

Cannot study rare outcomes

What if you are interested in

the rare outcome?

The association between regular Red Bull

consumption and

A perfect score on the USMLE Step 2

Graduating top 1% of the medical school
class
Acceptance into a highly selective residency
ANSWER: A Case-Control study

Study Design #2

A case-control study
Cases: 4th year med students accepted to
residency in highly selective specialty X.
Controls: 4th year med students who applied
but were not accepted.
Predictor: self-reported regular Red Bull
consumption

Additional covariates (age, sex, medical school,

undergraduate institution)

Case control studies

Investigator works backward

(from outcome to predictor)

Sample chosen on the basis of

outcome (cases), plus comparison
group (controls)

Case-control study structure

present

Red Bull consumption

YES
Red Bull consumption
NO

ACTUAL CASES
TARGET
CASES
4th year UCSF students
Medical students accepted to
who matched in highly
highly selective residencies
selective specialty X

ACTUAL
CONTROLS
TARGET
CONTROLS
th
4
year students
who to
failed
All unsuccessful
applicants
highly
to
match
in
highly
selective
selective residency programs
specialty X
time

Case control studies

Determines the strength of the

association between each
predictor variable and the
presence or absence of disease
Cannot yield estimates of
incidence or prevalence of
disease in the population (why?)
Odds Ratio is statistics

Case-control Study: pluses

+ Rare outcome/Long latent period
+ Inexpensive and efficient: may be only
feasible option
+ Establishes association (Odds ratio)
+ Useful for generating hypotheses
(multiple risk factors can be explored)

Case-control study-minuses
-

Causality still difficult to establish

-

Selection bias (appropriate controls)

-

Recall bias: sampling (retrospective)

-

Caffeine and Pancreatic cancer in the GI clinic

Abortion and risk of breast cancer in Sweden

Cannot tell about incidence or prevalence

-

Studies of diagnostic tests:

-

Sensitivity, specificity
Positive predictive value, negative predictive value

Measures of association
Disease

Yes

Sensitivity = A/A+C

Yes

No

Test

Specificity = D/B+D

PPV = A/A+B
NPV = D/C+D

No

Case-control - the house red

Rely tampons and toxic shock syndrome:

High rates of toxic shock syndrome in menstruating women

Suspected OCPs or meds for PMS

Cases: 180 women with TSS in 6 geographic areas

Controls: 180 female friends of these patients and 180

females in the same telephone code

Tampon associated with TSS (OR = 29!)

Super absorbency associated with TSS (OR 1.34 per gm

increase in absorbency)

Led to RELY brand tampons being taken off the market.

Where are we?

Preliminary results from our cross-sectional and

case-control study suggest an association
between Red Bull consumption and improved
academic performance among medical students

Whats missing? - strengthening evidence for a

causal link between Red Bull consumption and
academic performance

Use results from our previous studies to apply

for funding for a prospective cohort study!

Study design #3

Prospective cohort study of UCSF medical

students Class of 2009
All entering medical students surveyed
regarding beverage consumption and
variety of other potential covariates
Survey updated annually to record
changes in Red Bull consumption
Outcomes: USMLE Step 1 score, USMLE
Step 2 score, match in first choice
residency

Cohort studies

A cohort (follow-up, longitudinal) study is a

comparative, observational study in which
subjects are grouped by their exposure status,
i.e., whether or not the subject was exposed to
a suspected risk factor

The subjects, exposed and unexposed to the

risk factor, are followed forward in time to
determine if one or more new outcomes
(diseases) occur

Subjects should not have outcome variable on entry

No new subjects allowed in after initial recruitment

The rates of disease incidence among the

exposed and unexposed groups are
determined and compared.

Elements of a cohort study

Selection of sample from population

Measures predictor variables in sample
Follow population for period of time
Measure outcome variable

Famous cohort studies

Framingham
Nurses Health Study
Physicians Health Study
Olmsted County, Minnesota

Prospective cohort study structure

The present

The future

Top USMLE scorers

Everyone else
time

Strengths of cohort studies

Know that predictor variable was present

before outcome variable occurred (some
evidence of causality)

Directly measure incidence of a disease

outcome

Can study multiple outcomes of a single

exposure (RR is measure of association)

Weaknesses of cohort studies

Expensive and inefficient for studying rare

outcomes

Often need long follow-up period or a very large

population

HERS vs. WHI

CARDIA

Loss to follow-up can affect validity of findings

Framingham

Other types of cohort studies

Retrospective cohort

Identification of cohort, measurement of

predictor variables, follow-up and
measurement of outcomes have all occurred
in the past
Much less costly than prospective cohorts
Investigator has minimal control over study
design

Other types of cohort studies

Nested case-control study

Case-control study embedded in a cohort study

Controls are drawn randomly from study sample

Double cohort

Used to compare two separate cohorts with

different levels of exposure to predictor variable
(e.g., occupational groups)

What type of study is this?

Among individuals with coronary disease, what is the

association between baseline levels of B-type natriuretic
peptide and subsequent risk of heart failure?

Among individuals presenting to heart failure clinic, what

is the association between self-reported symptoms and
risk of hospitalization for heart failure?

Using data from HERS (RCT of HRT in women with

coronary disease):

Determine the risk factors for developing incident heart

failure among women without heart failure at baseline.

Determine whether HRT is associated with mortality

among women with heart failure.

Determine genetic markers for development of heart

failure among black women in HERS.

Hierarchy of Study Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome

What distinguishes observational

studies from experiments?

Ability to control for confounding

Confounder

Predictor

Outcome

Examples:
sex (men are more likely to drink red bull and men are
more likely to match in neurosurgery)
Undergraduate institution (students from northwest school are
more likely to drink red bull and also more likely to score higher on
USMLE)

But we measured all of the

potential confounders.

In a prospective cohort study you can

(maybe) measure all potential known
confounders, but

You cant control for unanticipated or

unmeasured confounders

Study design # 4

Randomized controlled trial of daily Red Bull

consumption among entering UCSF medical
students Class 2009

Randomized to daily consumption of Red Bull

vs. daily consumption of placebo

Outcomes: USMLE Step 1 score, USMLE Step

2 score, match in first choice residency

Randomized controlled trials

Investigator controls the predictor

variable (intervention or treatment)
Major advantage over observational
studies is ability to demonstrate
causality
Randomization controls unmeasured
confounding
Only for mature research questions

Basic Trial Design

Population

Sample

Treatment

Dx

No Dx

Randomization

Control
Placebo

Dx

No Dx

Steps in a randomized
controlled trial
1.

Select participants

2.
3.

high-risk for outcome (high incidence)

Likely to benefit and not be harmed
Likely to adhere

Measure baseline variables

Randomize

Eliminates baseline confounding

Types (simple, stratified, block)

Steps in a randomized
controlled trial
4.

Blinding the intervention

5.

Follow subjects

6.

As important as randomization
Eliminates

co intervention

biased outcome ascertainment

biased measurement of outcome

Adherence to protocol
Lost to follow up

Measure outcome

Clinically important measures

Adverse events

What is Blinding?

Single blind - participants are not

aware of treatment group
Double blind - both participants
and investigators unaware
Triple blind - various meanings

persons who perform tests

outcome adjudicators
safety monitoring group

Why blind?: Co interventions

Unintended effective interventions

participants use other therapy or change

behavior
study staff, medical providers, family or
friends treat participants differently

Nondifferential - decreases power

Differential - causes bias

Why blind?: Biased Outcome

Ascertainment or adjudication

If group assignment is known

participants may report symptoms or outcomes

differently
physicians or investigators may elicit symptoms
or outcomes differently
Study staff or adjudicators may classify similar
events differently in treatment groups

Problematic with soft outcomes

investigator judgement
participant reported symptoms, scales

Analysis of randomized
controlled trial

Analyzed like cohort study with RR

Intention to treat analysis

Most conservative interpretation

Include all persons assigned to
intervention group (including those who
did not get treatment or dropped out)

Subgroup analysis

Groups identified pre-randomization

High Quality Randomized Trials

Tamper-proof randomization
Blinding of participants, study
staff, lab staff, outcome
ascertainment and adjudication
Adherence to study intervention
and protocol
Complete follow-up

A study type of every budget,

purpose andof
research
Hierarchy
Study question
Types

Descriptive
Case report
Case series
Survey

Analytic

Observational
Cross sectional
Case-control
Cohort studies

Experimental
Randomized
controlled trials

Strength of evidence for causality between a risk factor and outcome