Reading Scientific Papers

And other useful mathematical tools for

journalists
A Mathematicians Perspective
Rebecca Goldin
December 8, 2014
National Press Foundation

Sense About Science: STATS.org

Rebecca Goldin, Director of Research, STATS.org
Trevor Butterworth, Director, Sense About Science, USA

Statistical Concepts in
Scientific Journal
Articles
Mean, median, mode
Standard deviation
Confidence intervals
Orders of magnitude
Confounding factors
Percentages
Absolute vs. relative risk
Scientific methods
Impact of Study Design
Causation versus correlation

In the beginning: a press release

Press Releases Dont Tell
the Whole Story.
Designed to get attention
by the press
Present the results in the
rosiest terms possible.
Dont put the results in
context of other research
Shy away from concrete
descriptions

Abstracts Dont Tell the

Whole Story. They dont
answer:
How were the subjected
recruited?
What was the design of
the experiment?
What methods were used
to analyze the data?
What are the weaknesses
of the conclusions?

(Easy) Questions to Ask While

Reading a Journal Article
How is data collected? Is there room for bias (this is

especially important in survey, opinion, and food

studies).
Is the data biologically relevant?
Are the numbers significant? (more on significance
in a bit)
Usually scientific articles are trying implicate
something or some behavior red meat eaters have
more cancer translates as red meat causes more
cancer. Are the authors considering several possible
explanations for observed data (such as the different

(Easy) Questions to Ask While

Reading a Journal Article
How are the people in the study recruited? In

particular, would the recruitment method itself bias

the results by involving people who might not be
typical with regard to the thing measured?
For example, are women who have a family history of
breast cancer more likely to get a mammogram? If
so, then rates of cancer detection among women
getting mammograms may be higher than they would
be if every woman gets a mammogram.

The Moral and Statistical

Collide
Medical Abortion
Obesity
Nursing vs. Bottle

Feeding
Smoking
Homosexuality
Daycare
Food/Alcohol
Natural versus
Chemical
Pollution
Crime

Causation or Correlation

Its easy to be fooled

Height correlates with reading skills in children

under 10.
Income correlates with success in college.
Ratio of finger lengths correlates with aggression.
Facebook correlates with poor grades.
Facebook correlates with good grades.
Doing heroin correlates with doing marijuana.
Higher taxes correlate with high annual growth,
and are inversely correlated with poverty rates.
Alcoholism correlates with less gray matter in the
prefrontal cortex.

fMRI studies a case study

fMRIs are large magnets

measuring oxygen levels in

blood
People can engage in
activities inside the machine
Patterns of blood flow are
thought to reflect patterns
of brain activity (more on
that in a bit).
Typical studies: assume that observed patterns
only occur when the tested behavior occurs.
Typical studies: assume that observed patterns
are caused by the tested behavior.

fMRI studies a case study

Lying can be determined by

patterns of fMRI scans.

But perhaps stress or anxiety
can lead to the same patterns
Violent video gaming leads
to violent brain patterns
But perhaps any competitive
play, including non-violent
non-video games has similar brain patterns. Plus, no
indication of actual violence.
Math anxiety triggers activity in the pain center of the
brain.
But no pain experienced by subject with math anxiety.
Perhaps anxiety, not mathematics, correlated with

Jumping from Correlation to

Cause
You dont always have to know why it may not be

causal. Be wary of any claims of causality.

Some common reasons that a correlation could look
causal when its not include: not adjusting for
confounders, misunderstanding the mechanism,
having an unknown confounder.
A causal relationship might be reasonable to suspect
when the statistics are
Overwhelming

Observed in many different contexts

Repeated tests show the same effect, on large numbers

of people
Double blind case-control studies.

On p-values
Suppose you are flipping a coin

many times, and you think this coin

is biased, because you arent
getting close to heads and
tails. How can you quantify your
The p-value is a measurement
based on the data you have
suspicion?

seen: it answers the question: if the coin were fair, how likely
would I be to see the data I am seeing? In other words, if
you had a fair coin, is it reasonable to see the proportion of
heads/tails, or is it very unlikely to see that?
If you flip 1000 times, and you get 520 heads, there is just
under a 10% chance of getting this many heads (or more). In
contrast, if you had 550 heads in 1000 flips, the chance of
this happening randomly is only about .1%., i.e. very unlikely
if the coin were fair.
The biomedical community generally accepts p=.05 (5%) as

The normal curve

Confidence Intervals, Odds Ratios,

Confounders

Confidence Intervals are significant when they do not

contain the number 1.0.

Multiple Testing, or How to

Guarantee Results
Once you have a standard, like p<.05, you have

ways of gaming the system. There is always a

small chance, something less than 5%, that you
will see something that looks suspicious when it
really isnt. Sometimes your coin will favor heads
by a suspicious amount, the coin really is fair.
The more hypotheses you check, the more likely
this it to happen.
And once you find something suspicious, you can
write a scientific article about that.

What happens in the lab:

Experiments Galore...

What the
rest of the
world sees

Hard to get our heads around

randomness
My Magic Penny

How I found it.

Predicts the presidency: H

I asked 1000 of my

for Democrats, T for

Republicans
It was correct 15 out of 17
of the previous elections.
Just .1% chance of that
happening by chance, so
it truly is magic.
If it werent a coin, but
were some indicator like
the economy, the party of
the incumbent, June
approval ratings, you
would probably think
theres a reason for
success.

best friends to flip a

penny 17 times,
writing H or T for each
trial. Compared to the
list of correct
presidents
Picked the best coin.
In a room of 1000
flippers, there is a
70% chance of finding
a magic coin.

Do you see any patterns?

CHOICE A:
H H T T T T H H T H T H H T T H H H T T T H T T H T T T

THHTTHHTTTTHHTTTTTHHHTHHTHHT
HTTHHTTTHTTHHTHTHHTHHHHTHTTH
THHHTHHTHHTTTHHH
CHOICE B:
T T T T T H T H T H H H H H T H H T H H H H T T T T H H
HTHTHHHHTHTHHHTHTTHHTTTTTTTT
HHTTTHHTHHTHHHTHTTTTHHHHTHHH
HHTTHHHTTTHHTTTH

Which is random?

Some flip data

20
15
10
5
0
1 in a 2 in a
row row 3 in a 4 in a 5 in a
6 in a 7 in a
row row
row
row
row

Choice A
8 in a
row

There is an 81% probability that a sequence of

100 flips will have a run of 6 or more Hs or Ts
in a row.

Metaphors for Bad Statistical

Methods
Texas Sharpshooter Fallacy

Absolute versus relative risk

Absolute risk is the risk you actually undergo.

Women who take the birth control pill have an

absolute risk of venous thrombosis (blood clot) of
about 1 in 10,000 per year. The absolute risk of
women who do not take the pill is 1 in 15,000 per
year.
Relative risk is a risk compared to another group.
Women who take the birth control pill have a 50%
increased risk of venous thrombosis, compared to
women who dont take the pill.

Basic Advice for a Journalist with

Limited Time and Ideas
Read as a skeptic at all times. Avoid most

conclusions of causality.
A lot can be understood by even a cursory read (<10
minutes) of the summary, the abstract, and the
conclusion. Avoid the press release.
The summary and abstract will tell you the results,
but hardly ever hint as to what the limitations are.
The conclusion will often tell you some caveats.
Look up on PubMed.gov key words and see if other
literature has been published on the topic give
other research equal time!

Research is routinely plagued

Research is plagued
Low levels of
significance
Multiple testing
No acknowledgement of
randomness in research
design
Lack of
context/repeated
experiments
Scientists dont know
how to talk to
journalists.
But if you are looking to
find one scientist willing

What can a journalist

do?
Write about the levels of
significance, bias,
caveats
Ask the researchers
about multiple testing.
Did they adjust for
them?
Write about absolute
risks.
Look for a body of
research rather than
one specific paper
Cite your sources!
DONT INDICATE

Thank you!