#Topic 6 : New Statistics

(MGMT90203)

Amarendra Singh
Hongyi Sun
Georg Tamm

The new statistics: Why

and how (Cumming,2014)
 The 3 Central Problems in Research
1.Published research is a biased selection of all research
(File Drawer Effect: Bias introduced into the scientific literature by
selective publication, by a tendency to publish +ve results but not to
publish –ve or non-confirmatory results)
2.Data analysis and reporting are often selective and biased
(Data selection [increasing or decreasing sample size], data manipulation
and try to achieve p-value, which is sufficiently small)
3.Research Studies are rarely replicated, therefore false conclusions often
persist
(Any finding that once meets the criterion of statistical significance, is
taken to be true and doesn’t require replication – however, as Cummings
has demonstrated in the paper, p-value changes dramatically)

Underlying Causes

(1)Reliance on Null-Hypothesis Significance Testing (NHST)

(Null Hypothesis [H0]: An assumption, which the researcher hopes to

prove ‘not true’)

(2)The imperative to achieve statistical significance (p-value)

(If p-value is <.05, reject the H0)

(This threshold is key to getting published, receiving PhD, advancing

research/academic careers, receiving research funding etc.)
 New Statistics as an Alternative

- Cummings suggests staying away from NHST

- Instead let's use New Statistics
- It can help solve the 3 problems
 What is New Statistics?

● Puts emphasis on the importance of effect

sizes, confidence intervals and Meta analysis
● Also emphasises Open Science Practice
● Does 3 things
○ Reduce overconfidence in small samples
(people like to make conclusions based
on small samples)
○ Reduces confirmation bias
○ Fosters more reserved judgements of
consistence

Meta-analysis
● Domain expertise and informed judgement at every steps,
defining questions, etc
● Meta analysis thinking, Consider past and future study on
contribution
● Model of meta-analytics, possibly random-effect model, but
assumptions are possibly unrealistic
● No use of NHST
 How do we do it?
The 8-Step New Statistics Strategy
1. Formulate a research question that avoids dichotomous expression and
instead uses estimation terms.
2. Identify values of interest that help answer your question. Also known as ES
3. Register details of intended study such as the Procedure and how the data
will be analysed. (This will stop you from meddling).
4. Calculate Point Estimates and Confidence Intervals for chosen ES’s
5. Make one or more figures including CI’s
6. Interpret the ES and CI’s
7. Use meta analytic thinking throughout the study
8. Report: Make your data available to others (including raw data)

The New Statistics for Better

Science (Jageman & Cumming
2019)
 What does New Statistics help with?

1. Reducing overconfidence in small samples

2. Reducing confirmation bias
3. Fostering more reserved judgements of consistency

What is overconfidence?

● People tend to make conclusions based on small samples

● New Statistics approach helps analyse problems differently

Example: Instead of asking does Drug A work, the new statistics method
will ask how much does Drug A work?

Note: New Statistics will estimate size effect and use confidence intervals
to express uncertainty when generalizing from small samples
 Reducing confirmation bias
● NHST reinforces
confirmation bias (lack of
significance is inconclusive
an experiment may not have
been carried out correctly)
● New Statistics puts
emphasis on applying
empirical standards through
Meta-Analysis.

Cautious Judgements of Consistency

● Judgement of Consistency is an important tool in understanding
mechanisms, like why an effect occurs and under what conditions
● NHST approach will often say something is conclusive if both
studies are statistically significant
● New statistics approach compares confidence intervals for both, as
such results are inconsistent only if intervals do not overlap
● This is problematic too as results can be statistically inconsistent
even with a small degree of overlap between confidence intervals
Questions?