Foundations of Research and Evidence- Based Practice

Lecture 12

Understanding Results 3

Nerida L. Klupp
Where are we up to.
Step 1. Is the result from data that is

Continuous
Categorical

Step 2. Is the result

Descriptive
Inferential




Where are we up to
Step 3. If inferential, is the result about

Statistical Significance
Clinical Significance

We will continue on with
clinical significance..
Minimum Important Difference

Minimum Important Difference (MID) is the

smallest worthwhile difference

(improvement) expected by a patient to
proceed with a treatment
MID Example
Pain scale 0-10 (O is no pain, 10 is worst pain
imaginable)
You have heel pain and have marked this as 8
on pain scale
Treatment options are
A) $10 comfy cushioned heel pad
B) $300 custom orthoses, daily strengthening and
stretching
C) $5000 surgery, 6 weeks functional interruption,
post-operative discomfort and some risk of harm
MID Example
What is the smallest improvement (points on a
pain scale) you would expect to make it
worthwhile to have each treatment?

Treatment MID
A
B
C
What do you do with it?
You compare the treatment effect size with the
minimal important difference. E.g. The amount of
improvement in pain from having surgery for
spinal pathology

MID for having surgery= 80% improvement
95% Confidence Interval = Between 50% and 60%

This means the best evidence suggests average improvement
in pain following this procedure is between 50% and 60%. The
patient does not proceed because their minimum
improvement wanted to undergo surgery is at least 80%.
Minimal important difference
At this stage you need to know!

What MID means
How it is used on a tree plot to determine
clinical significance

To set an MID for a patient or population
evidence question requires further clinical
reasoning skills. You are not expected to
suggest MID values until third year.
So what is a tree plot?

The box represents the point estimate and
the line represents the confidence interval
Confidence Interval = 6-10kg
Mean = 8kg
treatment is not
worthwhile
Smallest clinically
worthwhile effect or MID
treatment is
worthwhile
no effect
2 4 6 8 10 12 14 16 -10 -8 -6 -4 -2 0
Tree Plot
Line of no effect
(no difference between groups)
treatment is not
worthwhile
Smallest clinically worthwhile
effect or MID: 5 kg weight loss
treatment is
worthwhile
no effect
Treatment effect = 8 kg weight loss following resistance
training (95% CI 6 to 10) in favour of treatment group
2 4 6 8 10 12 14 16 -10 -8 -6 -4 -2 0
Tree Plot
Clinical Significance

Statistical significance is that p-value < 0.05

Clinical significance is when the treatment
effect (confidence interval) is equal or
more than the MID.
treatment is not
worthwhile
Smallest clinically worthwhile
effect or MID: 5 kg weight loss
treatment is
worthwhile
no effect
2 4 6 8 10 12 14 16 -10 -8 -6 -4 -2 0
Tree Plot
Is this result clinically significant?
Why?
If you put it all together

Difference in means =


The result that matters is the difference in weight
loss between the groups (not within a group)


8kg (95%CI 6kg-10kg) p<0.05
This is descriptive
It is the mean measured
from the sample
This is inferential
It is an estimation of the true
mean in the population
This is statistical
significance
This is clinical
significance
This is a
p-value
But this is for continuous data .
what do we do if the data is
categorical?
Revision
Continuous data involves an actual
measurement e.g. average student mark on
assignment = 62 marks

Categorical data involves a proportion of events
e.g. how many fails compared to passes = 5 fail
grades compared to 10 pass grades

Both examples could provide information about
student performance in an assignment
Categorical results
Categorical data are presented as
proportions experiencing an event or
outcome of interest.

Risk ratio is a comparison of risks

Also called relative risk
Risk does not mean a harmful thing

TITANIC
SURVIVAL
ALIVE DEAD
FEMALE 308 154
MALE 142 709
Risk of death on Titanic
TOTAL
462
Females
851
Males
Risk of death on Titanic
The risk ratio can compare the probability of
death according to sex
The probability of a woman dying was 154
(women who died) out of 462 (total women)
(154/462=0.33) or 33%.
The probability of a man dying was 709 (men
who died) out of 851 (total men)
(709/851=0.8331) or 83%.
Risk of death on Titanic
The risk ratio can compare the probability of
death according to sex
83% (male death proportion) compared to 33%
(female death proportion)
= (0.8331/0.3333).
= There is a 2.5 greater probability of death
for males than for females.

Risk ratio = 2.5
Categorical results
Minimal important differences (MIDs) are also
set in exactly the same way. How much more
likely do you want an event to occur to make an
intervention worthwhile?

Confidence intervals are also the same
e.g. RR =3 (95%CI 2.2- 3.8)
In the actual research the outcome was three times
more likely, but in the population we can estimate with
95% certainty that the outcome will be between 2.2
and 3.8 times more likely
Example
The box represents the point estimate and
the line represents the confidence interval
Confidence Interval = 2.2 3.8
Risk Ratio = 3
No effect
One thing is different

Continuous: If there is no difference between
2 actual measures, the difference = 0
E.g. If both groups lose 5kg then the difference
between their improvements = 0

Categorical: If there is no difference between
risks, the difference value is 1
No effect
E.g. If both groups (intervention and control)
have 10 persons out of 20 survive a heart attack
then 10/10 = 1/1 =1
If there is no difference between the groups and
the ratio will be 1 to 1, which is 1/1, which = 1
Therefore the line of no effect = 1

Odds ratios might also be provided and can be
interpreted much the same way.



No effect
treatment is not
worthwhile
Smallest clinically worthwhile
effect or MID: 2.5 times greater risk
treatment is
worthwhile
no effect
Risk Ratio = 3 (95% CI 2.2 to 3.8) in favour of
intervention group
Minimal Important Difference = 2.5
1 1.5 2 2.5 3 3.5 4 4.5 5 0.001 0.01 0.1 0.5
Tree plot for categorical data
Clinical significance

A result is clinically significant if

The point estimate is larger than the MID
Most of the confidence interval larger than MID
A narrow confidence interval. If it is narrow, it is
called precise.



treatment is not
worthwhile
Smallest clinically worthwhile
effect or MID: 5 kg weight loss
treatment is
worthwhile
no effect
Mean treatment effect = 2 kg in weight loss following
resistance training (95% CI -4 to 0) in favour of control group
group
2 4 6 8 10 12 14 16 -10 -8 -6 -4 -2 0
Study A
treatment is not
worthwhile
Smallest clinically worthwhile
effect or MID: 5 kg weight loss
treatment is
worthwhile
no effect
Mean treatment effect = 8 kg in weight loss following
resistance training (95% CI -2 to 18) in favour of treatment group
2 4 6 8 10 12 14 16 -10 -8 -6 -4 -2 0
Study B
Activity

From the Study A and B on the previous two slides

1) Which study found the treatment effect was
clinically significant ? (the intervention works)
2) Which study found the treatment effect was not
clinically significant? (the control group did better-
the intervention did not work)
3) Which study has the most precise results?



treatment is not
worthwhile
smallest clinically
worthwhile effect or
MID
very harmful
treatment
very effective
treatment
treatment is
worthwhile
no effect
Study A
Study B
Study C
Tree plot of effect size
Systematic reviews
The results of systematic reviews are
called forest plots
They plot multiple tree plots.
One tree plot = one clinical trial
Results will be either mean differences or
risk/odds ratios depending on type of data
A meta-analysis combines or pools the
data of the trials
Systematic review results
The Forest Plot

Each tree plot is a single clinical trial.
The diamond at the bottom is the pooled effect size.
The centre of the diamond is the point estimate and the
sides of the diamond are the confidence intervals.
Music for pain relief: Categorical
Categorical Forest Plot: Risk Ratio
Music for pain relief: Continuous
Continuous Forest Plot: Mean Difference
Tips on Reading Results
Dont skip the results section!!
Decide on what is your primary research question
Decide on your minimal important difference
Find the point estimate and confidence intervals
for that question
Make sure they are for group comparisons
And ignore all the less relevant numbers!



Video clip

Other studies
These lectures on results have only
discussed evidence on interventionswhat
about studies about diagnosis, prognosis,
and aetiology?
They might use risk ratios
You will learn some other common results
for these types of studies next year.
Still to come
Final lecture

Lecture 14
Revision session on challenging topics
Exam advice
Bring any questions before your exam

Thank you!